[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 13:00:10.702445   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:44365
I20260709 13:00:10.704697   431 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 13:00:10.706147   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:10.730391   438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:10.731731   440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:10.733114   437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:12.019671   439 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260709 13:00:12.020167   431 server_base.cc:1056] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260709 13:00:12.027047   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:12.027508   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:12.027880   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602012027812 us; error 0 us; skew 500 ppm
I20260709 13:00:12.029155   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:12.037925   431 webserver.cc:533] Webserver started at http://127.0.107.254:33737/ using document root <none> and password file <none>
I20260709 13:00:12.039023   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:12.039285   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:12.039860   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:12.046820   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "e52e29d1345340f3a1b57c3c7a3b84c1"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.057919   431 fs_manager.cc:696] Time spent creating directory manager: real 0.010s	user 0.011s	sys 0.000s
I20260709 13:00:12.066167   446 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.069041   431 fs_manager.cc:730] Time spent opening block manager: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:00:12.069693   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "e52e29d1345340f3a1b57c3c7a3b84c1"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.070168   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:12.140689   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:12.142881   431 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 13:00:12.143464   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:12.247737   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:44365
I20260709 13:00:12.247833   497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:44365 every 8 connection(s)
I20260709 13:00:12.259300   498 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:12.287456   498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: Bootstrap starting.
I20260709 13:00:12.296516   498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:12.299773   498 log.cc:826] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: Log is configured to *not* fsync() on all Append() calls
I20260709 13:00:12.305877   498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: No bootstrap required, opened a new log
I20260709 13:00:12.333302   498 raft_consensus.cc:359] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER }
I20260709 13:00:12.334165   498 raft_consensus.cc:385] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:12.334458   498 raft_consensus.cc:740] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e52e29d1345340f3a1b57c3c7a3b84c1, State: Initialized, Role: FOLLOWER
I20260709 13:00:12.335736   498 consensus_queue.cc:260] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER }
I20260709 13:00:12.336339   498 raft_consensus.cc:399] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:12.336660   498 raft_consensus.cc:493] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:12.337095   498 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:12.342298   498 raft_consensus.cc:515] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER }
I20260709 13:00:12.343200   498 leader_election.cc:304] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e52e29d1345340f3a1b57c3c7a3b84c1; no voters: 
I20260709 13:00:12.346001   498 leader_election.cc:290] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:12.346431   501 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:12.351217   501 raft_consensus.cc:697] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 1 LEADER]: Becoming Leader. State: Replica: e52e29d1345340f3a1b57c3c7a3b84c1, State: Running, Role: LEADER
I20260709 13:00:12.352393   501 consensus_queue.cc:237] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER }
I20260709 13:00:12.353749   498 sys_catalog.cc:565] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:12.371932   503 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e52e29d1345340f3a1b57c3c7a3b84c1. Latest consensus state: current_term: 1 leader_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER } }
I20260709 13:00:12.373195   503 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:12.372056   502 sys_catalog.cc:455] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e52e29d1345340f3a1b57c3c7a3b84c1" member_type: VOTER } }
I20260709 13:00:12.373898   502 sys_catalog.cc:458] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:12.378248   511 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:12.398941   511 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:12.402508   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:12.425812   511 catalog_manager.cc:1383] Generated new cluster ID: af8f708261a742c2a7957f47365a06ad
I20260709 13:00:12.426291   511 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:12.467267   511 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:12.470642   511 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:12.493302   511 catalog_manager.cc:6070] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: Generated new TSK 0
I20260709 13:00:12.494845   511 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:12.540258   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:12.552928   520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:12.555994   521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:12.561652   431 server_base.cc:1061] running on GCE node
W20260709 13:00:12.562165   523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:12.563428   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:12.563683   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:12.563889   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602012563858 us; error 0 us; skew 500 ppm
I20260709 13:00:12.564524   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:12.570890   431 webserver.cc:533] Webserver started at http://127.0.107.193:39377/ using document root <none> and password file <none>
I20260709 13:00:12.571583   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:12.571859   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:12.572233   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:12.573745   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "2ecb644060e84ef5b32b4fdd470200f6"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.581635   431 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:00:12.587460   528 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.589365   431 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 13:00:12.589816   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "2ecb644060e84ef5b32b4fdd470200f6"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.590189   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:12.613109   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:12.614969   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:12.618183   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:12.624265   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:12.624593   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.625022   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:12.625293   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.769435   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:36613
I20260709 13:00:12.769667   590 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:36613 every 8 connection(s)
I20260709 13:00:12.779742   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:12.793388   595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:12.799784   596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:12.809706   598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:12.810416   431 server_base.cc:1061] running on GCE node
I20260709 13:00:12.811606   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:12.811818   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:12.812031   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602012811993 us; error 0 us; skew 500 ppm
I20260709 13:00:12.812704   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:12.818105   431 webserver.cc:533] Webserver started at http://127.0.107.194:43629/ using document root <none> and password file <none>
I20260709 13:00:12.818922   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:12.819207   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:12.819628   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:12.821684   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.825079   591 heartbeater.cc:344] Connected to a master server at 127.0.107.254:44365
I20260709 13:00:12.825735   591 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:12.827256   591 heartbeater.cc:507] Master 127.0.107.254:44365 requested a full tablet report, sending...
I20260709 13:00:12.829865   431 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:00:12.832134   463 ts_manager.cc:194] Registered new tserver with Master: 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613)
I20260709 13:00:12.836000   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37836
I20260709 13:00:12.836222   603 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.837411   431 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.002s
I20260709 13:00:12.837842   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
format_stamp: "Formatted at 2026-07-09 13:00:12 on dist-test-slave-p8rg"
I20260709 13:00:12.838238   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:12.862975   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:12.865468   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:12.869635   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:12.876092   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:12.876410   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:12.877296   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:12.877669   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.002s	sys 0.000s
I20260709 13:00:13.029968   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:34251
I20260709 13:00:13.030081   665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:34251 every 8 connection(s)
I20260709 13:00:13.042171   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:13.057683   669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:13.060201   670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:13.066869   431 server_base.cc:1061] running on GCE node
W20260709 13:00:13.068939   672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:13.069660   666 heartbeater.cc:344] Connected to a master server at 127.0.107.254:44365
I20260709 13:00:13.070076   666 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:13.070344   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:13.070664   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:13.070880   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602013070863 us; error 0 us; skew 500 ppm
I20260709 13:00:13.070974   666 heartbeater.cc:507] Master 127.0.107.254:44365 requested a full tablet report, sending...
I20260709 13:00:13.071722   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:13.073395   463 ts_manager.cc:194] Registered new tserver with Master: 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:13.075305   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37850
I20260709 13:00:13.076627   431 webserver.cc:533] Webserver started at http://127.0.107.195:35903/ using document root <none> and password file <none>
I20260709 13:00:13.077418   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:13.077793   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:13.078147   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:13.079710   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "449207e0c86e420ab6e1ed5d5f78702e"
format_stamp: "Formatted at 2026-07-09 13:00:13 on dist-test-slave-p8rg"
I20260709 13:00:13.086797   431 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.004s	sys 0.004s
I20260709 13:00:13.091565   677 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:13.092778   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 13:00:13.093279   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "449207e0c86e420ab6e1ed5d5f78702e"
format_stamp: "Formatted at 2026-07-09 13:00:13 on dist-test-slave-p8rg"
I20260709 13:00:13.093704   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:13.126176   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:13.127849   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:13.131031   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:13.135181   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:13.135437   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:13.135785   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:13.135954   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:13.284418   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:42649
I20260709 13:00:13.284565   739 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:42649 every 8 connection(s)
I20260709 13:00:13.308401   740 heartbeater.cc:344] Connected to a master server at 127.0.107.254:44365
I20260709 13:00:13.308924   740 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:13.309984   740 heartbeater.cc:507] Master 127.0.107.254:44365 requested a full tablet report, sending...
I20260709 13:00:13.312379   463 ts_manager.cc:194] Registered new tserver with Master: 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:13.313259   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021084366s
I20260709 13:00:13.314332   463 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37862
I20260709 13:00:13.391932   463 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37868:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:13.396217   463 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:13.586462   556 tablet_service.cc:1511] Processing CreateTablet for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:13.584874   554 tablet_service.cc:1511] Processing CreateTablet for tablet 5da6944d7e7540178df4276919d61a50 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:13.591811   555 tablet_service.cc:1511] Processing CreateTablet for tablet c5b47257091443409545dfd5a0169002 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:13.607544   704 tablet_service.cc:1511] Processing CreateTablet for tablet c5b47257091443409545dfd5a0169002 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:13.609764   704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b47257091443409545dfd5a0169002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.597366   553 tablet_service.cc:1511] Processing CreateTablet for tablet f8f40bec15324be498bd1d7ceea30a33 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:13.612946   703 tablet_service.cc:1511] Processing CreateTablet for tablet 5da6944d7e7540178df4276919d61a50 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:13.602568   552 tablet_service.cc:1511] Processing CreateTablet for tablet 93d0bfc309764e4b9a4dd81be0180776 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:13.615278   702 tablet_service.cc:1511] Processing CreateTablet for tablet f8f40bec15324be498bd1d7ceea30a33 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:13.615940   705 tablet_service.cc:1511] Processing CreateTablet for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:13.619293   702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f40bec15324be498bd1d7ceea30a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.626303   705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.628711   555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b47257091443409545dfd5a0169002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.636242   703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da6944d7e7540178df4276919d61a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.636338   701 tablet_service.cc:1511] Processing CreateTablet for tablet 93d0bfc309764e4b9a4dd81be0180776 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:13.643796   556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.643904   631 tablet_service.cc:1511] Processing CreateTablet for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:00:13.643919   628 tablet_service.cc:1511] Processing CreateTablet for tablet f8f40bec15324be498bd1d7ceea30a33 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:00:13.648837   627 tablet_service.cc:1511] Processing CreateTablet for tablet 93d0bfc309764e4b9a4dd81be0180776 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:00:13.656658   701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0bfc309764e4b9a4dd81be0180776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.659090   631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b2b40f1cf5a40b8b736f70b16a2a19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.660813   554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da6944d7e7540178df4276919d61a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.659163   630 tablet_service.cc:1511] Processing CreateTablet for tablet c5b47257091443409545dfd5a0169002 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:00:13.668236   628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f40bec15324be498bd1d7ceea30a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.653911   629 tablet_service.cc:1511] Processing CreateTablet for tablet 5da6944d7e7540178df4276919d61a50 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:00:13.673287   626 tablet_service.cc:1511] Processing CreateTablet for tablet 2d4e88ab70364fabb16659caabfc6bb7 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:00:13.666114   551 tablet_service.cc:1511] Processing CreateTablet for tablet 2d4e88ab70364fabb16659caabfc6bb7 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:00:13.674680   627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0bfc309764e4b9a4dd81be0180776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.676503   629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5da6944d7e7540178df4276919d61a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.677698   553 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8f40bec15324be498bd1d7ceea30a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.689247   552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d0bfc309764e4b9a4dd81be0180776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.682430   630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b47257091443409545dfd5a0169002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.691375   551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4e88ab70364fabb16659caabfc6bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.691231   700 tablet_service.cc:1511] Processing CreateTablet for tablet 2d4e88ab70364fabb16659caabfc6bb7 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:00:13.694633   700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4e88ab70364fabb16659caabfc6bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.701294   626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4e88ab70364fabb16659caabfc6bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.748925   550 tablet_service.cc:1511] Processing CreateTablet for tablet d8f0614c8e594993817349c8bbe75a65 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:13.755915   550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f0614c8e594993817349c8bbe75a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.790114   625 tablet_service.cc:1511] Processing CreateTablet for tablet d8f0614c8e594993817349c8bbe75a65 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:13.792886   625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f0614c8e594993817349c8bbe75a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.804879   699 tablet_service.cc:1511] Processing CreateTablet for tablet d8f0614c8e594993817349c8bbe75a65 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:00:13.810664   699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8f0614c8e594993817349c8bbe75a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.814302   549 tablet_service.cc:1511] Processing CreateTablet for tablet fff0e3bb765e4a60bbc4505595a11923 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:13.815763   549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff0e3bb765e4a60bbc4505595a11923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.840636   591 heartbeater.cc:499] Master 127.0.107.254:44365 was elected leader, sending a full tablet report...
I20260709 13:00:13.887619   759 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:13.949538   759 tablet_bootstrap.cc:654] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:13.961653   631 tablet_service.cc:1511] Processing CreateTablet for tablet fff0e3bb765e4a60bbc4505595a11923 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:13.963726   631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff0e3bb765e4a60bbc4505595a11923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:13.966939   760 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:13.988481   758 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:13.999548   703 tablet_service.cc:1511] Processing CreateTablet for tablet fff0e3bb765e4a60bbc4505595a11923 (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:00:14.001245   703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fff0e3bb765e4a60bbc4505595a11923. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:14.016996   555 tablet_service.cc:1511] Processing CreateTablet for tablet cfb29ed2c45c42f6ac9794da45f1d54d (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:14.018946   555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb29ed2c45c42f6ac9794da45f1d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:14.027608   704 tablet_service.cc:1511] Processing CreateTablet for tablet cfb29ed2c45c42f6ac9794da45f1d54d (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:14.029573   704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb29ed2c45c42f6ac9794da45f1d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:14.031173   628 tablet_service.cc:1511] Processing CreateTablet for tablet cfb29ed2c45c42f6ac9794da45f1d54d (DEFAULT_TABLE table=test-workload [id=95dce29bb0724dc58c8d8eb08751e198]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:00:14.036918   628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb29ed2c45c42f6ac9794da45f1d54d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:14.040264   759 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.040988   759 ts_tablet_manager.cc:1403] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.155s	user 0.020s	sys 0.014s
I20260709 13:00:14.058221   759 raft_consensus.cc:359] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.059373   759 raft_consensus.cc:385] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.059739   759 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.060956   759 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.080561   666 heartbeater.cc:499] Master 127.0.107.254:44365 was elected leader, sending a full tablet report...
I20260709 13:00:14.084704   760 tablet_bootstrap.cc:654] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.104264   758 tablet_bootstrap.cc:654] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.116737   759 ts_tablet_manager.cc:1434] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.075s	user 0.025s	sys 0.032s
I20260709 13:00:14.118300   759 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.128051   759 tablet_bootstrap.cc:654] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.140182   758 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.140827   758 ts_tablet_manager.cc:1403] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.197s	user 0.021s	sys 0.004s
I20260709 13:00:14.146962   760 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:14.148068   760 ts_tablet_manager.cc:1403] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.210s	user 0.024s	sys 0.004s
I20260709 13:00:14.152194   759 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.153084   759 ts_tablet_manager.cc:1403] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.019s
I20260709 13:00:14.154245   758 raft_consensus.cc:359] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.155143   758 raft_consensus.cc:385] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.155788   758 raft_consensus.cc:740] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.156777   758 consensus_queue.cc:260] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.154438   760 raft_consensus.cc:359] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.157805   760 raft_consensus.cc:385] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.157462   759 raft_consensus.cc:359] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.158486   760 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.159044   759 raft_consensus.cc:385] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.159417   759 raft_consensus.cc:740] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.160472   759 consensus_queue.cc:260] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.160512   760 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.163380   759 ts_tablet_manager.cc:1434] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 13:00:14.164790   759 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.167738   758 ts_tablet_manager.cc:1434] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.027s	user 0.013s	sys 0.000s
I20260709 13:00:14.168859   758 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.170236   740 heartbeater.cc:499] Master 127.0.107.254:44365 was elected leader, sending a full tablet report...
I20260709 13:00:14.173983   759 tablet_bootstrap.cc:654] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.181943   758 tablet_bootstrap.cc:654] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.193815   759 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.194397   759 ts_tablet_manager.cc:1403] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.011s
I20260709 13:00:14.199380   759 raft_consensus.cc:359] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.200766   759 raft_consensus.cc:385] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.203392   760 ts_tablet_manager.cc:1434] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.054s	user 0.008s	sys 0.007s
I20260709 13:00:14.207122   760 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:14.201213   759 raft_consensus.cc:740] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.208669   759 consensus_queue.cc:260] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.215618   759 ts_tablet_manager.cc:1434] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.003s
I20260709 13:00:14.217269   759 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.230515   759 tablet_bootstrap.cc:654] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.237885   758 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.238482   760 tablet_bootstrap.cc:654] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.238651   758 ts_tablet_manager.cc:1403] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.070s	user 0.030s	sys 0.031s
I20260709 13:00:14.243225   758 raft_consensus.cc:359] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.244298   758 raft_consensus.cc:385] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.244861   758 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.245743   758 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.248855   758 ts_tablet_manager.cc:1434] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.004s
I20260709 13:00:14.250546   758 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.262228   758 tablet_bootstrap.cc:654] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.269374   760 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:14.270761   760 ts_tablet_manager.cc:1403] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.064s	user 0.006s	sys 0.018s
I20260709 13:00:14.273353   759 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.273986   759 ts_tablet_manager.cc:1403] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.057s	user 0.015s	sys 0.010s
I20260709 13:00:14.274824   760 raft_consensus.cc:359] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.275763   760 raft_consensus.cc:385] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.276109   760 raft_consensus.cc:740] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.276643   758 tablet_bootstrap.cc:492] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.277334   758 ts_tablet_manager.cc:1403] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.027s	user 0.017s	sys 0.004s
I20260709 13:00:14.277781   760 consensus_queue.cc:260] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.278404   759 raft_consensus.cc:359] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.279194   759 raft_consensus.cc:385] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.279646   759 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
W20260709 13:00:14.281025   592 tablet.cc:2406] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:00:14.281036   760 ts_tablet_manager.cc:1434] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:00:14.283600   760 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:14.283886   758 raft_consensus.cc:359] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.284637   758 raft_consensus.cc:385] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.284963   758 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.285892   758 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.289772   758 ts_tablet_manager.cc:1434] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.001s
I20260709 13:00:14.290997   758 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.280763   759 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.295974   759 ts_tablet_manager.cc:1434] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.022s	user 0.003s	sys 0.008s
I20260709 13:00:14.297115   759 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.302469   762 raft_consensus.cc:493] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.304059   762 raft_consensus.cc:515] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.307341   762 leader_election.cc:290] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.307658   758 tablet_bootstrap.cc:654] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.327890   760 tablet_bootstrap.cc:654] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.344168   762 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.344965   762 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.347270   759 tablet_bootstrap.cc:654] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.349366   762 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.367012   758 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.370968   758 ts_tablet_manager.cc:1403] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.080s	user 0.023s	sys 0.009s
I20260709 13:00:14.378461   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.379835   715 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 0.
I20260709 13:00:14.383780   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.387703   529 leader_election.cc:304] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:14.388696   762 raft_consensus.cc:2804] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:00:14.388939   529 leader_election.cc:343] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.389160   762 raft_consensus.cc:493] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:14.389593   762 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:14.384559   509 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:14.391264   760 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:14.388607   758 raft_consensus.cc:359] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.396136   760 ts_tablet_manager.cc:1403] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.113s	user 0.013s	sys 0.024s
I20260709 13:00:14.396605   758 raft_consensus.cc:385] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.396733   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.398799   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.399102   641 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 0.
I20260709 13:00:14.398041   758 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
W20260709 13:00:14.401880   532 leader_election.cc:343] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.402420   532 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6; no voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:14.403978   759 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.404865   759 ts_tablet_manager.cc:1403] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.108s	user 0.018s	sys 0.018s
I20260709 13:00:14.404621   758 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.408727   760 raft_consensus.cc:359] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.413064   762 raft_consensus.cc:515] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.415371   780 raft_consensus.cc:2749] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:14.415880   758 ts_tablet_manager.cc:1434] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.044s	user 0.010s	sys 0.002s
I20260709 13:00:14.415949   760 raft_consensus.cc:385] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.417565   760 raft_consensus.cc:740] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.418313   759 raft_consensus.cc:359] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.418552   760 consensus_queue.cc:260] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.419203   759 raft_consensus.cc:385] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.419549   759 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.420512   758 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.422665   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:14.424437   641 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.425549   760 ts_tablet_manager.cc:1434] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.028s	user 0.010s	sys 0.001s
I20260709 13:00:14.420321   759 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.427274   760 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:14.433008   759 ts_tablet_manager.cc:1434] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.005s
I20260709 13:00:14.434260   759 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.436990   767 raft_consensus.cc:493] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.437934   767 raft_consensus.cc:515] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.440510   760 tablet_bootstrap.cc:654] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.440778   758 tablet_bootstrap.cc:654] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.441954   641 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 1.
I20260709 13:00:14.443607   532 leader_election.cc:304] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:14.444605   780 raft_consensus.cc:2804] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:14.451510   767 leader_election.cc:290] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.460367   766 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.464321   766 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.486740   780 raft_consensus.cc:697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:14.491616   780 consensus_queue.cc:237] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.499342   766 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:14.500075   766 raft_consensus.cc:493] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.500645   766 raft_consensus.cc:515] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.508726   766 leader_election.cc:290] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:14.511499   759 tablet_bootstrap.cc:654] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.519882   758 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.520220   781 raft_consensus.cc:493] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.521163   758 ts_tablet_manager.cc:1403] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.101s	user 0.019s	sys 0.009s
I20260709 13:00:14.521091   781 raft_consensus.cc:515] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.526037   758 raft_consensus.cc:359] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.527591   758 raft_consensus.cc:385] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.528019   758 raft_consensus.cc:740] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.528988   758 consensus_queue.cc:260] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.532202   766 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.540383   758 ts_tablet_manager.cc:1434] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.005s
I20260709 13:00:14.553319   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.539939   766 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.537760   781 leader_election.cc:290] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.565140   758 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.583704   767 raft_consensus.cc:493] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:14.566431   532 leader_election.cc:343] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.588905   767 raft_consensus.cc:515] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.614823   766 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:14.628868   767 leader_election.cc:290] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.630571   760 tablet_bootstrap.cc:492] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:14.631174   760 ts_tablet_manager.cc:1403] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.204s	user 0.022s	sys 0.013s
I20260709 13:00:14.640774   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.644446   714 raft_consensus.cc:2468] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 0.
I20260709 13:00:14.662114   780 raft_consensus.cc:493] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.661175   529 leader_election.cc:304] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:14.668151   780 raft_consensus.cc:515] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.668850   787 raft_consensus.cc:2804] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:14.669519   787 raft_consensus.cc:493] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:14.670121   787 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.679154   760 raft_consensus.cc:359] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.683671   760 raft_consensus.cc:385] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.686035   762 leader_election.cc:290] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.689391   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:14.691756   760 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.727471   759 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:14.738624   787 raft_consensus.cc:515] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.756273   759 ts_tablet_manager.cc:1403] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.322s	user 0.021s	sys 0.050s
I20260709 13:00:14.713060   760 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.728103   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
W20260709 13:00:14.716799   605 leader_election.cc:343] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.714849   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:14.761886   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.736076   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.779151   713 raft_consensus.cc:2468] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 0.
I20260709 13:00:14.773420   714 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.713646   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
W20260709 13:00:14.774760   605 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:36613, user_credentials={real_user=slave}} blocked reactor thread for 59613us
I20260709 13:00:14.777143   780 leader_election.cc:290] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.777030   462 catalog_manager.cc:5697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:14.808562   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
W20260709 13:00:14.778405   532 leader_election.cc:343] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.784679   604 leader_election.cc:304] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
W20260709 13:00:14.816850   604 leader_election.cc:343] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.817442   604 leader_election.cc:304] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:14.818661   767 raft_consensus.cc:2804] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:14.819305   767 raft_consensus.cc:493] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:14.819451   758 tablet_bootstrap.cc:654] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.820572   767 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.819406   760 ts_tablet_manager.cc:1434] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.188s	user 0.019s	sys 0.023s
W20260709 13:00:14.825052   532 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:34251, user_credentials={real_user=slave}} blocked reactor thread for 47482.2us
I20260709 13:00:14.830140   787 leader_election.cc:290] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
W20260709 13:00:14.831106   529 leader_election.cc:343] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.831589   529 leader_election.cc:304] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6; no voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:14.833338   759 raft_consensus.cc:359] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.834158   759 raft_consensus.cc:385] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.834475   759 raft_consensus.cc:740] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.834604   714 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 1.
I20260709 13:00:14.834628   787 raft_consensus.cc:2749] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:14.823187   605 leader_election.cc:343] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.835541   759 consensus_queue.cc:260] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.839226   759 ts_tablet_manager.cc:1434] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.082s	user 0.004s	sys 0.005s
I20260709 13:00:14.845042   760 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:14.849747   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:14.853154   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:14.859180   715 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.861747   759 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:14.865128   566 raft_consensus.cc:2393] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 449207e0c86e420ab6e1ed5d5f78702e in current term 1: Already voted for candidate 2ecb644060e84ef5b32b4fdd470200f6 in this term.
I20260709 13:00:14.864583   641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.866302   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:14.867230   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:14.871646   767 raft_consensus.cc:515] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.873262   780 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.867332   640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.880589   564 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:14.875070   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.880869   780 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.881779   638 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:14.879436   758 tablet_bootstrap.cc:492] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:14.871532   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:14.882676   767 leader_election.cc:290] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.882688   679 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:14.885185   767 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.884479   758 ts_tablet_manager.cc:1403] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.327s	user 0.030s	sys 0.018s
I20260709 13:00:14.886735   780 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.887524   766 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:14.887948   766 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:14.888386   766 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.888783   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.889586   714 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 0.
I20260709 13:00:14.890726   758 raft_consensus.cc:359] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.891395   529 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:14.891707   758 raft_consensus.cc:385] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:14.886713   794 raft_consensus.cc:2749] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:14.895074   681 leader_election.cc:343] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.895692   681 leader_election.cc:304] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e; no voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b
W20260709 13:00:14.896799   532 leader_election.cc:343] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.896828   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
W20260709 13:00:14.898383   681 leader_election.cc:343] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.898548   714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:14.892027   758 raft_consensus.cc:740] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:14.899204   714 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.900436   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:14.884567   565 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
W20260709 13:00:14.904152   532 leader_election.cc:343] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:14.906206   605 leader_election.cc:343] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:14.906416   780 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:14.909912   780 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:14.910415   780 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.912076   766 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.886264   767 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.915656   758 consensus_queue.cc:260] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.917382   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:14.918111   564 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.918361   767 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:14.917713   760 tablet_bootstrap.cc:654] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:14.922158   766 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:14.924911   566 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:14.927251   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:14.927968   638 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:14.928169   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:14.929977   713 raft_consensus.cc:2393] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fe07b221f304ed1ac4a53ba51f6285b in current term 1: Already voted for candidate 449207e0c86e420ab6e1ed5d5f78702e in this term.
I20260709 13:00:14.950088   714 raft_consensus.cc:2468] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 1.
I20260709 13:00:14.952761   638 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:14.942966   605 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:14.956523   767 raft_consensus.cc:2749] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:14.956943   758 ts_tablet_manager.cc:1434] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.070s	user 0.020s	sys 0.001s
I20260709 13:00:14.957324   604 leader_election.cc:304] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:14.958360   794 raft_consensus.cc:2804] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:14.958904   794 raft_consensus.cc:697] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:14.960088   794 consensus_queue.cc:237] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.958406   758 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:14.967008   787 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:14.954329   780 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.962450   679 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:14.975627   681 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:14.976177   564 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:14.968230   715 raft_consensus.cc:2468] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 1.
I20260709 13:00:14.980096   786 raft_consensus.cc:2749] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:14.981388   798 raft_consensus.cc:493] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:14.984894   798 raft_consensus.cc:515] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:14.986543   786 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:14.978345   766 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.000615   766 raft_consensus.cc:697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:15.000764   798 leader_election.cc:290] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:14.995776   786 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.002732   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:15.004967   786 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.008029   638 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.011749   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
W20260709 13:00:15.014037   532 leader_election.cc:343] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:15.014190   681 leader_election.cc:304] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:15.002748   766 consensus_queue.cc:237] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.015852   798 raft_consensus.cc:2804] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:15.016779   798 raft_consensus.cc:493] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.017231   798 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:15.007848   679 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:36613, user_credentials={real_user=slave}} blocked reactor thread for 45561.6us
I20260709 13:00:15.021224   780 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:15.024803   529 leader_election.cc:304] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:15.029249   760 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:15.030179   759 tablet_bootstrap.cc:654] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.030803   760 ts_tablet_manager.cc:1403] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.186s	user 0.020s	sys 0.017s
I20260709 13:00:15.032363   780 raft_consensus.cc:2804] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.036125   780 raft_consensus.cc:697] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:15.038342   780 consensus_queue.cc:237] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.040337   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:15.041152   564 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.039477   760 raft_consensus.cc:359] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.043008   760 raft_consensus.cc:385] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.043514   760 raft_consensus.cc:740] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.044684   760 consensus_queue.cc:260] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.050037   760 ts_tablet_manager.cc:1434] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.003s
I20260709 13:00:15.051684   760 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:15.050851   798 raft_consensus.cc:515] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.058295   461 catalog_manager.cc:5697] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 1 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:15.080216   798 leader_election.cc:290] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.084162   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:15.086215   639 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.089478   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:15.090268   564 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.092995   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:15.101468   758 tablet_bootstrap.cc:654] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.111052   463 catalog_manager.cc:5697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 0 to 1, leader changed from <none> to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 1 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:15.115245   564 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.115000   780 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.116547   679 leader_election.cc:304] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:15.117821   798 raft_consensus.cc:2804] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.118626   798 raft_consensus.cc:697] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:15.116763   462 catalog_manager.cc:5697] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:15.119669   798 consensus_queue.cc:237] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.113304   639 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.130264   786 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.131721   715 raft_consensus.cc:2393] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2ecb644060e84ef5b32b4fdd470200f6 in current term 1: Already voted for candidate 449207e0c86e420ab6e1ed5d5f78702e in this term.
I20260709 13:00:15.137988   529 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6; no voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:15.140954   780 raft_consensus.cc:2749] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:15.143111   786 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.145565   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
W20260709 13:00:15.147691   681 leader_election.cc:343] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:15.160503   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:15.179683   758 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:15.181790   758 ts_tablet_manager.cc:1403] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.224s	user 0.021s	sys 0.004s
I20260709 13:00:15.184679   758 raft_consensus.cc:359] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.185271   758 raft_consensus.cc:385] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.180917   760 tablet_bootstrap.cc:654] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.185611   758 raft_consensus.cc:740] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.194972   758 consensus_queue.cc:260] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.191804   564 raft_consensus.cc:2393] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 449207e0c86e420ab6e1ed5d5f78702e in current term 1: Already voted for candidate 2ecb644060e84ef5b32b4fdd470200f6 in this term.
I20260709 13:00:15.200534   679 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e; no voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:15.206606   798 raft_consensus.cc:2749] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:15.209255   759 tablet_bootstrap.cc:492] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:15.210305   759 ts_tablet_manager.cc:1403] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.354s	user 0.021s	sys 0.044s
I20260709 13:00:15.211676   758 ts_tablet_manager.cc:1434] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 13:00:15.213454   758 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:15.215457   463 catalog_manager.cc:5697] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 0 to 1, leader changed from <none> to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 1 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:15.216424   759 raft_consensus.cc:359] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.217342   759 raft_consensus.cc:385] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.217746   759 raft_consensus.cc:740] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.219611   759 consensus_queue.cc:260] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.226541   760 tablet_bootstrap.cc:492] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:15.227380   760 ts_tablet_manager.cc:1403] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.176s	user 0.017s	sys 0.020s
I20260709 13:00:15.231488   759 ts_tablet_manager.cc:1434] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.021s	user 0.010s	sys 0.001s
I20260709 13:00:15.233482   759 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:15.238708   758 tablet_bootstrap.cc:654] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.240582   760 raft_consensus.cc:359] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.241494   760 raft_consensus.cc:385] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.244560   760 raft_consensus.cc:740] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.251641   759 tablet_bootstrap.cc:654] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.253818   760 consensus_queue.cc:260] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.260213   760 ts_tablet_manager.cc:1434] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 13:00:15.261337   760 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:15.267302   758 tablet_bootstrap.cc:492] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:15.267926   758 ts_tablet_manager.cc:1403] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.055s	user 0.027s	sys 0.007s
I20260709 13:00:15.268277   759 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:15.269793   759 ts_tablet_manager.cc:1403] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.037s	user 0.018s	sys 0.003s
I20260709 13:00:15.274274   759 raft_consensus.cc:359] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.275803   759 raft_consensus.cc:385] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.276180   759 raft_consensus.cc:740] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.276984   759 consensus_queue.cc:260] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.279776   760 tablet_bootstrap.cc:654] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.279911   759 ts_tablet_manager.cc:1434] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:00:15.277568   758 raft_consensus.cc:359] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.281004   759 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: Bootstrap starting.
I20260709 13:00:15.280938   758 raft_consensus.cc:385] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.282545   758 raft_consensus.cc:740] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.283695   758 consensus_queue.cc:260] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.286690   758 ts_tablet_manager.cc:1434] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 13:00:15.287878   758 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: Bootstrap starting.
I20260709 13:00:15.290596   798 raft_consensus.cc:493] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.291149   798 raft_consensus.cc:515] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.293439   798 leader_election.cc:290] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.294734   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:15.295447   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:15.295441   639 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.296301   564 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.297986   681 leader_election.cc:304] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:15.299736   798 raft_consensus.cc:2804] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:15.300309   798 raft_consensus.cc:493] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.301628   798 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.314035   758 tablet_bootstrap.cc:654] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.317220   759 tablet_bootstrap.cc:654] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.312801   798 raft_consensus.cc:515] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.327616   798 leader_election.cc:290] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.328985   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:15.330041   639 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.329922   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:15.330576   758 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: No bootstrap required, opened a new log
I20260709 13:00:15.330698   564 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.331710   758 ts_tablet_manager.cc:1403] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: Time spent bootstrapping tablet: real 0.044s	user 0.017s	sys 0.004s
I20260709 13:00:15.334578   759 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: No bootstrap required, opened a new log
I20260709 13:00:15.335119   759 ts_tablet_manager.cc:1403] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: Time spent bootstrapping tablet: real 0.054s	user 0.009s	sys 0.005s
I20260709 13:00:15.334753   758 raft_consensus.cc:359] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.335813   758 raft_consensus.cc:385] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.336194   758 raft_consensus.cc:740] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.337015   758 consensus_queue.cc:260] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.338091   759 raft_consensus.cc:359] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.339592   758 ts_tablet_manager.cc:1434] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:15.340126   564 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.342202   679 leader_election.cc:304] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:15.338742   759 raft_consensus.cc:385] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.343003   798 raft_consensus.cc:2804] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.345688   798 raft_consensus.cc:697] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:15.346627   798 consensus_queue.cc:237] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.343231   759 raft_consensus.cc:740] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.348632   760 tablet_bootstrap.cc:492] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:15.352790   760 ts_tablet_manager.cc:1403] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.092s	user 0.018s	sys 0.033s
I20260709 13:00:15.351933   639 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.355151   759 consensus_queue.cc:260] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.358214   759 ts_tablet_manager.cc:1434] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 13:00:15.358013   760 raft_consensus.cc:359] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.358937   760 raft_consensus.cc:385] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.359308   760 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.359751   463 catalog_manager.cc:5697] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 0 to 1, leader changed from <none> to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 1 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:15.360168   760 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.362910   760 ts_tablet_manager.cc:1434] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:00:15.366364   760 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:15.376044   760 tablet_bootstrap.cc:654] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.389318   760 tablet_bootstrap.cc:492] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:15.389909   760 ts_tablet_manager.cc:1403] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.014s
I20260709 13:00:15.395589   760 raft_consensus.cc:359] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.396384   760 raft_consensus.cc:385] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.396791   760 raft_consensus.cc:740] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.397619   760 consensus_queue.cc:260] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
W20260709 13:00:15.399137   509 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:15.399783   760 ts_tablet_manager.cc:1434] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:00:15.400763   760 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: Bootstrap starting.
I20260709 13:00:15.408041   760 tablet_bootstrap.cc:654] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:15.408591   798 raft_consensus.cc:493] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.409089   798 raft_consensus.cc:515] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.411288   798 leader_election.cc:290] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.412832   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:15.412832   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:15.413579   564 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.414171   639 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 0.
I20260709 13:00:15.414999   679 leader_election.cc:304] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:15.415992   798 raft_consensus.cc:2804] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:15.416424   798 raft_consensus.cc:493] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.416801   798 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.419428   760 tablet_bootstrap.cc:492] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: No bootstrap required, opened a new log
I20260709 13:00:15.419996   760 ts_tablet_manager.cc:1403] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.005s
I20260709 13:00:15.424825   760 raft_consensus.cc:359] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.425705   760 raft_consensus.cc:385] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:15.426208   760 raft_consensus.cc:740] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Initialized, Role: FOLLOWER
I20260709 13:00:15.426837   798 raft_consensus.cc:515] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.427202   760 consensus_queue.cc:260] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.431007   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:15.431850   564 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.434244   760 ts_tablet_manager.cc:1434] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 13:00:15.434607   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:15.435269   639 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.430096   798 leader_election.cc:290] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:15.442895   564 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.444221   679 leader_election.cc:304] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:15.445168   798 raft_consensus.cc:2804] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.445890   798 raft_consensus.cc:697] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:15.444669   639 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:15.446790   798 consensus_queue.cc:237] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.460328   460 catalog_manager.cc:5697] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 0 to 1, leader changed from <none> to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 1 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:15.464059   798 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:15.483974   801 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.513315   794 raft_consensus.cc:493] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.514492   794 raft_consensus.cc:515] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.517289   794 leader_election.cc:290] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:15.518769   794 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:15.519414   794 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.521999   794 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:15.522183   762 consensus_queue.cc:1048] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:15.525422   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:15.525774   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e" is_pre_election: true
I20260709 13:00:15.526448   713 raft_consensus.cc:2393] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fe07b221f304ed1ac4a53ba51f6285b in current term 1: Already voted for candidate 449207e0c86e420ab6e1ed5d5f78702e in this term.
I20260709 13:00:15.527374   715 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 0.
I20260709 13:00:15.529332   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:15.530133   564 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 0.
I20260709 13:00:15.530632   565 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:15.531454   565 raft_consensus.cc:2393] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9fe07b221f304ed1ac4a53ba51f6285b in current term 1: Already voted for candidate 2ecb644060e84ef5b32b4fdd470200f6 in this term.
I20260709 13:00:15.531476   605 leader_election.cc:304] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:15.532699   827 raft_consensus.cc:2804] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:15.533120   827 raft_consensus.cc:493] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:15.533866   605 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:15.533788   827 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.534795   794 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.535329   565 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.548833   794 raft_consensus.cc:2749] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:15.550127   827 raft_consensus.cc:515] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.552465   794 consensus_queue.cc:1048] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:00:15.558314   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:15.559468   564 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.557538   827 leader_election.cc:290] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:15.573585   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:15.575588   639 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.574317   713 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:15.577741   827 consensus_queue.cc:1048] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.583117   564 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 1.
I20260709 13:00:15.584293   605 leader_election.cc:304] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:15.585131   827 raft_consensus.cc:2804] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:15.586760   827 raft_consensus.cc:697] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:15.587805   827 consensus_queue.cc:237] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:15.593729   780 consensus_queue.cc:1048] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.608404   713 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 1.
I20260709 13:00:15.630332   460 catalog_manager.cc:5697] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 1 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:15.652633   766 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.670987   786 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:15.840214   798 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:15.852918   766 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:15.870683   766 consensus_queue.cc:1048] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:15.885896   766 consensus_queue.cc:1048] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:16.013263   827 consensus_queue.cc:1048] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:16.028057   827 consensus_queue.cc:1048] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:16.402479   509 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:17.263516   844 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:17.264117   844 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.266093   844 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:17.267331   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" is_pre_election: true
I20260709 13:00:17.267329   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6" is_pre_election: true
I20260709 13:00:17.268042   639 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:17.268317   564 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 1.
I20260709 13:00:17.269277   681 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:17.270110   844 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:17.270539   844 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:17.270825   844 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.275969   844 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.277895   844 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:17.278908   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:17.279263   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:17.279484   564 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.279834   639 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.284725   564 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:17.285061   639 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:17.285981   679 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:17.286849   844 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:17.287981   844 raft_consensus.cc:697] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:17.288858   844 consensus_queue.cc:237] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.296891   463 catalog_manager.cc:5697] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 0 to 2, leader changed from <none> to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 2 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:17.737793   564 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:17.739246   844 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:17.751856   639 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:17.753363   844 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:17.897085   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5b47257091443409545dfd5a0169002"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:17.897953   715 raft_consensus.cc:606] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Received request to transfer leadership to TS 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:17.902768   844 raft_consensus.cc:993] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:17.903360   847 raft_consensus.cc:1081] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Signalling peer 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:17.904467   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:17.905119   715 raft_consensus.cc:606] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Received request to transfer leadership to TS 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:17.905208   564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5b47257091443409545dfd5a0169002"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:60082
I20260709 13:00:17.905829   564 raft_consensus.cc:493] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:17.906222   564 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.913470   564 raft_consensus.cc:515] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.916945   564 leader_election.cc:290] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 2 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:17.917827   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:17.918380   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:17.918721   715 raft_consensus.cc:606] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:17.919034   639 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.919732   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:17.921267   713 raft_consensus.cc:3055] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:17.921631   713 raft_consensus.cc:740] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:17.922554   713 consensus_queue.cc:260] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.925329   713 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:17.926873   715 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:17.927549   715 raft_consensus.cc:606] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:17.931927   713 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 2.
I20260709 13:00:17.933323   529 leader_election.cc:304] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:17.934265   843 raft_consensus.cc:2804] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:17.935427   843 raft_consensus.cc:697] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:17.936298   843 consensus_queue.cc:237] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:17.936949   639 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 2.
I20260709 13:00:17.938609   713 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:17.939134   713 raft_consensus.cc:606] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:17.949429   463 catalog_manager.cc:5697] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 1 to 2, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:17.961629   639 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:51532
I20260709 13:00:17.962394   639 raft_consensus.cc:606] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Received request to transfer leadership to TS 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:18.130466   844 raft_consensus.cc:993] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.131039   844 raft_consensus.cc:1081] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.132898   639 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51526
I20260709 13:00:18.133483   639 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:18.133898   639 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.139551   639 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.141830   639 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:18.142998   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:18.143002   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:18.143903   564 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.143916   713 raft_consensus.cc:3055] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:18.144501   713 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:18.146014   713 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.148334   713 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.153818   564 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.154111   847 raft_consensus.cc:993] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.154625   844 raft_consensus.cc:1081] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.155243   605 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:18.156219   845 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:18.156523   713 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.156785   639 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51526
I20260709 13:00:18.157562   639 raft_consensus.cc:493] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:18.157699   845 raft_consensus.cc:697] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:18.158080   639 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.159103   845 consensus_queue.cc:237] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.164366   639 raft_consensus.cc:515] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.166579   639 leader_election.cc:290] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:18.167552   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:18.167699   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:18.168211   564 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.168718   713 raft_consensus.cc:3055] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:18.169072   713 raft_consensus.cc:740] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:18.169874   713 consensus_queue.cc:260] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.171185   713 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.174520   463 catalog_manager.cc:5697] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 1 to 2, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 2 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:18.175590   564 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.177449   605 leader_election.cc:304] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:18.178822   845 raft_consensus.cc:2804] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:18.184500   713 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.184208   845 raft_consensus.cc:697] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:18.186008   845 consensus_queue.cc:237] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.189141   847 raft_consensus.cc:993] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:18.189692   847 raft_consensus.cc:1081] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Signalling peer 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:18.192533   564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:60082
I20260709 13:00:18.193221   564 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:18.194103   564 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:18.202492   463 catalog_manager.cc:5697] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 1 to 2, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 2 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:18.203073   564 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.205721   564 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 3 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:18.206777   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:18.207428   639 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:18.208290   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:18.208935   713 raft_consensus.cc:3055] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:18.209280   713 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:18.210170   713 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.211663   713 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:18.216552   639 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 3.
I20260709 13:00:18.217948   532 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:18.218708   843 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:18.219148   843 raft_consensus.cc:697] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:18.219988   843 consensus_queue.cc:237] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.220371   713 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 3.
I20260709 13:00:18.232579   463 catalog_manager.cc:5697] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 2 to 3, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:18.239683   866 raft_consensus.cc:993] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:18.248595   845 raft_consensus.cc:1081] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Signalling peer 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:18.251505   713 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:59672
I20260709 13:00:18.252856   713 raft_consensus.cc:493] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:18.253345   713 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.265236   713 raft_consensus.cc:515] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.270356   713 leader_election.cc:290] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:18.271517   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:18.271600   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfb29ed2c45c42f6ac9794da45f1d54d" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:18.272594   564 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.272625   639 raft_consensus.cc:3055] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:18.273358   639 raft_consensus.cc:740] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:18.274596   639 consensus_queue.cc:260] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.276034   639 raft_consensus.cc:3060] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.281843   564 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:18.285061   679 leader_election.cc:304] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:18.286432   847 raft_consensus.cc:2804] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:18.287160   639 raft_consensus.cc:2468] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:18.287879   847 raft_consensus.cc:697] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:18.289052   847 consensus_queue.cc:237] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.298031   463 catalog_manager.cc:5697] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 1 to 2, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 2 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:18.365944   844 raft_consensus.cc:993] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.366591   844 raft_consensus.cc:1081] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:18.368868   639 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51526
I20260709 13:00:18.369577   639 raft_consensus.cc:493] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:18.370023   639 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.376438   639 raft_consensus.cc:515] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.378418   639 leader_election.cc:290] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:18.379657   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:18.379534   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fff0e3bb765e4a60bbc4505595a11923" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:18.380455   715 raft_consensus.cc:3055] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:18.380505   564 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.381145   715 raft_consensus.cc:740] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:18.382429   715 consensus_queue.cc:260] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.384159   715 raft_consensus.cc:3060] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:18.390795   564 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.392485   605 leader_election.cc:304] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:18.393154   715 raft_consensus.cc:2468] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:18.393849   845 raft_consensus.cc:2804] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:18.394434   845 raft_consensus.cc:697] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:18.395345   845 consensus_queue.cc:237] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:18.404184   463 catalog_manager.cc:5697] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 1 to 2, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 2 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:18.471189   715 raft_consensus.cc:1275] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.472788   843 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:18.487784   639 raft_consensus.cc:1275] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.492810   860 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:18.575292   564 raft_consensus.cc:1275] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.577391   845 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.590557   715 raft_consensus.cc:1275] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.592670   866 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.601442   564 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.603669   866 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:18.609336   715 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.610780   845 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.718801   715 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:18.720594   843 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.731141   639 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:18.733172   860 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.742889   564 raft_consensus.cc:1275] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.744262   844 consensus_queue.cc:1048] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.755822   639 raft_consensus.cc:1275] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.758072   885 consensus_queue.cc:1048] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:18.926419   715 raft_consensus.cc:1275] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.927954   866 consensus_queue.cc:1048] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:18.937319   564 raft_consensus.cc:1275] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:18.939249   879 consensus_queue.cc:1048] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:21.006592   431 auto_leader_rebalancer-test.cc:316] The leader distribution is 
I20260709 13:00:21.008322   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:21.032800   564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
mode: GRACEFUL
new_leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:43160
I20260709 13:00:21.033629   564 raft_consensus.cc:606] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Received request to transfer leadership to TS 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:21.047667   639 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:21.048328   639 raft_consensus.cc:606] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Received request to transfer leadership to TS 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:21.050064   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:00:21.050637   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:21.114198   892 raft_consensus.cc:993] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:21.114789   879 raft_consensus.cc:1081] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Signalling peer 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:21.116811   715 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:59672
I20260709 13:00:21.117529   715 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:21.118193   715 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:21.135766   715 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.138221   715 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 3 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:21.139302   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:21.139580   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:21.140357   639 raft_consensus.cc:3055] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:21.140695   639 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:21.141788   639 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.143155   639 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:21.154062   639 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 3.
I20260709 13:00:21.155947   681 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:21.157183   890 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:21.158442   890 raft_consensus.cc:697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:21.159421   890 consensus_queue.cc:237] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.167851   463 catalog_manager.cc:5697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 2 to 3, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 3 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:21.579236   903 raft_consensus.cc:993] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: : Instructing follower 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:21.579849   891 raft_consensus.cc:1081] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Signalling peer 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:21.582173   715 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:59668
I20260709 13:00:21.582842   715 raft_consensus.cc:493] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:21.583312   715 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:21.629364   564 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:21.668354   564 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:21.668088   715 raft_consensus.cc:515] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.670763   890 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:21.671814   715 leader_election.cc:290] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:21.671916   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:21.672839   564 raft_consensus.cc:3055] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:21.673594   564 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:21.674605   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:21.675333   639 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:21.675318   564 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.677419   564 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:21.691287   641 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:21.694429   905 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:21.737026   639 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:21.737066   564 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:21.738370   681 leader_election.cc:304] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:21.739292   905 raft_consensus.cc:2804] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:21.739847   905 raft_consensus.cc:697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:21.741207   905 consensus_queue.cc:237] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:21.751585   462 catalog_manager.cc:5697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 1 to 2, leader changed from 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193) to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 2 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:22.051669   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:22.064028   713 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:22.064987   713 raft_consensus.cc:606] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:22.084702   564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5b47257091443409545dfd5a0169002"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:43170
I20260709 13:00:22.085290   564 raft_consensus.cc:606] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:22.095288   564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:43170
I20260709 13:00:22.096071   564 raft_consensus.cc:606] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:22.103513   564 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:43170
I20260709 13:00:22.104182   564 raft_consensus.cc:606] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:22.110765   713 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
mode: GRACEFUL
new_leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:59674
I20260709 13:00:22.111459   713 raft_consensus.cc:606] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 LEADER]: Received request to transfer leadership to TS 9fe07b221f304ed1ac4a53ba51f6285b
I20260709 13:00:22.126945   903 raft_consensus.cc:993] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.127530   891 raft_consensus.cc:1081] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.129379   639 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51512
I20260709 13:00:22.129833   639 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:22.130086   639 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.136564   639 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.138520   639 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 4 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:22.139654   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:22.139658   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:22.140302   564 raft_consensus.cc:3055] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:22.140506   713 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.140616   564 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:22.141459   564 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.142683   564 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.147780   903 raft_consensus.cc:993] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.148427   891 raft_consensus.cc:1081] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.150763   639 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51512
I20260709 13:00:22.152451   639 raft_consensus.cc:493] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:22.152874   639 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:22.203450   641 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:22.205389   905 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:22.214098   890 raft_consensus.cc:993] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.214968   907 raft_consensus.cc:1081] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.217128   638 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51526
I20260709 13:00:22.221123   566 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:22.222306   638 raft_consensus.cc:493] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:22.222854   638 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.222919   905 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:22.232990   564 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 4.
I20260709 13:00:22.233476   639 raft_consensus.cc:515] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.235230   605 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:22.235909   639 leader_election.cc:290] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:22.237675   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:22.238346   715 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:22.240463   879 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:22.242187   879 raft_consensus.cc:697] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:22.243436   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8f40bec15324be498bd1d7ceea30a33" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:22.243213   879 consensus_queue.cc:237] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.244050   564 raft_consensus.cc:3055] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:22.244273   564 raft_consensus.cc:740] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:22.244887   564 consensus_queue.cc:260] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.245909   564 raft_consensus.cc:3060] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:22.232990   713 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 4.
I20260709 13:00:22.337167   564 raft_consensus.cc:2468] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:22.337167   715 raft_consensus.cc:2468] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 2.
I20260709 13:00:22.337208   638 raft_consensus.cc:515] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.339226   605 leader_election.cc:304] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:22.342000   915 raft_consensus.cc:2804] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:22.342733   915 raft_consensus.cc:697] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:22.342943   638 leader_election.cc:290] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 3 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:22.344408   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:22.344404   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:22.345225   564 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.345337   715 raft_consensus.cc:3055] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:22.346155   715 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:22.346444   915 consensus_queue.cc:237] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.347474   715 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.348899   715 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.357088   463 catalog_manager.cc:5697] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 1 to 2, leader changed from 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 2 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:22.358829   463 catalog_manager.cc:5697] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 3 to 4, leader changed from 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 4 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:22.382217   903 raft_consensus.cc:993] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.382938   903 raft_consensus.cc:1081] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.384752   638 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5b47257091443409545dfd5a0169002"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51512
I20260709 13:00:22.385252   638 raft_consensus.cc:493] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:22.385710   638 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.436837   564 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 3.
I20260709 13:00:22.436874   638 raft_consensus.cc:515] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.436846   715 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 3.
I20260709 13:00:22.438654   605 leader_election.cc:304] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:22.439476   638 leader_election.cc:290] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 3 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:22.440416   915 raft_consensus.cc:2804] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:22.440668   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:22.440833   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5b47257091443409545dfd5a0169002" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:22.440989   915 raft_consensus.cc:697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:22.441444   564 raft_consensus.cc:3055] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:22.441819   564 raft_consensus.cc:740] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:22.442163   915 consensus_queue.cc:237] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.442591   564 consensus_queue.cc:260] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.443761   564 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.441341   715 raft_consensus.cc:3060] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:22.455265   463 catalog_manager.cc:5697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 2 to 3, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 3 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:22.481290   890 raft_consensus.cc:993] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: : Instructing follower 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.482127   890 raft_consensus.cc:1081] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 LEADER]: Signalling peer 9fe07b221f304ed1ac4a53ba51f6285b to start an election
I20260709 13:00:22.484324   638 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
 from {username='slave'} at 127.0.0.1:51526
I20260709 13:00:22.484949   638 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:22.485356   638 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.543036   564 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 3.
I20260709 13:00:22.543036   715 raft_consensus.cc:2468] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 3.
I20260709 13:00:22.543095   638 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.545011   604 leader_election.cc:304] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:22.546149   915 raft_consensus.cc:2804] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:22.546881   915 raft_consensus.cc:697] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:22.545751   638 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 4 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:22.547564   715 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:22.547601   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:22.548326   715 raft_consensus.cc:3055] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:22.548616   564 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.547859   915 consensus_queue.cc:237] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.548662   715 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:22.550658   715 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.552917   715 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:22.558967   463 catalog_manager.cc:5697] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 2 to 3, leader changed from 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 3 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:22.642900   715 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 4.
I20260709 13:00:22.642866   564 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9fe07b221f304ed1ac4a53ba51f6285b in term 4.
I20260709 13:00:22.644724   604 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 449207e0c86e420ab6e1ed5d5f78702e, 9fe07b221f304ed1ac4a53ba51f6285b; no voters: 
I20260709 13:00:22.645764   915 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:22.646606   915 raft_consensus.cc:697] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Becoming Leader. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:22.647696   915 consensus_queue.cc:237] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:22.655754   463 catalog_manager.cc:5697] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b reported cstate change: term changed from 3 to 4, leader changed from 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195) to 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194). New cstate: current_term: 4 leader_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:22.720028   564 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:00:22.721729   915 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:22.731594   715 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:00:22.733134   932 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:22.794601   564 raft_consensus.cc:1275] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:22.796064   932 consensus_queue.cc:1048] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:22.806635   715 raft_consensus.cc:1275] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:22.808081   932 consensus_queue.cc:1048] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:22.868613   564 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:22.870532   932 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:22.879918   715 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:22.883224   879 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:22.947853   564 raft_consensus.cc:1275] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:22.949887   879 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:22.960691   715 raft_consensus.cc:1275] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:22.963739   932 consensus_queue.cc:1048] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:23.070552   564 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:23.072270   932 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:23.082396   715 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Refusing update from remote peer 9fe07b221f304ed1ac4a53ba51f6285b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:23.084194   923 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:25.128815   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:25.139384   638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:25.140085   638 raft_consensus.cc:606] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Received request to transfer leadership to TS 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:25.143208   638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:25.143833   638 raft_consensus.cc:606] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Received request to transfer leadership to TS 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:25.146662   638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:25.147333   638 raft_consensus.cc:606] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Received request to transfer leadership to TS 2ecb644060e84ef5b32b4fdd470200f6
I20260709 13:00:25.151177   638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:25.152968   638 raft_consensus.cc:606] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Received request to transfer leadership to TS 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:25.156581   638 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65"
dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
mode: GRACEFUL
new_leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:44808
I20260709 13:00:25.157236   638 raft_consensus.cc:606] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Received request to transfer leadership to TS 449207e0c86e420ab6e1ed5d5f78702e
I20260709 13:00:25.158847   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:00:25.159518   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:25.240473   915 raft_consensus.cc:993] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:25.241079   915 raft_consensus.cc:1081] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Signalling peer 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:25.243233   713 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:59672
I20260709 13:00:25.243759   713 raft_consensus.cc:493] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:25.244041   713 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:25.249459   713 raft_consensus.cc:515] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.251590   713 leader_election.cc:290] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:25.252748   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:25.253003   915 raft_consensus.cc:993] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.253016   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8f0614c8e594993817349c8bbe75a65" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:25.253566   564 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:25.253814   879 raft_consensus.cc:1081] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Signalling peer 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.253973   638 raft_consensus.cc:3055] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:25.254415   638 raft_consensus.cc:740] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:25.255200   638 consensus_queue.cc:260] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.255862   566 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5da6944d7e7540178df4276919d61a50"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:60072
I20260709 13:00:25.256412   638 raft_consensus.cc:3060] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:25.256490   566 raft_consensus.cc:493] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:25.256955   566 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.261412   564 raft_consensus.cc:2468] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:25.262081   638 raft_consensus.cc:2468] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 2.
I20260709 13:00:25.262972   679 leader_election.cc:304] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:25.263922   890 raft_consensus.cc:2804] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:25.265472   890 raft_consensus.cc:697] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:25.265547   566 raft_consensus.cc:515] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.266547   890 consensus_queue.cc:237] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.268513   566 leader_election.cc:290] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 5 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:25.269747   638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:25.270510   638 raft_consensus.cc:3055] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:25.269882   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5da6944d7e7540178df4276919d61a50" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:25.270979   638 raft_consensus.cc:740] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:25.271481   713 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.272199   638 consensus_queue.cc:260] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.273753   638 raft_consensus.cc:3060] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.275204   463 catalog_manager.cc:5697] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 1 to 2, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 2 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:25.277046   713 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 5.
I20260709 13:00:25.278426   529 leader_election.cc:304] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:25.280238   944 raft_consensus.cc:2804] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:25.281926   944 raft_consensus.cc:697] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:25.282565   638 raft_consensus.cc:2468] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 5.
I20260709 13:00:25.284822   944 consensus_queue.cc:237] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.296036   463 catalog_manager.cc:5697] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 4 to 5, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 5 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:25.405128   915 raft_consensus.cc:993] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.405915   915 raft_consensus.cc:1081] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Signalling peer 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.407627   564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:60072
I20260709 13:00:25.408130   564 raft_consensus.cc:493] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:25.408386   564 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:25.412528   564 raft_consensus.cc:515] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.414417   564 leader_election.cc:290] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 4 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:25.415493   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:25.415858   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b2b40f1cf5a40b8b736f70b16a2a19b" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:25.416217   639 raft_consensus.cc:3055] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:25.416549   713 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:25.416710   639 raft_consensus.cc:740] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:25.417742   639 consensus_queue.cc:260] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.418886   639 raft_consensus.cc:3060] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:25.422158   950 raft_consensus.cc:993] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.422686   915 raft_consensus.cc:1081] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Signalling peer 2ecb644060e84ef5b32b4fdd470200f6 to start an election
I20260709 13:00:25.422875   713 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 4.
I20260709 13:00:25.423991   529 leader_election.cc:304] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:25.424407   564 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7"
dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
 from {username='slave'} at 127.0.0.1:60072
I20260709 13:00:25.424906   944 raft_consensus.cc:2804] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:25.425050   564 raft_consensus.cc:493] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:25.425429   564 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:25.426126   639 raft_consensus.cc:2468] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 4.
I20260709 13:00:25.426139   944 raft_consensus.cc:697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:25.427181   944 consensus_queue.cc:237] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.432863   564 raft_consensus.cc:515] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.434597   564 leader_election.cc:290] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 3 election: Requested vote from peers 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251), 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195:42649)
I20260709 13:00:25.435614   713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
I20260709 13:00:25.435760   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d4e88ab70364fabb16659caabfc6bb7" candidate_uuid: "2ecb644060e84ef5b32b4fdd470200f6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:25.436210   713 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:25.436551   639 raft_consensus.cc:3055] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:25.436883   639 raft_consensus.cc:740] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:25.435817   463 catalog_manager.cc:5697] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 3 to 4, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 4 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:25.437640   639 consensus_queue.cc:260] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.438799   639 raft_consensus.cc:3060] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:25.442888   713 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 3.
I20260709 13:00:25.444126   529 leader_election.cc:304] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:25.444870   944 raft_consensus.cc:2804] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:25.445307   944 raft_consensus.cc:697] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Becoming Leader. State: Replica: 2ecb644060e84ef5b32b4fdd470200f6, State: Running, Role: LEADER
I20260709 13:00:25.445518   639 raft_consensus.cc:2468] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ecb644060e84ef5b32b4fdd470200f6 in term 3.
I20260709 13:00:25.446273   944 consensus_queue.cc:237] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.454879   463 catalog_manager.cc:5697] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 reported cstate change: term changed from 2 to 3, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "2ecb644060e84ef5b32b4fdd470200f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:25.480731   950 raft_consensus.cc:993] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: : Instructing follower 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:25.481212   950 raft_consensus.cc:1081] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Signalling peer 449207e0c86e420ab6e1ed5d5f78702e to start an election
I20260709 13:00:25.482947   713 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776"
dest_uuid: "449207e0c86e420ab6e1ed5d5f78702e"
 from {username='slave'} at 127.0.0.1:59672
I20260709 13:00:25.483528   713 raft_consensus.cc:493] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:25.483891   713 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.488727   713 raft_consensus.cc:515] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.490449   713 leader_election.cc:290] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 5 election: Requested vote from peers 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613), 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251)
I20260709 13:00:25.491526   564 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2ecb644060e84ef5b32b4fdd470200f6"
I20260709 13:00:25.492053   564 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.491572   639 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93d0bfc309764e4b9a4dd81be0180776" candidate_uuid: "449207e0c86e420ab6e1ed5d5f78702e" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9fe07b221f304ed1ac4a53ba51f6285b"
I20260709 13:00:25.492691   639 raft_consensus.cc:3055] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:25.492955   639 raft_consensus.cc:740] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9fe07b221f304ed1ac4a53ba51f6285b, State: Running, Role: LEADER
I20260709 13:00:25.493763   639 consensus_queue.cc:260] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.494740   639 raft_consensus.cc:3060] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:25.497232   564 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 5.
I20260709 13:00:25.498590   679 leader_election.cc:304] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ecb644060e84ef5b32b4fdd470200f6, 449207e0c86e420ab6e1ed5d5f78702e; no voters: 
I20260709 13:00:25.499420   890 raft_consensus.cc:2804] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:25.499943   890 raft_consensus.cc:697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 LEADER]: Becoming Leader. State: Replica: 449207e0c86e420ab6e1ed5d5f78702e, State: Running, Role: LEADER
I20260709 13:00:25.501169   639 raft_consensus.cc:2468] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 449207e0c86e420ab6e1ed5d5f78702e in term 5.
I20260709 13:00:25.500794   890 consensus_queue.cc:237] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } }
I20260709 13:00:25.508498   463 catalog_manager.cc:5697] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e reported cstate change: term changed from 4 to 5, leader changed from 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194) to 449207e0c86e420ab6e1ed5d5f78702e (127.0.107.195). New cstate: current_term: 5 leader_uuid: "449207e0c86e420ab6e1ed5d5f78702e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:25.712803   639 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:25.714134   944 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:25.727630   713 raft_consensus.cc:1275] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:00:25.730129   944 consensus_queue.cc:1048] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:25.838192   564 raft_consensus.cc:1275] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:25.839977   890 consensus_queue.cc:1048] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:25.856956   639 raft_consensus.cc:1275] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:25.857935   713 raft_consensus.cc:1275] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:25.858593   962 consensus_queue.cc:1048] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:25.859959   957 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:25.873749   639 raft_consensus.cc:1275] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:25.875703   944 consensus_queue.cc:1048] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:25.894843   639 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:25.896616   962 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:25.908592   564 raft_consensus.cc:1275] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Refusing update from remote peer 449207e0c86e420ab6e1ed5d5f78702e: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:25.910298   890 consensus_queue.cc:1048] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ecb644060e84ef5b32b4fdd470200f6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:00:25.934177   713 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:25.935961   945 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "449207e0c86e420ab6e1ed5d5f78702e" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:25.949106   639 raft_consensus.cc:1275] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Refusing update from remote peer 2ecb644060e84ef5b32b4fdd470200f6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:25.951077   969 consensus_queue.cc:1048] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe07b221f304ed1ac4a53ba51f6285b" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 34251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:26.161092   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:26.167882   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 13:00:26.196494   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:26.197393   431 tablet_replica.cc:333] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.198534   431 raft_consensus.cc:2243] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.199229   431 raft_consensus.cc:2272] T d8f0614c8e594993817349c8bbe75a65 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.202143   431 tablet_replica.cc:333] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.202644   431 raft_consensus.cc:2243] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.203027   431 raft_consensus.cc:2272] T cfb29ed2c45c42f6ac9794da45f1d54d P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.204860   431 tablet_replica.cc:333] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.205642   431 raft_consensus.cc:2243] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.206043   431 raft_consensus.cc:2272] T c5b47257091443409545dfd5a0169002 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.208040   431 tablet_replica.cc:333] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.208600   431 raft_consensus.cc:2243] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.209180   431 raft_consensus.cc:2272] T fff0e3bb765e4a60bbc4505595a11923 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.211910   431 tablet_replica.cc:333] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.212605   431 raft_consensus.cc:2243] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.213709   431 raft_consensus.cc:2272] T 5da6944d7e7540178df4276919d61a50 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.216591   431 tablet_replica.cc:333] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.217248   431 raft_consensus.cc:2243] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.218129   431 raft_consensus.cc:2272] T 2d4e88ab70364fabb16659caabfc6bb7 P 2ecb644060e84ef5b32b4fdd470200f6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.222689   431 tablet_replica.cc:333] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.223430   431 raft_consensus.cc:2243] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.223951   431 raft_consensus.cc:2272] T 93d0bfc309764e4b9a4dd81be0180776 P 2ecb644060e84ef5b32b4fdd470200f6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.227643   431 tablet_replica.cc:333] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.228190   431 raft_consensus.cc:2243] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.228914   431 raft_consensus.cc:2272] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 2ecb644060e84ef5b32b4fdd470200f6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.231326   431 tablet_replica.cc:333] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6: stopping tablet replica
I20260709 13:00:26.231823   431 raft_consensus.cc:2243] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.232281   431 raft_consensus.cc:2272] T f8f40bec15324be498bd1d7ceea30a33 P 2ecb644060e84ef5b32b4fdd470200f6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.359930   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
I20260709 13:00:26.382391   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:00:26.403831   679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:36613: connect: Connection refused (error 111)
I20260709 13:00:26.418267   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:26.419194   431 tablet_replica.cc:333] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
W20260709 13:00:26.418457   679 consensus_peers.cc:597] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e -> Peer 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613): Couldn't send request to peer 2ecb644060e84ef5b32b4fdd470200f6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:36613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:26.420223   431 raft_consensus.cc:2243] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.421325   431 raft_consensus.cc:2272] T f8f40bec15324be498bd1d7ceea30a33 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:00:26.422971   681 consensus_peers.cc:597] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e -> Peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Couldn't send request to peer 9fe07b221f304ed1ac4a53ba51f6285b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:26.423938   431 tablet_replica.cc:333] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.424464   431 raft_consensus.cc:2243] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.425194   431 raft_consensus.cc:2272] T fff0e3bb765e4a60bbc4505595a11923 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.427101   431 tablet_replica.cc:333] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.427660   431 raft_consensus.cc:2243] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.428094   431 raft_consensus.cc:2272] T 5da6944d7e7540178df4276919d61a50 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.430402   431 tablet_replica.cc:333] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.430878   431 raft_consensus.cc:2243] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.431222   431 raft_consensus.cc:2272] T 2d4e88ab70364fabb16659caabfc6bb7 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.433188   431 tablet_replica.cc:333] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.433777   431 raft_consensus.cc:2243] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.434141   431 raft_consensus.cc:2272] T d8f0614c8e594993817349c8bbe75a65 P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.436643   431 tablet_replica.cc:333] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.437117   431 raft_consensus.cc:2243] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.437471   431 raft_consensus.cc:2272] T cfb29ed2c45c42f6ac9794da45f1d54d P 9fe07b221f304ed1ac4a53ba51f6285b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.439710   431 tablet_replica.cc:333] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.440487   431 raft_consensus.cc:2243] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.443200   431 raft_consensus.cc:2272] T c5b47257091443409545dfd5a0169002 P 9fe07b221f304ed1ac4a53ba51f6285b [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.445842   431 tablet_replica.cc:333] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
W20260709 13:00:26.445879   681 consensus_peers.cc:597] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e -> Peer 9fe07b221f304ed1ac4a53ba51f6285b (127.0.107.194:34251): Couldn't send request to peer 9fe07b221f304ed1ac4a53ba51f6285b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:26.446679   431 raft_consensus.cc:2243] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.447461   431 raft_consensus.cc:2272] T 93d0bfc309764e4b9a4dd81be0180776 P 9fe07b221f304ed1ac4a53ba51f6285b [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.450138   431 tablet_replica.cc:333] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b: stopping tablet replica
I20260709 13:00:26.450678   431 raft_consensus.cc:2243] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.451115   431 raft_consensus.cc:2272] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 9fe07b221f304ed1ac4a53ba51f6285b [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.481110   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 13:00:26.502961   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 13:00:26.507324   679 consensus_peers.cc:597] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e -> Peer 2ecb644060e84ef5b32b4fdd470200f6 (127.0.107.193:36613): Couldn't send request to peer 2ecb644060e84ef5b32b4fdd470200f6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:36613: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:26.551968   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:26.553148   431 tablet_replica.cc:333] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.553896   431 raft_consensus.cc:2243] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.554811   431 raft_consensus.cc:2272] T cfb29ed2c45c42f6ac9794da45f1d54d P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.556883   431 tablet_replica.cc:333] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.557432   431 raft_consensus.cc:2243] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.557960   431 raft_consensus.cc:2272] T c5b47257091443409545dfd5a0169002 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.559790   431 tablet_replica.cc:333] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.560375   431 raft_consensus.cc:2243] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.561197   431 raft_consensus.cc:2272] T d8f0614c8e594993817349c8bbe75a65 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.563701   431 tablet_replica.cc:333] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.564294   431 raft_consensus.cc:2243] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.564854   431 raft_consensus.cc:2272] T 2d4e88ab70364fabb16659caabfc6bb7 P 449207e0c86e420ab6e1ed5d5f78702e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.568132   431 tablet_replica.cc:333] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.568871   431 raft_consensus.cc:2243] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.569286   431 raft_consensus.cc:2272] T fff0e3bb765e4a60bbc4505595a11923 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.571135   431 tablet_replica.cc:333] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.571610   431 raft_consensus.cc:2243] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.571997   431 raft_consensus.cc:2272] T 5da6944d7e7540178df4276919d61a50 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.574484   431 tablet_replica.cc:333] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.574955   431 raft_consensus.cc:2243] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.575708   431 raft_consensus.cc:2272] T 93d0bfc309764e4b9a4dd81be0180776 P 449207e0c86e420ab6e1ed5d5f78702e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.578084   431 tablet_replica.cc:333] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.578579   431 raft_consensus.cc:2243] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.578994   431 raft_consensus.cc:2272] T 2b2b40f1cf5a40b8b736f70b16a2a19b P 449207e0c86e420ab6e1ed5d5f78702e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.581337   431 tablet_replica.cc:333] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e: stopping tablet replica
I20260709 13:00:26.581979   431 raft_consensus.cc:2243] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:26.582455   431 raft_consensus.cc:2272] T f8f40bec15324be498bd1d7ceea30a33 P 449207e0c86e420ab6e1ed5d5f78702e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.613029   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:00:26.634629   431 master.cc:562] Master@127.0.107.254:44365 shutting down...
I20260709 13:00:26.783350   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:26.784010   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:26.784303   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P e52e29d1345340f3a1b57c3c7a3b84c1: stopping tablet replica
I20260709 13:00:26.805752   431 master.cc:584] Master@127.0.107.254:44365 shutdown complete.
2ecb644060e84ef5b32b4fdd470200f6  4
449207e0c86e420ab6e1ed5d5f78702e  1
9fe07b221f304ed1ac4a53ba51f6285b  4
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (16158 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 13:00:26.857649   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:36433
I20260709 13:00:26.858883   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:26.866324   975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:26.866324   976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:26.867373   978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:26.869477   431 server_base.cc:1061] running on GCE node
I20260709 13:00:26.870935   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:26.871124   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:26.871251   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602026871240 us; error 0 us; skew 500 ppm
I20260709 13:00:26.871749   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:26.875221   431 webserver.cc:533] Webserver started at http://127.0.107.254:40997/ using document root <none> and password file <none>
I20260709 13:00:26.875769   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:26.875946   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:26.876191   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:26.877394   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "b123de53400e4f4bb5b5f63b3fc9e465"
format_stamp: "Formatted at 2026-07-09 13:00:26 on dist-test-slave-p8rg"
I20260709 13:00:26.883816   431 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:00:26.888377   983 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:26.889307   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 13:00:26.889848   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "b123de53400e4f4bb5b5f63b3fc9e465"
format_stamp: "Formatted at 2026-07-09 13:00:26 on dist-test-slave-p8rg"
I20260709 13:00:26.890143   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:26.906823   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:26.908188   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:26.958431   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:36433
I20260709 13:00:26.958573  1034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:36433 every 8 connection(s)
I20260709 13:00:26.963094  1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:26.975757  1035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465: Bootstrap starting.
I20260709 13:00:26.982442  1035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:26.988433  1035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465: No bootstrap required, opened a new log
I20260709 13:00:26.991348  1035 raft_consensus.cc:359] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER }
I20260709 13:00:26.991927  1035 raft_consensus.cc:385] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:26.992236  1035 raft_consensus.cc:740] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b123de53400e4f4bb5b5f63b3fc9e465, State: Initialized, Role: FOLLOWER
I20260709 13:00:26.993041  1035 consensus_queue.cc:260] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER }
I20260709 13:00:26.993793  1035 raft_consensus.cc:399] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:26.994107  1035 raft_consensus.cc:493] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:26.994351  1035 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:26.999620  1035 raft_consensus.cc:515] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER }
I20260709 13:00:27.000339  1035 leader_election.cc:304] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b123de53400e4f4bb5b5f63b3fc9e465; no voters: 
I20260709 13:00:27.001976  1035 leader_election.cc:290] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:27.002233  1038 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:27.003795  1038 raft_consensus.cc:697] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 1 LEADER]: Becoming Leader. State: Replica: b123de53400e4f4bb5b5f63b3fc9e465, State: Running, Role: LEADER
I20260709 13:00:27.004676  1038 consensus_queue.cc:237] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER }
I20260709 13:00:27.005971  1035 sys_catalog.cc:565] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:27.010411  1040 sys_catalog.cc:455] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b123de53400e4f4bb5b5f63b3fc9e465. Latest consensus state: current_term: 1 leader_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER } }
I20260709 13:00:27.010437  1039 sys_catalog.cc:455] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b123de53400e4f4bb5b5f63b3fc9e465" member_type: VOTER } }
I20260709 13:00:27.011602  1040 sys_catalog.cc:458] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:27.011999  1039 sys_catalog.cc:458] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:27.018711  1044 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:27.029132  1044 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:27.031659   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:27.040437  1044 catalog_manager.cc:1383] Generated new cluster ID: cb10d13d8f4a43bf82c66e406e39d2ed
I20260709 13:00:27.040849  1044 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:27.058670  1044 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:27.060133  1044 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:27.076129  1044 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465: Generated new TSK 0
I20260709 13:00:27.076897  1044 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:27.099709   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:27.107131  1056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:27.108074  1057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:27.110303   431 server_base.cc:1061] running on GCE node
W20260709 13:00:27.110389  1059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:27.111490   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:27.111688   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:27.111825   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602027111813 us; error 0 us; skew 500 ppm
I20260709 13:00:27.112311   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:27.115285   431 webserver.cc:533] Webserver started at http://127.0.107.193:43317/ using document root <none> and password file <none>
I20260709 13:00:27.115881   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:27.116080   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:27.116315   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:27.117472   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.123898   431 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:00:27.128391  1064 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.129314   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:00:27.129971   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.130446   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:27.160060   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:27.161788   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:27.163877   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:27.166548   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:27.166812   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.167111   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:27.167330   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.217720   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:38045
I20260709 13:00:27.217815  1126 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:38045 every 8 connection(s)
I20260709 13:00:27.223070   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:27.232981  1131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:27.232188  1132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:27.234807  1134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:27.238404   431 server_base.cc:1061] running on GCE node
I20260709 13:00:27.239822   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:00:27.239881  1127 heartbeater.cc:344] Connected to a master server at 127.0.107.254:36433
W20260709 13:00:27.240128   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:27.240391   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602027240372 us; error 0 us; skew 500 ppm
I20260709 13:00:27.240420  1127 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:27.241044   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:27.241417  1127 heartbeater.cc:507] Master 127.0.107.254:36433 requested a full tablet report, sending...
I20260709 13:00:27.244570   431 webserver.cc:533] Webserver started at http://127.0.107.194:35723/ using document root <none> and password file <none>
I20260709 13:00:27.244699  1000 ts_manager.cc:194] Registered new tserver with Master: eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045)
I20260709 13:00:27.245657   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:27.246168   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:27.246587   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:27.247835  1000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47528
I20260709 13:00:27.248363   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "e775baeb4df04f29a16933f47713bdab"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.254626   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.004s
I20260709 13:00:27.258771  1139 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.259915   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:00:27.260296   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "e775baeb4df04f29a16933f47713bdab"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.260638   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:27.274062   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:27.275693   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:27.278115   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:27.280891   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:27.281543   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.001s	user 0.000s	sys 0.000s
I20260709 13:00:27.281862   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:27.282102   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.333798   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:40161
I20260709 13:00:27.333891  1201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:40161 every 8 connection(s)
I20260709 13:00:27.339823   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:27.350639  1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:27.350853  1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:27.354176  1202 heartbeater.cc:344] Connected to a master server at 127.0.107.254:36433
I20260709 13:00:27.355222  1202 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:27.356137   431 server_base.cc:1061] running on GCE node
W20260709 13:00:27.356171  1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:27.356616  1202 heartbeater.cc:507] Master 127.0.107.254:36433 requested a full tablet report, sending...
I20260709 13:00:27.357620   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:27.357879   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:27.358300   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602027358253 us; error 0 us; skew 500 ppm
I20260709 13:00:27.359297   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:27.359764  1000 ts_manager.cc:194] Registered new tserver with Master: e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:27.362603  1000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47542
I20260709 13:00:27.363965   431 webserver.cc:533] Webserver started at http://127.0.107.195:32979/ using document root <none> and password file <none>
I20260709 13:00:27.364632   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:27.364890   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:27.365219   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:27.366706   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "d98a73331940438fa4d42f653a45a71d"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.372092   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20260709 13:00:27.376076  1213 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.377039   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:00:27.377310   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "d98a73331940438fa4d42f653a45a71d"
format_stamp: "Formatted at 2026-07-09 13:00:27 on dist-test-slave-p8rg"
I20260709 13:00:27.377652   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:27.395619   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:27.397099   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:27.398849   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:27.401741   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:27.401968   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.402169   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:27.402329   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:27.453967   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:39403
I20260709 13:00:27.454182  1275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:39403 every 8 connection(s)
I20260709 13:00:27.470983  1276 heartbeater.cc:344] Connected to a master server at 127.0.107.254:36433
I20260709 13:00:27.471400  1276 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:27.472308  1276 heartbeater.cc:507] Master 127.0.107.254:36433 requested a full tablet report, sending...
I20260709 13:00:27.474370  1000 ts_manager.cc:194] Registered new tserver with Master: d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:27.474642   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016952978s
I20260709 13:00:27.475823  1000 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47548
I20260709 13:00:27.508118  1000 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47562:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:27.512579  1000 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:27.612213  1167 tablet_service.cc:1511] Processing CreateTablet for tablet 561aa5db68244ddab8f87ed38b2295e5 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:27.614282  1167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561aa5db68244ddab8f87ed38b2295e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.619333  1241 tablet_service.cc:1511] Processing CreateTablet for tablet 561aa5db68244ddab8f87ed38b2295e5 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:27.620985  1241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561aa5db68244ddab8f87ed38b2295e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.622818  1240 tablet_service.cc:1511] Processing CreateTablet for tablet 2c5b67a6aeb447c6bb97c157612cc3b2 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:27.624135  1164 tablet_service.cc:1511] Processing CreateTablet for tablet 777449d6216b4cec8aa37d01aeba7885 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:27.624665  1240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b67a6aeb447c6bb97c157612cc3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.625694  1164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777449d6216b4cec8aa37d01aeba7885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.626781  1239 tablet_service.cc:1511] Processing CreateTablet for tablet 8159458859ef47cfb00c03a3b353fa35 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:27.628295  1239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8159458859ef47cfb00c03a3b353fa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.631071  1166 tablet_service.cc:1511] Processing CreateTablet for tablet 2c5b67a6aeb447c6bb97c157612cc3b2 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:27.634805  1165 tablet_service.cc:1511] Processing CreateTablet for tablet 8159458859ef47cfb00c03a3b353fa35 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:27.632917  1166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b67a6aeb447c6bb97c157612cc3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.640484  1163 tablet_service.cc:1511] Processing CreateTablet for tablet 82c91326fc824d37a7b3bd5a02edcd6f (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:27.642194  1163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c91326fc824d37a7b3bd5a02edcd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.645572  1237 tablet_service.cc:1511] Processing CreateTablet for tablet 82c91326fc824d37a7b3bd5a02edcd6f (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:27.647011  1237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c91326fc824d37a7b3bd5a02edcd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.643107  1238 tablet_service.cc:1511] Processing CreateTablet for tablet 777449d6216b4cec8aa37d01aeba7885 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:27.649277  1238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777449d6216b4cec8aa37d01aeba7885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.654009  1165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8159458859ef47cfb00c03a3b353fa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.656329  1162 tablet_service.cc:1511] Processing CreateTablet for tablet 98f0ba6abb364f75b64408020f47fe61 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:27.658139  1162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f0ba6abb364f75b64408020f47fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.659219  1164 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5b866f5c7340c1bc5e6735a0be5189 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:27.660849  1164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5b866f5c7340c1bc5e6735a0be5189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.663528  1236 tablet_service.cc:1511] Processing CreateTablet for tablet 98f0ba6abb364f75b64408020f47fe61 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:27.665361  1236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f0ba6abb364f75b64408020f47fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.667270  1235 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5b866f5c7340c1bc5e6735a0be5189 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:27.668700  1235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5b866f5c7340c1bc5e6735a0be5189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.668866  1161 tablet_service.cc:1511] Processing CreateTablet for tablet 81e8fe3749e54d4eb6630dd76cd47146 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:27.670818  1161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e8fe3749e54d4eb6630dd76cd47146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.670708  1234 tablet_service.cc:1511] Processing CreateTablet for tablet 81e8fe3749e54d4eb6630dd76cd47146 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:27.672128  1234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e8fe3749e54d4eb6630dd76cd47146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.675416  1233 tablet_service.cc:1511] Processing CreateTablet for tablet 339a602e06ff42d28ca3f7cf5b97603b (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:27.677145  1233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 339a602e06ff42d28ca3f7cf5b97603b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.679139  1232 tablet_service.cc:1511] Processing CreateTablet for tablet 04832b9c7dd546bbb9c58655e83d7543 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:27.682400  1160 tablet_service.cc:1511] Processing CreateTablet for tablet 339a602e06ff42d28ca3f7cf5b97603b (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:27.683863  1160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 339a602e06ff42d28ca3f7cf5b97603b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.689954  1232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04832b9c7dd546bbb9c58655e83d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.693043  1159 tablet_service.cc:1511] Processing CreateTablet for tablet 04832b9c7dd546bbb9c58655e83d7543 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:27.694726  1159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04832b9c7dd546bbb9c58655e83d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.715354  1092 tablet_service.cc:1511] Processing CreateTablet for tablet 561aa5db68244ddab8f87ed38b2295e5 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:00:27.716861  1092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 561aa5db68244ddab8f87ed38b2295e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.721161  1091 tablet_service.cc:1511] Processing CreateTablet for tablet 2c5b67a6aeb447c6bb97c157612cc3b2 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:00:27.722627  1091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c5b67a6aeb447c6bb97c157612cc3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.724829  1090 tablet_service.cc:1511] Processing CreateTablet for tablet 8159458859ef47cfb00c03a3b353fa35 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:00:27.727267  1089 tablet_service.cc:1511] Processing CreateTablet for tablet 777449d6216b4cec8aa37d01aeba7885 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:00:27.735476  1291 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:27.739013  1088 tablet_service.cc:1511] Processing CreateTablet for tablet 82c91326fc824d37a7b3bd5a02edcd6f (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:00:27.741567  1090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8159458859ef47cfb00c03a3b353fa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.751262  1292 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:27.754231  1084 tablet_service.cc:1511] Processing CreateTablet for tablet 339a602e06ff42d28ca3f7cf5b97603b (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:00:27.758632  1085 tablet_service.cc:1511] Processing CreateTablet for tablet 81e8fe3749e54d4eb6630dd76cd47146 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:00:27.760795  1087 tablet_service.cc:1511] Processing CreateTablet for tablet 98f0ba6abb364f75b64408020f47fe61 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:00:27.761814  1089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777449d6216b4cec8aa37d01aeba7885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.766911  1086 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5b866f5c7340c1bc5e6735a0be5189 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:00:27.773392  1083 tablet_service.cc:1511] Processing CreateTablet for tablet 04832b9c7dd546bbb9c58655e83d7543 (DEFAULT_TABLE table=test-workload [id=bc34738380c34b0cbb0e5efe0a1a4f23]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:00:27.775760  1085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81e8fe3749e54d4eb6630dd76cd47146. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.777474  1088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c91326fc824d37a7b3bd5a02edcd6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.779129  1087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98f0ba6abb364f75b64408020f47fe61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.780107  1084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 339a602e06ff42d28ca3f7cf5b97603b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.785161  1086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5b866f5c7340c1bc5e6735a0be5189. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.795768  1083 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04832b9c7dd546bbb9c58655e83d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:27.824532  1292 tablet_bootstrap.cc:654] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.833922  1291 tablet_bootstrap.cc:654] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.854974  1293 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:27.866777  1291 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:27.867385  1291 ts_tablet_manager.cc:1403] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.132s	user 0.007s	sys 0.011s
I20260709 13:00:27.871240  1291 raft_consensus.cc:359] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.873219  1291 raft_consensus.cc:385] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.873626  1291 raft_consensus.cc:740] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.874522  1291 consensus_queue.cc:260] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.876937  1202 heartbeater.cc:499] Master 127.0.107.254:36433 was elected leader, sending a full tablet report...
I20260709 13:00:27.877897  1291 ts_tablet_manager.cc:1434] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.004s
I20260709 13:00:27.879473  1291 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:27.885349  1291 tablet_bootstrap.cc:654] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.885455  1293 tablet_bootstrap.cc:654] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.889007  1292 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:27.889672  1292 ts_tablet_manager.cc:1403] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.139s	user 0.015s	sys 0.004s
I20260709 13:00:27.901187  1292 raft_consensus.cc:359] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.902328  1292 raft_consensus.cc:385] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.902750  1292 raft_consensus.cc:740] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.903491  1292 consensus_queue.cc:260] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.905646  1291 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:27.906322  1276 heartbeater.cc:499] Master 127.0.107.254:36433 was elected leader, sending a full tablet report...
I20260709 13:00:27.906620  1292 ts_tablet_manager.cc:1434] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.001s
I20260709 13:00:27.907516  1293 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:27.907572  1292 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:27.908155  1293 ts_tablet_manager.cc:1403] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.054s	user 0.015s	sys 0.011s
I20260709 13:00:27.908545  1291 ts_tablet_manager.cc:1403] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.008s
I20260709 13:00:27.911820  1293 raft_consensus.cc:359] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.912901  1293 raft_consensus.cc:385] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.914580  1291 raft_consensus.cc:359] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.915318  1291 raft_consensus.cc:385] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.915634  1291 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.913239  1293 raft_consensus.cc:740] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.915877  1292 tablet_bootstrap.cc:654] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.916785  1293 consensus_queue.cc:260] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.918946  1291 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.922668  1291 ts_tablet_manager.cc:1434] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 13:00:27.924129  1291 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:27.932340  1127 heartbeater.cc:499] Master 127.0.107.254:36433 was elected leader, sending a full tablet report...
I20260709 13:00:27.933409  1291 tablet_bootstrap.cc:654] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.935277  1292 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:27.935873  1292 ts_tablet_manager.cc:1403] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.013s
I20260709 13:00:27.940002  1292 raft_consensus.cc:359] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.932322  1293 ts_tablet_manager.cc:1434] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.004s
I20260709 13:00:27.941028  1292 raft_consensus.cc:385] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.941365  1292 raft_consensus.cc:740] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.942113  1293 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:27.942210  1292 consensus_queue.cc:260] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.943543  1291 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:27.944089  1291 ts_tablet_manager.cc:1403] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20260709 13:00:27.944927  1292 ts_tablet_manager.cc:1434] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.004s
I20260709 13:00:27.946403  1292 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:27.948179  1291 raft_consensus.cc:359] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.949172  1291 raft_consensus.cc:385] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.949570  1291 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.950331  1293 tablet_bootstrap.cc:654] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.950506  1291 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.953197  1291 ts_tablet_manager.cc:1434] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:27.954198  1292 tablet_bootstrap.cc:654] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.954387  1291 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:27.964190  1291 tablet_bootstrap.cc:654] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.965682  1293 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:27.966239  1293 ts_tablet_manager.cc:1403] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.004s
I20260709 13:00:27.967252  1292 tablet_bootstrap.cc:492] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:27.967754  1292 ts_tablet_manager.cc:1403] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.007s
I20260709 13:00:27.969666  1293 raft_consensus.cc:359] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.970554  1293 raft_consensus.cc:385] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.970856  1293 raft_consensus.cc:740] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.971151  1292 raft_consensus.cc:359] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.971855  1293 consensus_queue.cc:260] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.971836  1292 raft_consensus.cc:385] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.973111  1292 raft_consensus.cc:740] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.974251  1292 consensus_queue.cc:260] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.975524  1293 ts_tablet_manager.cc:1434] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.002s
I20260709 13:00:27.977394  1293 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:27.977537  1292 ts_tablet_manager.cc:1434] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:27.978930  1292 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:27.983538  1291 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:27.984205  1291 ts_tablet_manager.cc:1403] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.008s
I20260709 13:00:27.986567  1292 tablet_bootstrap.cc:654] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:27.987579  1291 raft_consensus.cc:359] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.988555  1291 raft_consensus.cc:385] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:27.989213  1291 raft_consensus.cc:740] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:27.990104  1291 consensus_queue.cc:260] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:27.993726  1291 ts_tablet_manager.cc:1434] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.009s	user 0.012s	sys 0.000s
I20260709 13:00:27.994796  1291 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.001269  1293 tablet_bootstrap.cc:654] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.002384  1296 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.003005  1296 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.005067  1291 tablet_bootstrap.cc:654] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.008975  1292 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.010329  1292 ts_tablet_manager.cc:1403] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.006s
I20260709 13:00:28.014710  1292 raft_consensus.cc:359] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.015952  1292 raft_consensus.cc:385] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.016553  1292 raft_consensus.cc:740] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.017421  1292 consensus_queue.cc:260] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.020114  1292 ts_tablet_manager.cc:1434] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:28.021765  1292 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
W20260709 13:00:28.025200  1048 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:28.027658  1291 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:28.028451  1291 ts_tablet_manager.cc:1403] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.034s	user 0.015s	sys 0.008s
I20260709 13:00:28.030169  1292 tablet_bootstrap.cc:654] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.041229  1296 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.045056  1291 raft_consensus.cc:359] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.046059  1291 raft_consensus.cc:385] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.046520  1291 raft_consensus.cc:740] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.047663  1291 consensus_queue.cc:260] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.050724  1293 tablet_bootstrap.cc:492] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.051569  1293 ts_tablet_manager.cc:1403] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.074s	user 0.017s	sys 0.038s
I20260709 13:00:28.051429  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
W20260709 13:00:28.053476  1142 leader_election.cc:343] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.058030  1291 ts_tablet_manager.cc:1434] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.001s
I20260709 13:00:28.056721  1293 raft_consensus.cc:359] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.064437  1293 raft_consensus.cc:385] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.064733  1291 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.064898  1293 raft_consensus.cc:740] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.066093  1293 consensus_queue.cc:260] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.071170  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
W20260709 13:00:28.075933  1141 leader_election.cc:343] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.076400  1292 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.076458  1141 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e775baeb4df04f29a16933f47713bdab; no voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.077311  1292 ts_tablet_manager.cc:1403] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.056s	user 0.019s	sys 0.023s
I20260709 13:00:28.078084  1293 ts_tablet_manager.cc:1434] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.008s
I20260709 13:00:28.078658  1296 raft_consensus.cc:2749] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.079437  1291 tablet_bootstrap.cc:654] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.079522  1293 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.082322  1292 raft_consensus.cc:359] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.083510  1292 raft_consensus.cc:385] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.083843  1292 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.084666  1292 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.087522  1292 ts_tablet_manager.cc:1434] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:00:28.089648  1292 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:28.092226  1293 tablet_bootstrap.cc:654] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.098099  1291 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:28.098886  1291 ts_tablet_manager.cc:1403] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.003s
I20260709 13:00:28.103670  1291 raft_consensus.cc:359] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.104804  1291 raft_consensus.cc:385] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.105695  1291 raft_consensus.cc:740] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.107939  1291 consensus_queue.cc:260] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.109427  1292 tablet_bootstrap.cc:654] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.111821  1291 ts_tablet_manager.cc:1434] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.013s	user 0.011s	sys 0.001s
I20260709 13:00:28.112914  1299 raft_consensus.cc:493] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.113633  1299 raft_consensus.cc:515] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.114745  1300 raft_consensus.cc:493] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.115272  1300 raft_consensus.cc:515] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.119544  1299 leader_election.cc:290] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.123204  1300 leader_election.cc:290] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.131305  1291 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.136641  1299 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.137408  1299 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.140585  1299 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.143489  1299 raft_consensus.cc:493] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.144074  1299 raft_consensus.cc:515] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.147671  1291 tablet_bootstrap.cc:654] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.157157  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.158154  1251 raft_consensus.cc:2468] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.159104  1293 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.159626  1293 ts_tablet_manager.cc:1403] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.080s	user 0.029s	sys 0.020s
I20260709 13:00:28.159848  1292 tablet_bootstrap.cc:492] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.160470  1066 leader_election.cc:304] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.161231  1296 raft_consensus.cc:493] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.161568  1300 raft_consensus.cc:2804] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.161974  1300 raft_consensus.cc:493] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.161751  1296 raft_consensus.cc:515] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.162348  1300 raft_consensus.cc:3060] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.160440  1292 ts_tablet_manager.cc:1403] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.014s
I20260709 13:00:28.162981  1299 leader_election.cc:290] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.164055  1296 leader_election.cc:290] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.166234  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.166266  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.166580  1292 raft_consensus.cc:359] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.166976  1102 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.167063  1251 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.167296  1292 raft_consensus.cc:385] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.167654  1292 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.168321  1142 leader_election.cc:304] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.168466  1292 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.169580  1296 raft_consensus.cc:2804] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.169974  1296 raft_consensus.cc:493] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.170334  1296 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.171447  1292 ts_tablet_manager.cc:1434] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.003s
I20260709 13:00:28.172540  1292 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:28.175344  1300 raft_consensus.cc:515] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.178169  1300 leader_election.cc:290] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.180003  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.181890  1251 raft_consensus.cc:3060] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.181770  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.182611  1176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.183470  1176 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 0.
I20260709 13:00:28.182891  1177 raft_consensus.cc:2468] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 0.
I20260709 13:00:28.185070  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.187294  1216 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.191753  1251 raft_consensus.cc:2468] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 1.
I20260709 13:00:28.193852  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.200309  1216 leader_election.cc:304] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.200939  1175 raft_consensus.cc:2468] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.200215  1296 raft_consensus.cc:515] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.198369  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.199303  1291 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
W20260709 13:00:28.201367  1216 leader_election.cc:343] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.202226  1291 ts_tablet_manager.cc:1403] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.071s	user 0.023s	sys 0.008s
I20260709 13:00:28.202628  1066 leader_election.cc:304] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.196096  1293 raft_consensus.cc:359] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.203433  1293 raft_consensus.cc:385] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.203560  1300 raft_consensus.cc:2804] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.205669  1293 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.206045  1322 raft_consensus.cc:2764] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:00:28.206625  1293 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.207850  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.205672  1291 raft_consensus.cc:359] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.200121  1299 raft_consensus.cc:2804] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.202091  1177 raft_consensus.cc:3060] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.208483  1102 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.209627  1292 tablet_bootstrap.cc:654] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.209029  1299 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.218427  1299 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.221727  1291 raft_consensus.cc:385] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.222218  1291 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.225666  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.226366  1251 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.228530  1102 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.229943  1142 leader_election.cc:304] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.230495  1299 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.232007  1291 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.233042  1101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.233073  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.234891  1299 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
W20260709 13:00:28.235761  1216 leader_election.cc:343] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:28.237052  1216 leader_election.cc:343] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.237677  1216 leader_election.cc:304] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d98a73331940438fa4d42f653a45a71d; no voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.238358  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8159458859ef47cfb00c03a3b353fa35" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.240061  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
W20260709 13:00:28.242349  1216 leader_election.cc:343] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.243355  1299 raft_consensus.cc:2749] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.246276  1327 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.247586  1327 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.252782  1327 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.244745  1296 leader_election.cc:290] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.249646  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.246860  1326 raft_consensus.cc:2804] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.258594  1175 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.253815  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.261947  1100 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.255244  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.263358  1250 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.264737  1142 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.267841  1328 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.268507  1328 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.269040  1328 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.256871  1300 raft_consensus.cc:697] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:28.255966  1291 ts_tablet_manager.cc:1434] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.053s	user 0.012s	sys 0.006s
I20260709 13:00:28.258648  1326 raft_consensus.cc:697] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:28.270352  1300 consensus_queue.cc:237] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.269827  1296 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.274109  1296 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.278975  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.281028  1251 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
W20260709 13:00:28.281224  1142 leader_election.cc:343] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.284013  1326 consensus_queue.cc:237] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.285914  1177 raft_consensus.cc:2468] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 1.
I20260709 13:00:28.287003  1291 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.291857  1293 ts_tablet_manager.cc:1434] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.132s	user 0.027s	sys 0.051s
I20260709 13:00:28.293268  1293 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.297401  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
W20260709 13:00:28.299017  1141 leader_election.cc:343] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.299458  1296 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.289243  1323 raft_consensus.cc:493] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.299849  1141 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e775baeb4df04f29a16933f47713bdab; no voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.295441  1175 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 1.
I20260709 13:00:28.302973  1296 raft_consensus.cc:2749] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.304114  1216 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.306481  1293 tablet_bootstrap.cc:654] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.306936  1299 raft_consensus.cc:2804] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.307570  1299 raft_consensus.cc:697] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:28.308480  1291 tablet_bootstrap.cc:654] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.308629  1299 consensus_queue.cc:237] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.302004  1323 raft_consensus.cc:515] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.321108  1322 raft_consensus.cc:493] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.321240  1323 leader_election.cc:290] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.321823  1322 raft_consensus.cc:515] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.322665  1292 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.323213  1292 ts_tablet_manager.cc:1403] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.151s	user 0.020s	sys 0.038s
I20260709 13:00:28.327076  1322 leader_election.cc:290] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.329227  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.330303  1100 raft_consensus.cc:2468] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 0.
I20260709 13:00:28.331251  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.332713  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
W20260709 13:00:28.332857  1216 leader_election.cc:343] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.333163  1292 raft_consensus.cc:359] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.334213  1292 raft_consensus.cc:385] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.334676  1292 raft_consensus.cc:740] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.334789  1216 leader_election.cc:304] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: e775baeb4df04f29a16933f47713bdab
I20260709 13:00:28.335800  1292 consensus_queue.cc:260] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.336472   998 catalog_manager.cc:5697] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 0 to 1, leader changed from <none> to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 1 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.342275  1328 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.345296  1322 raft_consensus.cc:2804] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.346858  1322 raft_consensus.cc:493] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.347237  1322 raft_consensus.cc:3060] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.349157  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.365051  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.365687  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
W20260709 13:00:28.358428  1067 leader_election.cc:343] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.365834  1102 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.372615  1293 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.373243  1293 ts_tablet_manager.cc:1403] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.080s	user 0.024s	sys 0.023s
I20260709 13:00:28.366356  1250 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.374356   997 catalog_manager.cc:5697] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.376004  1328 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.381822  1102 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.382378   999 catalog_manager.cc:5697] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 0 to 1, leader changed from <none> to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 1 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:28.377297  1292 ts_tablet_manager.cc:1434] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.054s	user 0.018s	sys 0.001s
I20260709 13:00:28.387244  1293 raft_consensus.cc:359] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.388370  1292 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:28.388216  1293 raft_consensus.cc:385] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.388163  1142 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.389590  1328 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.390375  1328 raft_consensus.cc:697] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:28.392019  1250 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.398396  1322 raft_consensus.cc:515] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.399024  1292 tablet_bootstrap.cc:654] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.400002  1251 raft_consensus.cc:2393] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eb915232cd5045fb89f41b25cc9d0ba2 in current term 1: Already voted for candidate d98a73331940438fa4d42f653a45a71d in this term.
I20260709 13:00:28.391703  1328 consensus_queue.cc:237] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.402715  1066 leader_election.cc:304] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eb915232cd5045fb89f41b25cc9d0ba2; no voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab
I20260709 13:00:28.389031  1293 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.406986  1291 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:28.407433  1291 ts_tablet_manager.cc:1403] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.121s	user 0.025s	sys 0.008s
I20260709 13:00:28.404663  1300 raft_consensus.cc:3060] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.409835  1322 leader_election.cc:290] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.414641  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.414642  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
W20260709 13:00:28.416965  1216 leader_election.cc:343] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.417840  1296 raft_consensus.cc:493] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.419361  1296 raft_consensus.cc:515] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.420642  1292 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.420728  1291 raft_consensus.cc:359] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.421456  1291 raft_consensus.cc:385] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.421782  1291 raft_consensus.cc:740] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.422415  1296 leader_election.cc:290] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.415132  1293 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.423281  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.423616  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
W20260709 13:00:28.424999  1142 leader_election.cc:343] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.421609  1292 ts_tablet_manager.cc:1403] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.005s
I20260709 13:00:28.425443  1300 raft_consensus.cc:2749] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.425647  1293 ts_tablet_manager.cc:1434] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.003s
W20260709 13:00:28.425575  1141 leader_election.cc:343] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.426276  1141 leader_election.cc:304] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e775baeb4df04f29a16933f47713bdab; no voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.427000  1293 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.427492  1296 raft_consensus.cc:2749] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.428407  1291 consensus_queue.cc:260] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.428686  1292 raft_consensus.cc:359] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.429363  1292 raft_consensus.cc:385] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.429700  1292 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.431145  1291 ts_tablet_manager.cc:1434] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.001s
I20260709 13:00:28.431802  1292 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.433485  1291 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.434665  1292 ts_tablet_manager.cc:1434] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:00:28.435648  1292 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:28.438953  1293 tablet_bootstrap.cc:654] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.441700  1292 tablet_bootstrap.cc:654] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.442299  1291 tablet_bootstrap.cc:654] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.442399  1102 raft_consensus.cc:2468] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 1.
I20260709 13:00:28.444826  1216 leader_election.cc:304] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: e775baeb4df04f29a16933f47713bdab
I20260709 13:00:28.446316  1322 raft_consensus.cc:2804] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.447134  1322 raft_consensus.cc:697] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:28.448457  1322 consensus_queue.cc:237] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.459239  1334 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.459957  1334 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.463711  1334 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.466411  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.467579  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
W20260709 13:00:28.468367  1216 leader_election.cc:343] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.469731  1293 tablet_bootstrap.cc:492] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
W20260709 13:00:28.470093  1216 leader_election.cc:343] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:28.472290  1216 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d98a73331940438fa4d42f653a45a71d; no voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.473605  1334 raft_consensus.cc:2749] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.471437  1293 ts_tablet_manager.cc:1403] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.045s	user 0.016s	sys 0.003s
I20260709 13:00:28.465561   997 catalog_manager.cc:5697] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 0 to 1, leader changed from <none> to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 1 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.479056  1291 tablet_bootstrap.cc:492] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:28.478953  1293 raft_consensus.cc:359] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.479746  1291 ts_tablet_manager.cc:1403] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.047s	user 0.010s	sys 0.008s
I20260709 13:00:28.479841  1293 raft_consensus.cc:385] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.480309  1293 raft_consensus.cc:740] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.481230  1293 consensus_queue.cc:260] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.484303  1293 ts_tablet_manager.cc:1434] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:00:28.485459  1293 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.484826   998 catalog_manager.cc:5697] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 0 to 1, leader changed from <none> to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 1 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:28.489396  1291 raft_consensus.cc:359] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.490895  1291 raft_consensus.cc:385] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.491353  1291 raft_consensus.cc:740] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.490957  1292 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.492679  1292 ts_tablet_manager.cc:1403] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.022s
I20260709 13:00:28.492130  1291 consensus_queue.cc:260] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.494928  1291 ts_tablet_manager.cc:1434] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 13:00:28.496039  1291 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: Bootstrap starting.
I20260709 13:00:28.496151  1292 raft_consensus.cc:359] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.496989  1292 raft_consensus.cc:385] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.502383  1292 raft_consensus.cc:740] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.502981  1291 tablet_bootstrap.cc:654] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.503304  1292 consensus_queue.cc:260] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.505640  1292 ts_tablet_manager.cc:1434] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.005s
I20260709 13:00:28.506687  1292 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: Bootstrap starting.
I20260709 13:00:28.502028  1293 tablet_bootstrap.cc:654] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.513347  1291 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: No bootstrap required, opened a new log
I20260709 13:00:28.514076  1291 ts_tablet_manager.cc:1403] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.003s
I20260709 13:00:28.517184  1292 tablet_bootstrap.cc:654] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.518684  1291 raft_consensus.cc:359] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.519856  1291 raft_consensus.cc:385] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.520248  1291 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.521222  1291 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.525975  1291 ts_tablet_manager.cc:1434] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 13:00:28.529644  1292 tablet_bootstrap.cc:492] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: No bootstrap required, opened a new log
I20260709 13:00:28.530292  1292 ts_tablet_manager.cc:1403] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: Time spent bootstrapping tablet: real 0.024s	user 0.011s	sys 0.007s
I20260709 13:00:28.534916  1292 raft_consensus.cc:359] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.535761  1292 raft_consensus.cc:385] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.536033  1292 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.536662  1292 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.537051  1293 tablet_bootstrap.cc:492] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.537775  1293 ts_tablet_manager.cc:1403] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.053s	user 0.023s	sys 0.012s
I20260709 13:00:28.538609  1292 ts_tablet_manager.cc:1434] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:00:28.541095  1296 raft_consensus.cc:493] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.541029  1293 raft_consensus.cc:359] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.542033  1293 raft_consensus.cc:385] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.541886  1296 raft_consensus.cc:515] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.542349  1293 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.543460  1293 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.546590  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "339a602e06ff42d28ca3f7cf5b97603b" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.547302  1293 ts_tablet_manager.cc:1434] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:28.547441  1251 raft_consensus.cc:2468] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.548561  1293 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.549010  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "339a602e06ff42d28ca3f7cf5b97603b" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.549053  1141 leader_election.cc:304] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.549854  1102 raft_consensus.cc:2468] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.550156  1328 raft_consensus.cc:2804] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.550519  1328 raft_consensus.cc:493] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.550894  1328 raft_consensus.cc:3060] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.552286  1296 leader_election.cc:290] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.557503  1293 tablet_bootstrap.cc:654] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.559478  1328 raft_consensus.cc:515] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.563097  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "339a602e06ff42d28ca3f7cf5b97603b" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.563751  1102 raft_consensus.cc:3060] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.566694  1328 leader_election.cc:290] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.568022  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "339a602e06ff42d28ca3f7cf5b97603b" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.568678  1251 raft_consensus.cc:3060] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.570307  1293 tablet_bootstrap.cc:492] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.570804  1293 ts_tablet_manager.cc:1403] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.011s
I20260709 13:00:28.571967  1102 raft_consensus.cc:2468] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.573590  1142 leader_election.cc:304] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.573205  1293 raft_consensus.cc:359] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.574249  1293 raft_consensus.cc:385] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.574579  1328 raft_consensus.cc:2804] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.574668  1293 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.575276  1328 raft_consensus.cc:697] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:28.575769  1293 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.576313  1328 consensus_queue.cc:237] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.578356  1293 ts_tablet_manager.cc:1434] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:28.579591  1293 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.580140  1251 raft_consensus.cc:2468] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.586563  1293 tablet_bootstrap.cc:654] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.586472   999 catalog_manager.cc:5697] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 0 to 1, leader changed from <none> to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 1 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.591869  1293 tablet_bootstrap.cc:492] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.592463  1293 ts_tablet_manager.cc:1403] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 13:00:28.595391  1293 raft_consensus.cc:359] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.596098  1293 raft_consensus.cc:385] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.596398  1293 raft_consensus.cc:740] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.597086  1293 consensus_queue.cc:260] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.598996  1293 ts_tablet_manager.cc:1434] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:00:28.600229  1293 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: Bootstrap starting.
I20260709 13:00:28.605062  1293 tablet_bootstrap.cc:654] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:28.610317  1293 tablet_bootstrap.cc:492] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: No bootstrap required, opened a new log
I20260709 13:00:28.610836  1293 ts_tablet_manager.cc:1403] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent bootstrapping tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:00:28.613307  1293 raft_consensus.cc:359] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.613883  1293 raft_consensus.cc:385] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:28.614063  1293 raft_consensus.cc:740] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Initialized, Role: FOLLOWER
I20260709 13:00:28.614581  1293 consensus_queue.cc:260] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.616587  1293 ts_tablet_manager.cc:1434] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:00:28.648190  1300 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.649116  1300 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.652321  1300 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.652892  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.653097  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.653208  1328 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.653847  1251 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.653927  1328 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.655103  1175 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.655144  1066 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.656155  1300 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.656388  1328 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.656605  1300 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.656997  1300 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.657240  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.658394  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.658893  1251 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.659817  1141 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.660593  1328 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.660919  1328 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.661204  1328 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.663357  1300 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.664413  1102 raft_consensus.cc:2393] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e775baeb4df04f29a16933f47713bdab in current term 1: Already voted for candidate eb915232cd5045fb89f41b25cc9d0ba2 in this term.
I20260709 13:00:28.666102  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.666240  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.666797  1251 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.667078  1300 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.667678  1328 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.668780  1175 raft_consensus.cc:2393] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate eb915232cd5045fb89f41b25cc9d0ba2 in current term 1: Already voted for candidate e775baeb4df04f29a16933f47713bdab in this term.
I20260709 13:00:28.670027  1328 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.671286  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.671692  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.672489  1102 raft_consensus.cc:2393] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e775baeb4df04f29a16933f47713bdab in current term 1: Already voted for candidate eb915232cd5045fb89f41b25cc9d0ba2 in this term.
I20260709 13:00:28.674000  1142 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e775baeb4df04f29a16933f47713bdab; no voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.674612  1251 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 1.
I20260709 13:00:28.675084  1328 raft_consensus.cc:2749] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.675973  1066 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: e775baeb4df04f29a16933f47713bdab
I20260709 13:00:28.676924  1300 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.677385  1300 raft_consensus.cc:697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:28.678381  1300 consensus_queue.cc:237] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.681599  1328 raft_consensus.cc:493] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.682197  1328 raft_consensus.cc:515] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.684514  1328 leader_election.cc:290] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.684974  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.685695  1102 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.685559  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.686331  1251 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 0.
I20260709 13:00:28.686744  1142 leader_election.cc:304] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.687565  1328 raft_consensus.cc:2804] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.687887  1328 raft_consensus.cc:493] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.688174  1328 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.688177   999 catalog_manager.cc:5697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.691051  1300 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.691689  1300 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.694315  1300 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.694655  1328 raft_consensus.cc:515] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.695173  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.695972  1175 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.697172  1067 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.698057  1328 leader_election.cc:290] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.698292  1300 raft_consensus.cc:2804] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.698714  1300 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.698705  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.699229  1300 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.699533  1251 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.699755  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.700335  1250 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 0.
I20260709 13:00:28.701251  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.701828  1102 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.707784  1251 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.708057  1300 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.708861  1102 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 1.
I20260709 13:00:28.708449  1334 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.709331  1141 leader_election.cc:304] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.711747  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.712486  1175 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.710250  1328 raft_consensus.cc:2804] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.714572  1328 raft_consensus.cc:697] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:28.715533  1328 consensus_queue.cc:237] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.723048  1299 raft_consensus.cc:493] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.723830  1299 raft_consensus.cc:515] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.726107  1300 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.729574  1176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab" is_pre_election: true
I20260709 13:00:28.730451  1176 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 0.
I20260709 13:00:28.733536  1216 leader_election.cc:304] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:28.729118  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.735873  1102 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 0.
I20260709 13:00:28.729336  1175 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 1.
I20260709 13:00:28.734982  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:28.737387  1251 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.737553  1100 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.738600  1067 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.739526  1300 raft_consensus.cc:2804] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.740011  1300 raft_consensus.cc:697] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:28.740900  1300 consensus_queue.cc:237] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.740020  1334 raft_consensus.cc:2804] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:28.742784  1334 raft_consensus.cc:493] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:28.743134  1334 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.726541  1299 leader_election.cc:290] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.746304  1251 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 1.
I20260709 13:00:28.751711   998 catalog_manager.cc:5697] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 0 to 1, leader changed from <none> to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 1 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.751463   999 catalog_manager.cc:5697] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:28.755795  1299 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.757648  1334 raft_consensus.cc:515] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.759853  1334 leader_election.cc:290] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:28.761951  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:28.767773  1102 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.767658  1300 consensus_queue.cc:1048] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:28.769199  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:28.769837  1177 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.776381  1102 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 1.
I20260709 13:00:28.778398  1216 leader_election.cc:304] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:28.779628  1299 raft_consensus.cc:2804] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:28.779858  1177 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 1.
I20260709 13:00:28.780179  1299 raft_consensus.cc:697] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:28.780896  1299 consensus_queue.cc:237] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.792269   999 catalog_manager.cc:5697] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 0 to 1, leader changed from <none> to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 1 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:28.794219  1323 consensus_queue.cc:1048] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.801208  1328 raft_consensus.cc:493] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:28.801898  1328 raft_consensus.cc:515] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:28.805318  1328 leader_election.cc:290] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:28.805912  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" is_pre_election: true
I20260709 13:00:28.806782  1102 raft_consensus.cc:2393] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e775baeb4df04f29a16933f47713bdab in current term 1: Already voted for candidate d98a73331940438fa4d42f653a45a71d in this term.
I20260709 13:00:28.809808  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04832b9c7dd546bbb9c58655e83d7543" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98a73331940438fa4d42f653a45a71d" is_pre_election: true
I20260709 13:00:28.811479  1141 leader_election.cc:304] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e775baeb4df04f29a16933f47713bdab; no voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:28.811836  1328 consensus_queue.cc:1048] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.812553  1296 raft_consensus.cc:3060] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:28.820346  1296 raft_consensus.cc:2749] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:28.826857  1364 consensus_queue.cc:1048] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.890202  1299 consensus_queue.cc:1048] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.902145  1322 consensus_queue.cc:1048] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:28.969486  1364 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:28.985575  1296 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:29.015571  1296 consensus_queue.cc:1048] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:29.022801  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:29.023530  1175 raft_consensus.cc:606] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:29.029098  1369 consensus_queue.cc:1048] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.042021  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:29.042924  1175 raft_consensus.cc:606] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:29.049954  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:29.050632  1175 raft_consensus.cc:606] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:29.066479  1251 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
mode: GRACEFUL
new_leader_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:46600
I20260709 13:00:29.067101  1251 raft_consensus.cc:606] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Received request to transfer leadership to TS e775baeb4df04f29a16933f47713bdab
I20260709 13:00:29.077706  1324 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.086001  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:36114
I20260709 13:00:29.086706  1102 raft_consensus.cc:606] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Received request to transfer leadership to TS e775baeb4df04f29a16933f47713bdab
I20260709 13:00:29.092948  1323 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:29.094056  1250 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
mode: GRACEFUL
new_leader_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:46600
I20260709 13:00:29.094758  1250 raft_consensus.cc:606] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Received request to transfer leadership to TS e775baeb4df04f29a16933f47713bdab
I20260709 13:00:29.187990  1323 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.196498  1300 raft_consensus.cc:993] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.196339  1369 consensus_queue.cc:1048] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.197005  1300 raft_consensus.cc:1081] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Signalling peer e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.199997  1177 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:60548
I20260709 13:00:29.200510  1177 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.200857  1177 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.205857  1322 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:29.205888  1324 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:29.208331  1177 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.210511  1177 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.211400  1175 raft_consensus.cc:1240] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Rejecting Update request from peer eb915232cd5045fb89f41b25cc9d0ba2 for earlier term 1. Current term is 2. Ops: []
I20260709 13:00:29.216415  1324 consensus_queue.cc:1059] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.217376  1324 raft_consensus.cc:3055] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.217829  1324 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:29.219004  1324 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.211733  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:29.213356  1249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.219890  1324 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.225368  1369 consensus_queue.cc:1048] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.233932  1322 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:29.240813  1102 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 2.
I20260709 13:00:29.253258  1142 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: d98a73331940438fa4d42f653a45a71d
I20260709 13:00:29.254235  1364 raft_consensus.cc:2804] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.258946  1364 raft_consensus.cc:697] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.260146  1364 consensus_queue.cc:237] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.271835  1328 raft_consensus.cc:993] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.272172  1322 raft_consensus.cc:993] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: : Instructing follower e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.272353  1328 raft_consensus.cc:1081] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.272634  1322 raft_consensus.cc:1081] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Signalling peer e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.274641  1175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:60554
I20260709 13:00:29.275306  1175 raft_consensus.cc:493] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.275689  1175 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.278374  1100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:29.279091  1100 raft_consensus.cc:493] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.279564  1100 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.284783  1175 raft_consensus.cc:515] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.287494  1175 leader_election.cc:290] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.289098  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:29.289279  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.289510  1100 raft_consensus.cc:515] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.289961  1250 raft_consensus.cc:3055] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.289875  1102 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.290369  1250 raft_consensus.cc:740] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:29.291339  1250 consensus_queue.cc:260] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.293251  1100 leader_election.cc:290] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.294798  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.295411  1251 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.298091  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c91326fc824d37a7b3bd5a02edcd6f" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:29.298763  1175 raft_consensus.cc:3055] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.299084  1175 raft_consensus.cc:740] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.299927  1175 consensus_queue.cc:260] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.302098  1175 raft_consensus.cc:3060] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.302139  1102 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 2.
I20260709 13:00:29.303593  1142 leader_election.cc:304] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:29.292838  1250 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.304378  1364 raft_consensus.cc:2804] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.304826  1364 raft_consensus.cc:697] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.305662  1364 consensus_queue.cc:237] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.310353  1251 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.311682  1175 raft_consensus.cc:2468] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.311816  1066 leader_election.cc:304] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:29.312932  1324 raft_consensus.cc:2804] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.313449  1324 raft_consensus.cc:697] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:29.313179  1250 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 2.
I20260709 13:00:29.314455  1324 consensus_queue.cc:237] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.314718  1000 catalog_manager.cc:5697] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 1 to 2, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 2 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.327737  1299 raft_consensus.cc:993] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: : Instructing follower e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.328377  1334 raft_consensus.cc:1081] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Signalling peer e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:29.330439  1175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:60554
I20260709 13:00:29.331351  1175 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.331704  1175 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.330220   999 catalog_manager.cc:5697] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 1 to 2, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.338712  1000 catalog_manager.cc:5697] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 1 to 2, leader changed from d98a73331940438fa4d42f653a45a71d (127.0.107.195) to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 2 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.339826  1364 raft_consensus.cc:993] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.340476  1364 raft_consensus.cc:1081] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.340219  1175 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.344175  1175 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.345953  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.346565  1250 raft_consensus.cc:3055] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.346902  1250 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:29.347745  1250 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.348937  1250 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.350566  1100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:29.351195  1100 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.349108  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:29.355108  1102 raft_consensus.cc:493] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.355531  1102 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.358103  1250 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 2.
I20260709 13:00:29.359179  1141 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:29.359910  1296 raft_consensus.cc:2804] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.360009  1100 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 2.
I20260709 13:00:29.360404  1296 raft_consensus.cc:697] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.361297  1296 consensus_queue.cc:237] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.364215  1102 raft_consensus.cc:515] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.366125  1102 leader_election.cc:290] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.366901  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:29.367570  1175 raft_consensus.cc:3055] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.367607  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "561aa5db68244ddab8f87ed38b2295e5" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.368040  1175 raft_consensus.cc:740] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.368234  1250 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.369236  1175 consensus_queue.cc:260] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.370146   999 catalog_manager.cc:5697] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 1 to 2, leader changed from d98a73331940438fa4d42f653a45a71d (127.0.107.195) to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 2 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.371577  1175 raft_consensus.cc:3060] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.374996  1250 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.375761  1251 raft_consensus.cc:1240] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Rejecting Update request from peer e775baeb4df04f29a16933f47713bdab for earlier term 1. Current term is 2. Ops: []
I20260709 13:00:29.376413  1066 leader_election.cc:304] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:29.377345  1300 raft_consensus.cc:2804] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.377892  1300 raft_consensus.cc:697] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:29.379595  1300 consensus_queue.cc:237] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.386531  1175 raft_consensus.cc:2468] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.389071   999 catalog_manager.cc:5697] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 1 to 2, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.541141  1369 raft_consensus.cc:993] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.541659  1296 raft_consensus.cc:1081] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:29.543365  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:29.543955  1102 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:29.544341  1102 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.550220  1102 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.552032  1102 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:29.552863  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:29.553086  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:29.553591  1175 raft_consensus.cc:3055] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:29.553825  1250 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.553977  1175 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:29.554885  1175 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.556161  1175 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.562565  1250 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.563705  1066 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:29.564292  1175 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 2.
I20260709 13:00:29.564432  1300 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:29.564996  1300 raft_consensus.cc:697] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:29.565912  1300 consensus_queue.cc:237] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:29.573278  1000 catalog_manager.cc:5697] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 1 to 2, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:29.689074  1250 raft_consensus.cc:1275] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.690488  1296 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.699776  1102 raft_consensus.cc:1275] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.701599  1369 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.773056  1250 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.774549  1369 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.783962  1102 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.786566  1369 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.799829  1102 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.801231  1369 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.810842  1250 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:29.818113  1175 raft_consensus.cc:1275] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.819582  1300 consensus_queue.cc:1048] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.819854  1250 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.821487  1328 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.831570  1250 raft_consensus.cc:1275] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.833388  1323 consensus_queue.cc:1048] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.914482  1250 raft_consensus.cc:1275] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.918453  1300 consensus_queue.cc:1048] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.924544  1175 raft_consensus.cc:1275] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.926375  1323 consensus_queue.cc:1048] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.987246  1250 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:29.988703  1323 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:29.998654  1175 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:30.000672  1323 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:32.142540   431 auto_leader_rebalancer-test.cc:348] The leader distribution is 
I20260709 13:00:32.143536   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:32.166781  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:32.167560  1102 raft_consensus.cc:606] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Received request to transfer leadership to TS d98a73331940438fa4d42f653a45a71d
I20260709 13:00:32.170747  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:32.171262  1102 raft_consensus.cc:606] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Received request to transfer leadership to TS d98a73331940438fa4d42f653a45a71d
I20260709 13:00:32.172408   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:00:32.173024   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:32.443207  1323 raft_consensus.cc:993] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:32.443879  1398 raft_consensus.cc:1081] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Signalling peer d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:32.445786  1250 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:46584
I20260709 13:00:32.446319  1250 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:32.446730  1250 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:32.453989  1250 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.456337  1250 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 2 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:32.457670  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:32.457326  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:32.458288  1175 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:32.458312  1102 raft_consensus.cc:3055] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:32.458711  1102 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:32.459461  1102 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.460453  1102 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:32.465696  1175 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 2.
I20260709 13:00:32.467118  1102 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 2.
I20260709 13:00:32.467093  1216 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:32.468245  1334 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:32.469808  1334 raft_consensus.cc:697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:32.470932  1334 consensus_queue.cc:237] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.483649  1000 catalog_manager.cc:5697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 1 to 2, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 2 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:32.536369  1323 raft_consensus.cc:993] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:32.536950  1398 raft_consensus.cc:1081] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Signalling peer d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:32.538635  1250 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:46584
I20260709 13:00:32.539162  1250 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:32.539491  1250 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:32.544840  1250 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.546679  1250 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 3 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:32.547649  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:32.547964  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:32.548264  1102 raft_consensus.cc:3055] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:32.548527  1175 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:32.548707  1102 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:32.549640  1102 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.550905  1102 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:32.555930  1175 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 3.
I20260709 13:00:32.557533  1216 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:32.558429  1334 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:32.558566  1102 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 3.
I20260709 13:00:32.558930  1334 raft_consensus.cc:697] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:32.559917  1334 consensus_queue.cc:237] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:32.567590   999 catalog_manager.cc:5697] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 2 to 3, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 3 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:32.864794  1100 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:32.866158  1334 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:32.876329  1175 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:00:32.878072  1334 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:32.996826  1175 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:32.998478  1334 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:33.008248  1100 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:33.010370  1334 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:33.174103   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:33.188966  1250 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:46600
I20260709 13:00:33.189596  1250 raft_consensus.cc:606] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:33.198055  1250 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:46600
I20260709 13:00:33.198709  1250 raft_consensus.cc:606] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:33.206838  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:33.207372  1175 raft_consensus.cc:606] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:33.212483  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:33.213019  1175 raft_consensus.cc:606] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:33.218459  1175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
mode: GRACEFUL
new_leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:60558
I20260709 13:00:33.218946  1175 raft_consensus.cc:606] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Received request to transfer leadership to TS eb915232cd5045fb89f41b25cc9d0ba2
I20260709 13:00:33.345789  1369 raft_consensus.cc:993] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.346659  1414 raft_consensus.cc:1081] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.348522  1100 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:33.349017  1100 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:33.349282  1100 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.354215  1369 raft_consensus.cc:993] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.354828  1369 raft_consensus.cc:1081] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.357364  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:33.358152  1102 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:33.358457  1102 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.365712  1100 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.368326  1100 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:33.369546  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:33.369827  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:33.370244  1175 raft_consensus.cc:3055] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:33.370347  1250 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.370682  1175 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:33.371548  1175 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.372664  1175 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.382731  1250 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.383021  1175 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.382789  1102 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.384912  1066 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:33.385668  1102 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:33.386683  1398 raft_consensus.cc:2804] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:33.387398  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:33.387984  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:33.388123  1175 raft_consensus.cc:3055] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:33.388600  1175 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:33.390319  1175 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.389129  1398 raft_consensus.cc:697] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:33.392750  1175 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.393450  1414 raft_consensus.cc:993] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.394308  1414 raft_consensus.cc:1081] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.393325  1398 consensus_queue.cc:237] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.396027  1250 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.396708  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36098
I20260709 13:00:33.397347  1102 raft_consensus.cc:493] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:33.397794  1102 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.411158   999 catalog_manager.cc:5697] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 2 to 3, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:33.418730  1175 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.420508  1067 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e775baeb4df04f29a16933f47713bdab, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:33.418764  1102 raft_consensus.cc:515] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.421844  1398 raft_consensus.cc:2804] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:33.423539  1398 raft_consensus.cc:697] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:33.424476  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:33.424628  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81e8fe3749e54d4eb6630dd76cd47146" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:33.425094  1175 raft_consensus.cc:3055] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:33.424619  1398 consensus_queue.cc:237] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.425324  1251 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.425451  1175 raft_consensus.cc:740] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:33.426390  1175 consensus_queue.cc:260] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.427548  1175 raft_consensus.cc:3060] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.419788  1250 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.432528  1102 leader_election.cc:290] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:33.434333  1334 raft_consensus.cc:993] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.434890  1334 raft_consensus.cc:1081] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.438071  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36106
I20260709 13:00:33.438952  1102 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:33.438115   999 catalog_manager.cc:5697] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 2 to 3, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:33.439411  1102 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.463609  1251 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.463652  1102 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.463609  1175 raft_consensus.cc:2468] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.465482  1066 leader_election.cc:304] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:33.466233  1102 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:33.467144  1398 raft_consensus.cc:2804] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:33.467710  1398 raft_consensus.cc:697] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:33.468086  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:33.468014  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:33.468675  1250 raft_consensus.cc:3055] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:00:33.468735  1177 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.469041  1250 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:33.469554  1398 consensus_queue.cc:237] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.469995  1250 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.471246  1250 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:33.483513   999 catalog_manager.cc:5697] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 2 to 3, leader changed from e775baeb4df04f29a16933f47713bdab (127.0.107.194) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:33.494165  1177 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.494336  1250 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 3.
I20260709 13:00:33.495699  1066 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:33.496656  1398 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:33.497126  1398 raft_consensus.cc:697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:33.497901  1398 consensus_queue.cc:237] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.504549   999 catalog_manager.cc:5697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 2 to 3, leader changed from d98a73331940438fa4d42f653a45a71d (127.0.107.195) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:33.627664  1399 raft_consensus.cc:993] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: : Instructing follower eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.628300  1412 raft_consensus.cc:1081] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 LEADER]: Signalling peer eb915232cd5045fb89f41b25cc9d0ba2 to start an election
I20260709 13:00:33.630555  1102 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
 from {username='slave'} at 127.0.0.1:36106
I20260709 13:00:33.631112  1102 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:33.631532  1102 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:33.667598  1102 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.669397  1102 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 4 election: Requested vote from peers e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:33.670681  1250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:33.670704  1177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:33.671416  1250 raft_consensus.cc:3055] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:33.671510  1177 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:33.671730  1250 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:33.672547  1250 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.673451  1250 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:33.762310  1250 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 4.
I20260709 13:00:33.762315  1177 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb915232cd5045fb89f41b25cc9d0ba2 in term 4.
I20260709 13:00:33.763989  1066 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, eb915232cd5045fb89f41b25cc9d0ba2; no voters: 
I20260709 13:00:33.765111  1398 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:33.765615  1398 raft_consensus.cc:697] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 LEADER]: Becoming Leader. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:33.766809  1398 consensus_queue.cc:237] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:33.775158   999 catalog_manager.cc:5697] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 reported cstate change: term changed from 3 to 4, leader changed from d98a73331940438fa4d42f653a45a71d (127.0.107.195) to eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193). New cstate: current_term: 4 leader_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:33.809710  1250 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:33.811056  1398 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:33.821906  1177 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:33.823793  1424 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:33.918897  1177 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:33.920800  1424 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:33.931843  1250 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:33.933707  1415 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:34.029583  1251 raft_consensus.cc:1275] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:34.031107  1398 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:34.031612  1251 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:34.031836  1177 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:34.033376  1398 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:34.034724  1415 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:34.044323  1175 raft_consensus.cc:1275] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:00:34.049017  1424 consensus_queue.cc:1048] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:34.253247  1177 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:34.254698  1415 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:34.265908  1249 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Refusing update from remote peer eb915232cd5045fb89f41b25cc9d0ba2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:34.267364  1415 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:36.226850   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:36.235636  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:36.236707  1102 raft_consensus.cc:606] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Received request to transfer leadership to TS d98a73331940438fa4d42f653a45a71d
I20260709 13:00:36.240790  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:36.241415  1102 raft_consensus.cc:606] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Received request to transfer leadership to TS e775baeb4df04f29a16933f47713bdab
I20260709 13:00:36.244607  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:36.245179  1102 raft_consensus.cc:606] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 LEADER]: Received request to transfer leadership to TS e775baeb4df04f29a16933f47713bdab
I20260709 13:00:36.248246  1102 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
mode: GRACEFUL
new_leader_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:51064
I20260709 13:00:36.248909  1102 raft_consensus.cc:606] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Received request to transfer leadership to TS d98a73331940438fa4d42f653a45a71d
I20260709 13:00:36.250176   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:00:36.250893   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:00:36.386380  1424 raft_consensus.cc:993] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:36.386890  1437 raft_consensus.cc:1081] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 LEADER]: Signalling peer e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:36.388465  1175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:60548
I20260709 13:00:36.389004  1175 raft_consensus.cc:493] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:36.389317  1175 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:36.394608  1175 raft_consensus.cc:515] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.396497  1175 leader_election.cc:290] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 5 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:36.397708  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:36.397904  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "777449d6216b4cec8aa37d01aeba7885" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:36.398347  1102 raft_consensus.cc:3055] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:00:36.398572  1251 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:36.398856  1102 raft_consensus.cc:740] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:36.399861  1102 consensus_queue.cc:260] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.401175  1102 raft_consensus.cc:3060] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:00:36.405493  1251 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 5.
I20260709 13:00:36.405982  1102 raft_consensus.cc:2468] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 5.
I20260709 13:00:36.407073  1141 leader_election.cc:304] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:36.407938  1369 raft_consensus.cc:2804] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:00:36.409004  1369 raft_consensus.cc:697] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 5 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:36.409945  1369 consensus_queue.cc:237] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.417201   997 catalog_manager.cc:5697] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 4 to 5, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 5 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:36.469238  1424 raft_consensus.cc:993] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:36.469811  1437 raft_consensus.cc:1081] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Signalling peer d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:36.471921  1251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:46584
I20260709 13:00:36.472587  1251 raft_consensus.cc:493] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:36.472848  1251 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.478546  1251 raft_consensus.cc:515] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.480715  1251 leader_election.cc:290] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 4 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:36.482076  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:36.482125  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5b866f5c7340c1bc5e6735a0be5189" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:36.482784  1102 raft_consensus.cc:3055] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:36.482923  1175 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.483145  1102 raft_consensus.cc:740] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:36.483973  1102 consensus_queue.cc:260] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.484947  1102 raft_consensus.cc:3060] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.487974  1175 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 4.
I20260709 13:00:36.489310  1216 leader_election.cc:304] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:36.490341  1412 raft_consensus.cc:2804] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:36.490983  1102 raft_consensus.cc:2468] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 4.
I20260709 13:00:36.492764  1412 raft_consensus.cc:697] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 4 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:36.493810  1412 consensus_queue.cc:237] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.501405   997 catalog_manager.cc:5697] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 3 to 4, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 4 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:36.520828  1424 raft_consensus.cc:993] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:36.521435  1437 raft_consensus.cc:1081] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Signalling peer e775baeb4df04f29a16933f47713bdab to start an election
I20260709 13:00:36.523891  1175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2"
dest_uuid: "e775baeb4df04f29a16933f47713bdab"
 from {username='slave'} at 127.0.0.1:60548
I20260709 13:00:36.524549  1175 raft_consensus.cc:493] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:36.524848  1175 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.530172  1175 raft_consensus.cc:515] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.531757  1175 leader_election.cc:290] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 4 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), d98a73331940438fa4d42f653a45a71d (127.0.107.195:39403)
I20260709 13:00:36.532863  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:36.532908  1251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c5b67a6aeb447c6bb97c157612cc3b2" candidate_uuid: "e775baeb4df04f29a16933f47713bdab" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d98a73331940438fa4d42f653a45a71d"
I20260709 13:00:36.533557  1102 raft_consensus.cc:3055] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:36.533732  1251 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.533875  1102 raft_consensus.cc:740] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:36.534715  1102 consensus_queue.cc:260] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.535684  1102 raft_consensus.cc:3060] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.541708  1251 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 4.
I20260709 13:00:36.542414  1102 raft_consensus.cc:2468] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e775baeb4df04f29a16933f47713bdab in term 4.
I20260709 13:00:36.542981  1141 leader_election.cc:304] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:36.543946  1369 raft_consensus.cc:2804] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:36.544646  1369 raft_consensus.cc:697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 4 LEADER]: Becoming Leader. State: Replica: e775baeb4df04f29a16933f47713bdab, State: Running, Role: LEADER
I20260709 13:00:36.545691  1369 consensus_queue.cc:237] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.554036   997 catalog_manager.cc:5697] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab reported cstate change: term changed from 3 to 4, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to e775baeb4df04f29a16933f47713bdab (127.0.107.194). New cstate: current_term: 4 leader_uuid: "e775baeb4df04f29a16933f47713bdab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:36.760998  1424 raft_consensus.cc:993] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: : Instructing follower d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:36.761644  1437 raft_consensus.cc:1081] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Signalling peer d98a73331940438fa4d42f653a45a71d to start an election
I20260709 13:00:36.764361  1251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61"
dest_uuid: "d98a73331940438fa4d42f653a45a71d"
 from {username='slave'} at 127.0.0.1:46584
I20260709 13:00:36.764876  1251 raft_consensus.cc:493] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:00:36.765159  1251 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.770304  1251 raft_consensus.cc:515] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.772122  1251 leader_election.cc:290] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 4 election: Requested vote from peers eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045), e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161)
I20260709 13:00:36.773186  1102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "eb915232cd5045fb89f41b25cc9d0ba2"
I20260709 13:00:36.773289  1175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "98f0ba6abb364f75b64408020f47fe61" candidate_uuid: "d98a73331940438fa4d42f653a45a71d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e775baeb4df04f29a16933f47713bdab"
I20260709 13:00:36.774040  1102 raft_consensus.cc:3055] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:00:36.774226  1175 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.774341  1102 raft_consensus.cc:740] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: eb915232cd5045fb89f41b25cc9d0ba2, State: Running, Role: LEADER
I20260709 13:00:36.775424  1102 consensus_queue.cc:260] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.776324  1102 raft_consensus.cc:3060] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:00:36.780947  1175 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 4.
I20260709 13:00:36.781931  1102 raft_consensus.cc:2468] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d98a73331940438fa4d42f653a45a71d in term 4.
I20260709 13:00:36.782212  1216 leader_election.cc:304] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98a73331940438fa4d42f653a45a71d, e775baeb4df04f29a16933f47713bdab; no voters: 
I20260709 13:00:36.783017  1412 raft_consensus.cc:2804] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:00:36.783456  1412 raft_consensus.cc:697] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 4 LEADER]: Becoming Leader. State: Replica: d98a73331940438fa4d42f653a45a71d, State: Running, Role: LEADER
I20260709 13:00:36.784463  1412 consensus_queue.cc:237] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } }
I20260709 13:00:36.792817   997 catalog_manager.cc:5697] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d reported cstate change: term changed from 3 to 4, leader changed from eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193) to d98a73331940438fa4d42f653a45a71d (127.0.107.195). New cstate: current_term: 4 leader_uuid: "d98a73331940438fa4d42f653a45a71d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:36.914459  1175 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:36.915872  1412 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:36.926988  1251 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 5 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:36.929468  1369 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:36.931452  1102 raft_consensus.cc:1275] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:36.934114  1444 consensus_queue.cc:1048] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:36.943789  1100 raft_consensus.cc:1275] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 5 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:00:36.945572  1100 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:36.945422  1438 consensus_queue.cc:1048] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:00:36.947369  1438 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:36.966657  1249 raft_consensus.cc:1275] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Refusing update from remote peer e775baeb4df04f29a16933f47713bdab: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:36.968396  1369 consensus_queue.cc:1048] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [LEADER]: Connected to new peer: Peer: permanent_uuid: "d98a73331940438fa4d42f653a45a71d" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:37.191890  1175 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:37.193324  1412 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e775baeb4df04f29a16933f47713bdab" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 40161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:37.203032  1100 raft_consensus.cc:1275] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Refusing update from remote peer d98a73331940438fa4d42f653a45a71d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:00:37.204819  1444 consensus_queue.cc:1048] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb915232cd5045fb89f41b25cc9d0ba2" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 38045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:37.251875   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:00:37.258379   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 13:00:37.289043   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:37.290079   431 tablet_replica.cc:333] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.291256   431 raft_consensus.cc:2243] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.292232   431 raft_consensus.cc:2272] T 81e8fe3749e54d4eb6630dd76cd47146 P eb915232cd5045fb89f41b25cc9d0ba2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.294621   431 tablet_replica.cc:333] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.295264   431 raft_consensus.cc:2243] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.296124   431 raft_consensus.cc:2272] T 82c91326fc824d37a7b3bd5a02edcd6f P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.298401   431 tablet_replica.cc:333] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.298965   431 raft_consensus.cc:2243] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.299494   431 raft_consensus.cc:2272] T 2c5b67a6aeb447c6bb97c157612cc3b2 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.302423   431 tablet_replica.cc:333] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.303064   431 raft_consensus.cc:2243] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.303428   431 raft_consensus.cc:2272] T 98f0ba6abb364f75b64408020f47fe61 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.305703   431 tablet_replica.cc:333] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.306213   431 raft_consensus.cc:2243] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.306567   431 raft_consensus.cc:2272] T 777449d6216b4cec8aa37d01aeba7885 P eb915232cd5045fb89f41b25cc9d0ba2 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.308660   431 tablet_replica.cc:333] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.309185   431 raft_consensus.cc:2243] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.309962   431 raft_consensus.cc:2272] T 8159458859ef47cfb00c03a3b353fa35 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.311637   431 tablet_replica.cc:333] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.312090   431 raft_consensus.cc:2243] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.312448   431 raft_consensus.cc:2272] T 339a602e06ff42d28ca3f7cf5b97603b P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.314095   431 tablet_replica.cc:333] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.314534   431 raft_consensus.cc:2243] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.314970   431 raft_consensus.cc:2272] T 1c5b866f5c7340c1bc5e6735a0be5189 P eb915232cd5045fb89f41b25cc9d0ba2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.317202   431 tablet_replica.cc:333] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.317785   431 raft_consensus.cc:2243] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.318217   431 raft_consensus.cc:2272] T 04832b9c7dd546bbb9c58655e83d7543 P eb915232cd5045fb89f41b25cc9d0ba2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.320644   431 tablet_replica.cc:333] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2: stopping tablet replica
I20260709 13:00:37.321256   431 raft_consensus.cc:2243] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.322314   431 raft_consensus.cc:2272] T 561aa5db68244ddab8f87ed38b2295e5 P eb915232cd5045fb89f41b25cc9d0ba2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.353272   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:00:37.353237  1142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260709 13:00:37.357374  1142 consensus_peers.cc:597] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab -> Peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Couldn't send request to peer eb915232cd5045fb89f41b25cc9d0ba2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:00:37.374452  1142 consensus_peers.cc:597] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab -> Peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Couldn't send request to peer eb915232cd5045fb89f41b25cc9d0ba2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:37.379436   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:00:37.414255  1216 consensus_peers.cc:597] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d -> Peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Couldn't send request to peer eb915232cd5045fb89f41b25cc9d0ba2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:00:37.418901  1142 consensus_peers.cc:597] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab -> Peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Couldn't send request to peer eb915232cd5045fb89f41b25cc9d0ba2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:37.422106   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:37.422907   431 tablet_replica.cc:333] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.423620   431 raft_consensus.cc:2243] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.424538   431 raft_consensus.cc:2272] T 2c5b67a6aeb447c6bb97c157612cc3b2 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.426913   431 tablet_replica.cc:333] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.427469   431 raft_consensus.cc:2243] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.427868   431 raft_consensus.cc:2272] T 82c91326fc824d37a7b3bd5a02edcd6f P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.429785   431 tablet_replica.cc:333] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.430261   431 raft_consensus.cc:2243] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.430609   431 raft_consensus.cc:2272] T 81e8fe3749e54d4eb6630dd76cd47146 P e775baeb4df04f29a16933f47713bdab [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.432227   431 tablet_replica.cc:333] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.432686   431 raft_consensus.cc:2243] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.433051   431 raft_consensus.cc:2272] T 98f0ba6abb364f75b64408020f47fe61 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.435348   431 tablet_replica.cc:333] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.435850   431 raft_consensus.cc:2243] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.436219   431 raft_consensus.cc:2272] T 04832b9c7dd546bbb9c58655e83d7543 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.437979   431 tablet_replica.cc:333] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.438447   431 raft_consensus.cc:2243] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.438822   431 raft_consensus.cc:2272] T 1c5b866f5c7340c1bc5e6735a0be5189 P e775baeb4df04f29a16933f47713bdab [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.440735   431 tablet_replica.cc:333] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.441206   431 raft_consensus.cc:2243] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.442257   431 raft_consensus.cc:2272] T 339a602e06ff42d28ca3f7cf5b97603b P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.444628   431 tablet_replica.cc:333] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.445096   431 raft_consensus.cc:2243] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.445433   431 raft_consensus.cc:2272] T 8159458859ef47cfb00c03a3b353fa35 P e775baeb4df04f29a16933f47713bdab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.447562   431 tablet_replica.cc:333] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.448016   431 raft_consensus.cc:2243] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.448853   431 raft_consensus.cc:2272] T 777449d6216b4cec8aa37d01aeba7885 P e775baeb4df04f29a16933f47713bdab [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.452390   431 tablet_replica.cc:333] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab: stopping tablet replica
I20260709 13:00:37.453105   431 raft_consensus.cc:2243] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:00:37.453049  1216 consensus_peers.cc:597] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d -> Peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Couldn't send request to peer e775baeb4df04f29a16933f47713bdab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:37.453904   431 raft_consensus.cc:2272] T 561aa5db68244ddab8f87ed38b2295e5 P e775baeb4df04f29a16933f47713bdab [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.478324   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
W20260709 13:00:37.484218  1216 consensus_peers.cc:597] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d -> Peer e775baeb4df04f29a16933f47713bdab (127.0.107.194:40161): Couldn't send request to peer e775baeb4df04f29a16933f47713bdab. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:40161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:37.502732   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 13:00:37.525695  1216 consensus_peers.cc:597] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d -> Peer eb915232cd5045fb89f41b25cc9d0ba2 (127.0.107.193:38045): Couldn't send request to peer eb915232cd5045fb89f41b25cc9d0ba2. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:38045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:37.542402   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:00:37.543411   431 tablet_replica.cc:333] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.544203   431 raft_consensus.cc:2243] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.545085   431 raft_consensus.cc:2272] T 81e8fe3749e54d4eb6630dd76cd47146 P d98a73331940438fa4d42f653a45a71d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.547808   431 tablet_replica.cc:333] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.548439   431 raft_consensus.cc:2243] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.548918   431 raft_consensus.cc:2272] T 2c5b67a6aeb447c6bb97c157612cc3b2 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.551630   431 tablet_replica.cc:333] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.552261   431 raft_consensus.cc:2243] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.552735   431 raft_consensus.cc:2272] T 777449d6216b4cec8aa37d01aeba7885 P d98a73331940438fa4d42f653a45a71d [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.555197   431 tablet_replica.cc:333] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.555809   431 raft_consensus.cc:2243] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.556664   431 raft_consensus.cc:2272] T 98f0ba6abb364f75b64408020f47fe61 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.558892   431 tablet_replica.cc:333] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.559435   431 raft_consensus.cc:2243] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.560236   431 raft_consensus.cc:2272] T 1c5b866f5c7340c1bc5e6735a0be5189 P d98a73331940438fa4d42f653a45a71d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.562812   431 tablet_replica.cc:333] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.563369   431 raft_consensus.cc:2243] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.563789   431 raft_consensus.cc:2272] T 339a602e06ff42d28ca3f7cf5b97603b P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.565594   431 tablet_replica.cc:333] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.566099   431 raft_consensus.cc:2243] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.566864   431 raft_consensus.cc:2272] T 04832b9c7dd546bbb9c58655e83d7543 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.568745   431 tablet_replica.cc:333] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.569262   431 raft_consensus.cc:2243] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.569810   431 raft_consensus.cc:2272] T 561aa5db68244ddab8f87ed38b2295e5 P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.571595   431 tablet_replica.cc:333] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.572116   431 raft_consensus.cc:2243] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.572652   431 raft_consensus.cc:2272] T 8159458859ef47cfb00c03a3b353fa35 P d98a73331940438fa4d42f653a45a71d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.575105   431 tablet_replica.cc:333] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d: stopping tablet replica
I20260709 13:00:37.575878   431 raft_consensus.cc:2243] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:37.576416   431 raft_consensus.cc:2272] T 82c91326fc824d37a7b3bd5a02edcd6f P d98a73331940438fa4d42f653a45a71d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.608668   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:00:37.630961   431 master.cc:562] Master@127.0.107.254:36433 shutting down...
I20260709 13:00:37.655870   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:00:37.656649   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:37.657107   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P b123de53400e4f4bb5b5f63b3fc9e465: stopping tablet replica
I20260709 13:00:37.680436   431 master.cc:584] Master@127.0.107.254:36433 shutdown complete.
d98a73331940438fa4d42f653a45a71d  1
e775baeb4df04f29a16933f47713bdab  4
eb915232cd5045fb89f41b25cc9d0ba2  5
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (10875 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 13:00:37.733358   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:39071
I20260709 13:00:37.734843   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:37.742496  1466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:37.743093  1467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:37.743366  1469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:37.744798   431 server_base.cc:1061] running on GCE node
I20260709 13:00:37.745841   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:37.746052   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:37.746171   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602037746159 us; error 0 us; skew 500 ppm
I20260709 13:00:37.746870   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:37.749893   431 webserver.cc:533] Webserver started at http://127.0.107.254:35579/ using document root <none> and password file <none>
I20260709 13:00:37.750408   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:37.750711   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:37.751049   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:37.752360   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "e6b16382849d482ba544fc5ee58e0b0a"
format_stamp: "Formatted at 2026-07-09 13:00:37 on dist-test-slave-p8rg"
I20260709 13:00:37.757284   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.000s	sys 0.007s
I20260709 13:00:37.761699  1474 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:37.762535   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:00:37.762858   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "e6b16382849d482ba544fc5ee58e0b0a"
format_stamp: "Formatted at 2026-07-09 13:00:37 on dist-test-slave-p8rg"
I20260709 13:00:37.763177   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:37.776274   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:37.777776   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:37.828027   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:39071
I20260709 13:00:37.828135  1525 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:39071 every 8 connection(s)
I20260709 13:00:37.832697  1526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:37.844519  1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Bootstrap starting.
I20260709 13:00:37.850495  1526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:37.855774  1526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: No bootstrap required, opened a new log
I20260709 13:00:37.858420  1526 raft_consensus.cc:359] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER }
I20260709 13:00:37.859009  1526 raft_consensus.cc:385] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:37.859352  1526 raft_consensus.cc:740] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6b16382849d482ba544fc5ee58e0b0a, State: Initialized, Role: FOLLOWER
I20260709 13:00:37.860119  1526 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER }
I20260709 13:00:37.861150  1526 raft_consensus.cc:399] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:00:37.861824  1526 raft_consensus.cc:493] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:00:37.862262  1526 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:37.868703  1526 raft_consensus.cc:515] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER }
I20260709 13:00:37.869449  1526 leader_election.cc:304] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6b16382849d482ba544fc5ee58e0b0a; no voters: 
I20260709 13:00:37.870914  1526 leader_election.cc:290] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:00:37.871681  1529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:37.874559  1529 raft_consensus.cc:697] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 1 LEADER]: Becoming Leader. State: Replica: e6b16382849d482ba544fc5ee58e0b0a, State: Running, Role: LEADER
I20260709 13:00:37.875411  1529 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER }
I20260709 13:00:37.876773  1526 sys_catalog.cc:565] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:00:37.882025  1531 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6b16382849d482ba544fc5ee58e0b0a. Latest consensus state: current_term: 1 leader_uuid: "e6b16382849d482ba544fc5ee58e0b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER } }
I20260709 13:00:37.882109  1530 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6b16382849d482ba544fc5ee58e0b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6b16382849d482ba544fc5ee58e0b0a" member_type: VOTER } }
I20260709 13:00:37.882980  1531 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:37.883122  1530 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [sys.catalog]: This master's current role is: LEADER
I20260709 13:00:37.887535  1536 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:00:37.895157  1536 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:00:37.898659   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:00:37.907025  1536 catalog_manager.cc:1383] Generated new cluster ID: 4255dca32d3e41e297bccc35f8f3d06e
I20260709 13:00:37.907416  1536 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:00:37.942018  1536 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:00:37.943620  1536 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:00:37.959390  1536 catalog_manager.cc:6070] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Generated new TSK 0
I20260709 13:00:37.960170  1536 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:00:37.966349   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:37.974045  1547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:37.975009  1548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:37.979045   431 server_base.cc:1061] running on GCE node
W20260709 13:00:37.978974  1550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:37.980661   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:37.980964   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:37.981158   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602037981136 us; error 0 us; skew 500 ppm
I20260709 13:00:37.981815   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:37.985137   431 webserver.cc:533] Webserver started at http://127.0.107.193:35553/ using document root <none> and password file <none>
I20260709 13:00:37.985826   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:37.986074   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:37.986387   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:37.988318   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "316f3bf18ed94e32b238130a6bfedf1a"
format_stamp: "Formatted at 2026-07-09 13:00:37 on dist-test-slave-p8rg"
I20260709 13:00:37.995311   431 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.003s
I20260709 13:00:37.999763  1555 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.001006   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:00:38.001443   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "316f3bf18ed94e32b238130a6bfedf1a"
format_stamp: "Formatted at 2026-07-09 13:00:37 on dist-test-slave-p8rg"
I20260709 13:00:38.001901   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:38.029642   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:38.031237   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:38.033236   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:38.036008   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:38.036257   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.036526   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:38.036720   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.097781   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:41267
I20260709 13:00:38.097859  1617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:41267 every 8 connection(s)
I20260709 13:00:38.106333   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:38.115981  1622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:38.116107  1623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:38.121007  1625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:38.121248   431 server_base.cc:1061] running on GCE node
I20260709 13:00:38.123010   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:38.123235   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:38.123620   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602038123594 us; error 0 us; skew 500 ppm
I20260709 13:00:38.124459   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:38.127974   431 webserver.cc:533] Webserver started at http://127.0.107.194:36861/ using document root <none> and password file <none>
I20260709 13:00:38.128448  1618 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39071
I20260709 13:00:38.128592   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:38.128919  1618 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:38.128945   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:38.129449   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:38.130045  1618 heartbeater.cc:507] Master 127.0.107.254:39071 requested a full tablet report, sending...
I20260709 13:00:38.132085   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "ff44a4643e01482cb6b6a79b099fa766"
format_stamp: "Formatted at 2026-07-09 13:00:38 on dist-test-slave-p8rg"
I20260709 13:00:38.132704  1491 ts_manager.cc:194] Registered new tserver with Master: 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:00:38.134927  1491 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59532
I20260709 13:00:38.138464   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20260709 13:00:38.143476  1630 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.144526   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.002s
I20260709 13:00:38.144831   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "ff44a4643e01482cb6b6a79b099fa766"
format_stamp: "Formatted at 2026-07-09 13:00:38 on dist-test-slave-p8rg"
I20260709 13:00:38.145125   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:38.167521   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:38.168918   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:38.170923   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:38.173882   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:38.174083   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.174270   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:38.174399   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.228845   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:37489
I20260709 13:00:38.229266  1692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:37489 every 8 connection(s)
I20260709 13:00:38.234522   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:38.243984  1697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:38.244022  1696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:38.248037  1699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:38.249092   431 server_base.cc:1061] running on GCE node
I20260709 13:00:38.250965   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:00:38.251045  1693 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39071
W20260709 13:00:38.251391   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:38.251551  1693 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:38.251647   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602038251620 us; error 0 us; skew 500 ppm
I20260709 13:00:38.252542   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:38.252589  1693 heartbeater.cc:507] Master 127.0.107.254:39071 requested a full tablet report, sending...
I20260709 13:00:38.255460  1491 ts_manager.cc:194] Registered new tserver with Master: ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:38.256126   431 webserver.cc:533] Webserver started at http://127.0.107.195:33087/ using document root <none> and password file <none>
I20260709 13:00:38.256733   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:38.256968   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:38.257249   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:38.257230  1491 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59536
I20260709 13:00:38.259402   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "df6a18af06d543b2931837f5e6bab95f"
format_stamp: "Formatted at 2026-07-09 13:00:38 on dist-test-slave-p8rg"
I20260709 13:00:38.264634   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:00:38.268715  1704 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.269955   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:00:38.270323   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "df6a18af06d543b2931837f5e6bab95f"
format_stamp: "Formatted at 2026-07-09 13:00:38 on dist-test-slave-p8rg"
I20260709 13:00:38.270660   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:38.285279   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:38.287032   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:38.288816   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:38.292205   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:38.292477   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.292754   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:38.292949   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:38.347627   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:33737
I20260709 13:00:38.347690  1766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:33737 every 8 connection(s)
I20260709 13:00:38.366158  1767 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39071
I20260709 13:00:38.366676  1767 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:38.367467  1767 heartbeater.cc:507] Master 127.0.107.254:39071 requested a full tablet report, sending...
I20260709 13:00:38.369796  1491 ts_manager.cc:194] Registered new tserver with Master: df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:38.370223   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018623649s
I20260709 13:00:38.371635  1491 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59548
I20260709 13:00:38.404615  1491 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59552:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:00:38.412106  1491 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:00:38.638072  1658 tablet_service.cc:1511] Processing CreateTablet for tablet fbe322e4447d486680550d1490266b17 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:38.640329  1732 tablet_service.cc:1511] Processing CreateTablet for tablet fbe322e4447d486680550d1490266b17 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:38.642139  1657 tablet_service.cc:1511] Processing CreateTablet for tablet d4817dbc3c2e469195aa3f5e05a1534a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:38.643388  1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe322e4447d486680550d1490266b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.644302  1731 tablet_service.cc:1511] Processing CreateTablet for tablet d4817dbc3c2e469195aa3f5e05a1534a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:38.646162  1732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe322e4447d486680550d1490266b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.647205  1656 tablet_service.cc:1511] Processing CreateTablet for tablet 648ab3b5826e49fe81c3363d752d8af8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:38.648655  1656 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ab3b5826e49fe81c3363d752d8af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.649394  1730 tablet_service.cc:1511] Processing CreateTablet for tablet 648ab3b5826e49fe81c3363d752d8af8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:38.652865  1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4817dbc3c2e469195aa3f5e05a1534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.652629  1655 tablet_service.cc:1511] Processing CreateTablet for tablet 5a57c404b2c145bf95efa4a0649c0e33 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:38.650018  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4817dbc3c2e469195aa3f5e05a1534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.654832  1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ab3b5826e49fe81c3363d752d8af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.657596  1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a57c404b2c145bf95efa4a0649c0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.659502  1729 tablet_service.cc:1511] Processing CreateTablet for tablet 5a57c404b2c145bf95efa4a0649c0e33 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:38.661118  1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a57c404b2c145bf95efa4a0649c0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.661679  1583 tablet_service.cc:1511] Processing CreateTablet for tablet fbe322e4447d486680550d1490266b17 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:00:38.663074  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe322e4447d486680550d1490266b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.667485  1582 tablet_service.cc:1511] Processing CreateTablet for tablet d4817dbc3c2e469195aa3f5e05a1534a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:00:38.669307  1582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4817dbc3c2e469195aa3f5e05a1534a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.670598  1654 tablet_service.cc:1511] Processing CreateTablet for tablet e7f90c9079e24cf2a4c9ab3dd55172ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:38.672324  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f90c9079e24cf2a4c9ab3dd55172ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.674221  1581 tablet_service.cc:1511] Processing CreateTablet for tablet 648ab3b5826e49fe81c3363d752d8af8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:00:38.675947  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ab3b5826e49fe81c3363d752d8af8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.681370  1580 tablet_service.cc:1511] Processing CreateTablet for tablet 5a57c404b2c145bf95efa4a0649c0e33 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:00:38.683432  1580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a57c404b2c145bf95efa4a0649c0e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.687211  1579 tablet_service.cc:1511] Processing CreateTablet for tablet e7f90c9079e24cf2a4c9ab3dd55172ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:38.689199  1579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f90c9079e24cf2a4c9ab3dd55172ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.693125  1728 tablet_service.cc:1511] Processing CreateTablet for tablet e7f90c9079e24cf2a4c9ab3dd55172ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:00:38.694851  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f90c9079e24cf2a4c9ab3dd55172ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.704689  1578 tablet_service.cc:1511] Processing CreateTablet for tablet 8029bb31c40d4c01bebc7c65c2301a85 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:38.706447  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8029bb31c40d4c01bebc7c65c2301a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.706367  1653 tablet_service.cc:1511] Processing CreateTablet for tablet 8029bb31c40d4c01bebc7c65c2301a85 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:38.708184  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8029bb31c40d4c01bebc7c65c2301a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.707863  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 8029bb31c40d4c01bebc7c65c2301a85 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:00:38.709393  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8029bb31c40d4c01bebc7c65c2301a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.736150  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 219129ac643e4650b611a85f565f5990 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:38.737851  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219129ac643e4650b611a85f565f5990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.738014  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 219129ac643e4650b611a85f565f5990 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:38.739472  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219129ac643e4650b611a85f565f5990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.741145  1576 tablet_service.cc:1511] Processing CreateTablet for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:38.742697  1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.746688  1731 tablet_service.cc:1511] Processing CreateTablet for tablet 219129ac643e4650b611a85f565f5990 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:00:38.761785  1651 tablet_service.cc:1511] Processing CreateTablet for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:38.762975  1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219129ac643e4650b611a85f565f5990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.763253  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:00:38.763295  1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.765094  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c15d2efd3c74bc9a5e8d3a23d50cced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.786422  1575 tablet_service.cc:1511] Processing CreateTablet for tablet bb611776ed864c47b9f843e0746b3683 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:38.788070  1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb611776ed864c47b9f843e0746b3683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.794533  1782 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:38.796449  1650 tablet_service.cc:1511] Processing CreateTablet for tablet bb611776ed864c47b9f843e0746b3683 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:38.800341  1727 tablet_service.cc:1511] Processing CreateTablet for tablet bb611776ed864c47b9f843e0746b3683 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:00:38.802665  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb611776ed864c47b9f843e0746b3683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.806444  1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb611776ed864c47b9f843e0746b3683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.809254  1574 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40d5b254644257a6d39df949366e1b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:38.810978  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40d5b254644257a6d39df949366e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.824362  1649 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40d5b254644257a6d39df949366e1b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:38.828748  1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40d5b254644257a6d39df949366e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.840848  1783 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:38.841593  1782 tablet_bootstrap.cc:654] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:38.845269  1784 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:38.850668  1730 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40d5b254644257a6d39df949366e1b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:00:38.868299  1783 tablet_bootstrap.cc:654] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:38.867743  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 24ec87af7f274132adaaf5e59ef12b7b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:38.869690  1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40d5b254644257a6d39df949366e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.879555  1784 tablet_bootstrap.cc:654] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:38.882687  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 24ec87af7f274132adaaf5e59ef12b7b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:38.884923  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec87af7f274132adaaf5e59ef12b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.885880  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec87af7f274132adaaf5e59ef12b7b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:38.898844  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:38.900141  1731 tablet_service.cc:1511] Processing CreateTablet for tablet 24ec87af7f274132adaaf5e59ef12b7b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:00:38.901688  1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24ec87af7f274132adaaf5e59ef12b7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.904559  1651 tablet_service.cc:1511] Processing CreateTablet for tablet 5a44e16fd1e74bb585aa62b3d15e7384 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:38.909248  1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a44e16fd1e74bb585aa62b3d15e7384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.909677  1574 tablet_service.cc:1511] Processing CreateTablet for tablet 5a44e16fd1e74bb585aa62b3d15e7384 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:38.911096  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a44e16fd1e74bb585aa62b3d15e7384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.918852  1782 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:38.919559  1782 ts_tablet_manager.cc:1403] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.125s	user 0.017s	sys 0.013s
I20260709 13:00:38.919866  1783 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:38.920379  1783 ts_tablet_manager.cc:1403] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.080s	user 0.015s	sys 0.029s
I20260709 13:00:38.920058  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 5a44e16fd1e74bb585aa62b3d15e7384 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:00:38.921522  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a44e16fd1e74bb585aa62b3d15e7384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.923458  1783 raft_consensus.cc:359] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.924189  1783 raft_consensus.cc:385] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:38.924489  1783 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:38.925208  1783 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.928539  1783 ts_tablet_manager.cc:1434] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:38.930639  1581 tablet_service.cc:1511] Processing CreateTablet for tablet 9f303f4bc5e84b46b9bca2a44911578b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:38.932060  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f303f4bc5e84b46b9bca2a44911578b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.931064  1618 heartbeater.cc:499] Master 127.0.107.254:39071 was elected leader, sending a full tablet report...
I20260709 13:00:38.934563  1782 raft_consensus.cc:359] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.935336  1782 raft_consensus.cc:385] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:38.935642  1782 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:38.936465  1782 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.932024  1783 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:38.941738  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 9f303f4bc5e84b46b9bca2a44911578b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:38.943297  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f303f4bc5e84b46b9bca2a44911578b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.950600  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 9f303f4bc5e84b46b9bca2a44911578b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:00:38.952076  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f303f4bc5e84b46b9bca2a44911578b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.965904  1784 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:38.966615  1784 ts_tablet_manager.cc:1403] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.122s	user 0.013s	sys 0.003s
I20260709 13:00:38.970077  1784 raft_consensus.cc:359] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.970876  1784 raft_consensus.cc:385] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:38.971203  1784 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:38.971994  1784 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:38.973938  1783 tablet_bootstrap.cc:654] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:38.974499  1649 tablet_service.cc:1511] Processing CreateTablet for tablet bf77b84dc20e4118ad47baa1808a85d9 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:38.975904  1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf77b84dc20e4118ad47baa1808a85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.974495  1575 tablet_service.cc:1511] Processing CreateTablet for tablet bf77b84dc20e4118ad47baa1808a85d9 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:38.979565  1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf77b84dc20e4118ad47baa1808a85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.979252  1731 tablet_service.cc:1511] Processing CreateTablet for tablet bf77b84dc20e4118ad47baa1808a85d9 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:00:38.981227  1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf77b84dc20e4118ad47baa1808a85d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:38.998375  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 421876549c8f408587a97543338bca1d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:38.999185  1782 ts_tablet_manager.cc:1434] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.079s	user 0.008s	sys 0.009s
I20260709 13:00:39.000448  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421876549c8f408587a97543338bca1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.009878  1767 heartbeater.cc:499] Master 127.0.107.254:39071 was elected leader, sending a full tablet report...
I20260709 13:00:39.011270  1782 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:39.018774  1782 tablet_bootstrap.cc:654] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.022490  1693 heartbeater.cc:499] Master 127.0.107.254:39071 was elected leader, sending a full tablet report...
I20260709 13:00:39.033675  1784 ts_tablet_manager.cc:1434] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.067s	user 0.012s	sys 0.017s
I20260709 13:00:39.034868  1784 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:39.041750  1784 tablet_bootstrap.cc:654] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.047950  1783 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:39.048570  1783 ts_tablet_manager.cc:1403] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.117s	user 0.011s	sys 0.023s
I20260709 13:00:39.050860  1782 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:39.051504  1782 ts_tablet_manager.cc:1403] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.040s	user 0.010s	sys 0.003s
I20260709 13:00:39.051875  1783 raft_consensus.cc:359] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.052700  1783 raft_consensus.cc:385] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.053020  1783 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.054171  1783 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.055233  1782 raft_consensus.cc:359] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.056454  1782 raft_consensus.cc:385] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.056768  1782 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.057658  1782 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.060230  1782 ts_tablet_manager.cc:1434] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:00:39.060365  1783 ts_tablet_manager.cc:1434] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:00:39.067924  1789 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.068459  1783 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:39.068653  1789 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.072388  1784 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:39.070052  1649 tablet_service.cc:1511] Processing CreateTablet for tablet 421876549c8f408587a97543338bca1d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:39.072798  1789 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:39.072934  1784 ts_tablet_manager.cc:1403] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.038s	user 0.006s	sys 0.007s
I20260709 13:00:39.074082  1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421876549c8f408587a97543338bca1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.077888  1782 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:39.079578  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 421876549c8f408587a97543338bca1d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:00:39.081045  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421876549c8f408587a97543338bca1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.081254  1783 tablet_bootstrap.cc:654] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.087333  1784 raft_consensus.cc:359] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:39.088617  1784 raft_consensus.cc:385] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.089262  1784 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.090667  1784 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:39.093735  1575 tablet_service.cc:1511] Processing CreateTablet for tablet e165279fd0cb4e84a8c9e244d087f8db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:39.096494  1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e165279fd0cb4e84a8c9e244d087f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.096629  1651 tablet_service.cc:1511] Processing CreateTablet for tablet e165279fd0cb4e84a8c9e244d087f8db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:39.097947  1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e165279fd0cb4e84a8c9e244d087f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.099080  1782 tablet_bootstrap.cc:654] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.112066  1784 ts_tablet_manager.cc:1434] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.001s
I20260709 13:00:39.114279  1784 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:39.122128  1577 tablet_service.cc:1511] Processing CreateTablet for tablet c9e456da7a864a1b97fb4c468e75980d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:39.124809  1654 tablet_service.cc:1511] Processing CreateTablet for tablet c9e456da7a864a1b97fb4c468e75980d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:39.127760  1574 tablet_service.cc:1511] Processing CreateTablet for tablet b9c6fca959014bb6a785fac033e5fe47 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:39.130112  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c6fca959014bb6a785fac033e5fe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.132848  1581 tablet_service.cc:1511] Processing CreateTablet for tablet a9c383ed9b7c4624b8fa546326e129ff (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:39.142167  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e456da7a864a1b97fb4c468e75980d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.142661  1653 tablet_service.cc:1511] Processing CreateTablet for tablet b9c6fca959014bb6a785fac033e5fe47 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:39.145673  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c6fca959014bb6a785fac033e5fe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.145953  1580 tablet_service.cc:1511] Processing CreateTablet for tablet 200c02cbe43c4af8b4c59f21f210a9bc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:39.148885  1657 tablet_service.cc:1511] Processing CreateTablet for tablet a9c383ed9b7c4624b8fa546326e129ff (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:39.150146  1788 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.150636  1788 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.152722  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383ed9b7c4624b8fa546326e129ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.152912  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e456da7a864a1b97fb4c468e75980d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.154215  1655 tablet_service.cc:1511] Processing CreateTablet for tablet 200c02cbe43c4af8b4c59f21f210a9bc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:39.155689  1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c02cbe43c4af8b4c59f21f210a9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.159017  1788 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:39.165117  1784 tablet_bootstrap.cc:654] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.175581  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383ed9b7c4624b8fa546326e129ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.181219  1731 tablet_service.cc:1511] Processing CreateTablet for tablet e165279fd0cb4e84a8c9e244d087f8db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:00:39.182952  1731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e165279fd0cb4e84a8c9e244d087f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.191529  1789 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.192127  1789 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.194711  1789 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:39.197985  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:39.199620  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5331c1cc5e4cbf8847232f0d5835c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.243503  1730 tablet_service.cc:1511] Processing CreateTablet for tablet c9e456da7a864a1b97fb4c468e75980d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:00:39.244552  1729 tablet_service.cc:1511] Processing CreateTablet for tablet 200c02cbe43c4af8b4c59f21f210a9bc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:00:39.245117  1730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e456da7a864a1b97fb4c468e75980d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.245864  1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c02cbe43c4af8b4c59f21f210a9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.247257  1732 tablet_service.cc:1511] Processing CreateTablet for tablet a9c383ed9b7c4624b8fa546326e129ff (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:00:39.248899  1732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383ed9b7c4624b8fa546326e129ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.250592  1728 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:39.251946  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5331c1cc5e4cbf8847232f0d5835c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.257274  1580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 200c02cbe43c4af8b4c59f21f210a9bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.259044  1790 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.259595  1790 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:39.261050  1725 tablet_service.cc:1511] Processing CreateTablet for tablet d66c6d05a0074fe2a7b7b2466f6182d7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:39.262557  1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c6d05a0074fe2a7b7b2466f6182d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.263479  1783 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:39.264096  1783 ts_tablet_manager.cc:1403] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.200s	user 0.013s	sys 0.008s
I20260709 13:00:39.243503  1726 tablet_service.cc:1511] Processing CreateTablet for tablet b9c6fca959014bb6a785fac033e5fe47 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:00:39.266819  1782 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:39.267350  1782 ts_tablet_manager.cc:1403] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.191s	user 0.045s	sys 0.041s
I20260709 13:00:39.267455  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c6fca959014bb6a785fac033e5fe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.270601  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 8a654178d0194a7388155af12a626d35 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:00:39.272078  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a654178d0194a7388155af12a626d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.272575  1782 raft_consensus.cc:359] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.273391  1782 raft_consensus.cc:385] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.273906  1782 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.273420  1790 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:00:39.281689  1782 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.283845  1724 tablet_service.cc:1511] Processing CreateTablet for tablet 8a654178d0194a7388155af12a626d35 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:00:39.293677  1789 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.270601  1649 tablet_service.cc:1511] Processing CreateTablet for tablet d66c6d05a0074fe2a7b7b2466f6182d7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:39.310093  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:39.311314  1650 tablet_service.cc:1511] Processing CreateTablet for tablet 22ef4803b7a94ddcaacb46f38d689c5d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:39.311846  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 22ef4803b7a94ddcaacb46f38d689c5d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:39.311707  1783 raft_consensus.cc:359] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.313097  1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ef4803b7a94ddcaacb46f38d689c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.312971  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:39.314002  1705 leader_election.cc:343] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:39.314893  1705 leader_election.cc:343] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.315125  1581 tablet_service.cc:1511] Processing CreateTablet for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:00:39.308564  1789 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.316983  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5331c1cc5e4cbf8847232f0d5835c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.315737  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a654178d0194a7388155af12a626d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.317865  1574 tablet_service.cc:1511] Processing CreateTablet for tablet d66c6d05a0074fe2a7b7b2466f6182d7 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:00:39.319234  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c6d05a0074fe2a7b7b2466f6182d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.322928  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ef4803b7a94ddcaacb46f38d689c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.312773  1783 raft_consensus.cc:385] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.325914  1783 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.330036  1789 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:39.330647  1575 tablet_service.cc:1511] Processing CreateTablet for tablet 8a654178d0194a7388155af12a626d35 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:00:39.338946  1576 tablet_service.cc:1511] Processing CreateTablet for tablet 22ef4803b7a94ddcaacb46f38d689c5d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:00:39.340075  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:39.341938  1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d66c6d05a0074fe2a7b7b2466f6182d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.343974  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:39.344249  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:39.345701  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:39.351531  1591 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:39.350476  1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a654178d0194a7388155af12a626d35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.353047  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:39.353739  1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ef4803b7a94ddcaacb46f38d689c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.368419  1784 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:39.370560  1784 ts_tablet_manager.cc:1403] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.257s	user 0.012s	sys 0.019s
I20260709 13:00:39.373031  1651 tablet_service.cc:1511] Processing CreateTablet for tablet 873c3a32540d4c2cb6d882298cb1eedc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:39.373241  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 873c3a32540d4c2cb6d882298cb1eedc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:39.374521  1790 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.374794  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873c3a32540d4c2cb6d882298cb1eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.375274  1790 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:39.375815  1705 leader_election.cc:343] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.378527  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:39.381731  1790 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:39.375018  1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873c3a32540d4c2cb6d882298cb1eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.384467  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.385494  1789 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:39.385903  1789 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.386224  1789 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:39.386551  1784 raft_consensus.cc:359] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.387399  1784 raft_consensus.cc:385] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:00:39.391134  1708 leader_election.cc:343] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.393920  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:00:39.410669  1634 leader_election.cc:343] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.412236  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:39.410346  1583 tablet_service.cc:1511] Processing CreateTablet for tablet 873c3a32540d4c2cb6d882298cb1eedc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:00:39.383467  1783 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.415165  1788 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:39.415711  1634 leader_election.cc:343] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.415772  1788 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:39.416896  1556 leader_election.cc:343] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.417681  1592 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:39.415128  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873c3a32540d4c2cb6d882298cb1eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.420830  1788 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:39.423709  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9cc5e1ffa141e295028cdd523da631 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:39.425222  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9cc5e1ffa141e295028cdd523da631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.426625  1708 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.436741  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:39.387713  1784 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.405030  1789 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.451329  1806 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.452410  1806 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:39.452796  1806 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:39.453123  1806 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:39.461834  1556 leader_election.cc:343] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.470252  1653 tablet_service.cc:1511] Processing CreateTablet for tablet 278074adc39c4830a2791129a7815557 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:39.474547  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278074adc39c4830a2791129a7815557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.476873  1784 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.479369  1784 ts_tablet_manager.cc:1434] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.108s	user 0.007s	sys 0.000s
I20260709 13:00:39.480808  1784 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:39.483003  1782 ts_tablet_manager.cc:1434] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.215s	user 0.024s	sys 0.004s
I20260709 13:00:39.491887  1784 tablet_bootstrap.cc:654] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.503537  1580 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9cc5e1ffa141e295028cdd523da631 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:39.505733  1580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9cc5e1ffa141e295028cdd523da631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.506842  1783 ts_tablet_manager.cc:1434] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.242s	user 0.008s	sys 0.004s
I20260709 13:00:39.508020  1783 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:39.512239  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:39.515496  1632 leader_election.cc:343] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.516234  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:39.516660  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:39.517719  1790 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.512207  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:39.520819  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:39.523617  1782 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:39.527987  1789 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:39.510892  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48149.3us
I20260709 13:00:39.542222  1582 tablet_service.cc:1511] Processing CreateTablet for tablet 92fb459de10a4e038d6cbee7b88dc3ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
W20260709 13:00:39.551626  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 125115us
I20260709 13:00:39.554035  1582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb459de10a4e038d6cbee7b88dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.555393  1806 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.556813  1739 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:39.556715  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:39.557926  1592 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:39.566606  1557 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.568128  1788 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:39.575595  1557 leader_election.cc:343] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.576462  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.587265  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 278074adc39c4830a2791129a7815557 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:39.589046  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278074adc39c4830a2791129a7815557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.551083  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:39.590039  1783 tablet_bootstrap.cc:654] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:39.591905  1705 leader_election.cc:343] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.592067  1732 tablet_service.cc:1511] Processing CreateTablet for tablet 7f9cc5e1ffa141e295028cdd523da631 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:00:39.594591  1732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9cc5e1ffa141e295028cdd523da631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.594964  1579 tablet_service.cc:1511] Processing CreateTablet for tablet 2dffe36aa52848e8962262b0e01bc346 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:00:39.596300  1579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dffe36aa52848e8962262b0e01bc346. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:39.605073  1632 leader_election.cc:343] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.611904  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:39.613546  1790 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.643486  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 92fb459de10a4e038d6cbee7b88dc3ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:00:39.654417  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 278074adc39c4830a2791129a7815557 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:00:39.656754  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 278074adc39c4830a2791129a7815557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.660392  1806 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:39.660670  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:39.669143  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb459de10a4e038d6cbee7b88dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.670898  1649 tablet_service.cc:1511] Processing CreateTablet for tablet 2dffe36aa52848e8962262b0e01bc346 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:00:39.670902  1578 tablet_service.cc:1511] Processing CreateTablet for tablet 4db9dd8c47ea465786759da787f2b1ab (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:39.672003  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:39.681165  1649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dffe36aa52848e8962262b0e01bc346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.677155  1782 tablet_bootstrap.cc:654] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.682091  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db9dd8c47ea465786759da787f2b1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.681963  1728 tablet_service.cc:1511] Processing CreateTablet for tablet 2dffe36aa52848e8962262b0e01bc346 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
W20260709 13:00:39.681584  1705 leader_election.cc:343] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.688472  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 4db9dd8c47ea465786759da787f2b1ab (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:39.652948  1657 tablet_service.cc:1511] Processing CreateTablet for tablet 92fb459de10a4e038d6cbee7b88dc3ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:00:39.691088  1724 tablet_service.cc:1511] Processing CreateTablet for tablet 4db9dd8c47ea465786759da787f2b1ab (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:00:39.691701  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb459de10a4e038d6cbee7b88dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.695223  1574 tablet_service.cc:1511] Processing CreateTablet for tablet eed985f1202940a1ab59ea7b7b3c745a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:39.697106  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed985f1202940a1ab59ea7b7b3c745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.696600  1651 tablet_service.cc:1511] Processing CreateTablet for tablet eed985f1202940a1ab59ea7b7b3c745a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:39.698580  1651 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed985f1202940a1ab59ea7b7b3c745a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.704427  1725 tablet_service.cc:1511] Processing CreateTablet for tablet eed985f1202940a1ab59ea7b7b3c745a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:00:39.718508  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dffe36aa52848e8962262b0e01bc346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.719679  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db9dd8c47ea465786759da787f2b1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.721493  1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eed985f1202940a1ab59ea7b7b3c745a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:39.679450  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 103934us
I20260709 13:00:39.724459  1784 tablet_bootstrap.cc:492] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:39.725190  1784 ts_tablet_manager.cc:1403] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.245s	user 0.040s	sys 0.045s
I20260709 13:00:39.728941  1784 raft_consensus.cc:359] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.729842  1784 raft_consensus.cc:385] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.730384  1784 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.731420  1784 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.736286  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db9dd8c47ea465786759da787f2b1ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.743137  1788 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.744176  1788 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.755409  1583 tablet_service.cc:1511] Processing CreateTablet for tablet 337669b46a6b4f75a35716542d1e20ae (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:39.756896  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 337669b46a6b4f75a35716542d1e20ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.758868  1593 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:39.767205  1708 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.768186  1806 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:39.768805  1806 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:39.771541  1806 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.794512  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 337669b46a6b4f75a35716542d1e20ae (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:39.794709  1576 tablet_service.cc:1511] Processing CreateTablet for tablet 026ea522d0f44f06bfab3c6eddc5e0b3 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:39.796643  1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026ea522d0f44f06bfab3c6eddc5e0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.797044  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 337669b46a6b4f75a35716542d1e20ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.794516  1783 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:39.798238  1783 ts_tablet_manager.cc:1403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.290s	user 0.039s	sys 0.033s
I20260709 13:00:39.799310  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 337669b46a6b4f75a35716542d1e20ae (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:00:39.800807  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 337669b46a6b4f75a35716542d1e20ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.801837  1653 tablet_service.cc:1511] Processing CreateTablet for tablet 026ea522d0f44f06bfab3c6eddc5e0b3 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:39.803169  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026ea522d0f44f06bfab3c6eddc5e0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.803444  1782 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:39.804243  1782 ts_tablet_manager.cc:1403] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.281s	user 0.036s	sys 0.041s
I20260709 13:00:39.808269  1783 raft_consensus.cc:359] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.813709  1783 raft_consensus.cc:385] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.814278  1783 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.813958  1782 raft_consensus.cc:359] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.791904  1592 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:39.820672  1782 raft_consensus.cc:385] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:39.824373  1784 ts_tablet_manager.cc:1434] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.099s	user 0.009s	sys 0.001s
I20260709 13:00:39.832934  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:39.834622  1806 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:39.834501  1581 tablet_service.cc:1511] Processing CreateTablet for tablet e676e54141f7470db4551f4de439c017 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:39.829030  1782 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:39.835216  1806 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:39.835868  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676e54141f7470db4551f4de439c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.836004  1782 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.836263  1806 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.834659  1650 tablet_service.cc:1511] Processing CreateTablet for tablet e676e54141f7470db4551f4de439c017 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:39.838487  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 026ea522d0f44f06bfab3c6eddc5e0b3 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:00:39.839619  1650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676e54141f7470db4551f4de439c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.840093  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 026ea522d0f44f06bfab3c6eddc5e0b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.844112  1782 ts_tablet_manager.cc:1434] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.003s
I20260709 13:00:39.845379  1782 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:39.849939  1784 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:39.855715  1488 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:39.856843  1727 tablet_service.cc:1511] Processing CreateTablet for tablet e676e54141f7470db4551f4de439c017 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:00:39.866333  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676e54141f7470db4551f4de439c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.829905  1783 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.890837  1578 tablet_service.cc:1511] Processing CreateTablet for tablet 110eba2eb7b34ea1b70ed63aca2feda8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:39.892560  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110eba2eb7b34ea1b70ed63aca2feda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.898066  1783 ts_tablet_manager.cc:1434] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.099s	user 0.007s	sys 0.002s
I20260709 13:00:39.912050  1783 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:39.921095  1782 tablet_bootstrap.cc:654] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:39.921696  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:39.924078  1784 tablet_bootstrap.cc:654] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.926108  1783 tablet_bootstrap.cc:654] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:39.931489  1790 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:39.932057  1790 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:39.938007  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:39.940155  1634 leader_election.cc:343] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.942978  1790 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:39.943333  1583 tablet_service.cc:1511] Processing CreateTablet for tablet 0444300de6924432844fa6519c92b570 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:39.944823  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0444300de6924432844fa6519c92b570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.947072  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:39.950786  1632 leader_election.cc:343] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:39.951352  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:39.952720  1790 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:39.954877  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 0444300de6924432844fa6519c92b570 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:39.956595  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0444300de6924432844fa6519c92b570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.957540  1655 tablet_service.cc:1511] Processing CreateTablet for tablet 0444300de6924432844fa6519c92b570 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:00:39.959787  1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0444300de6924432844fa6519c92b570. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.965829  1657 tablet_service.cc:1511] Processing CreateTablet for tablet 110eba2eb7b34ea1b70ed63aca2feda8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:39.967501  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110eba2eb7b34ea1b70ed63aca2feda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.971082  1724 tablet_service.cc:1511] Processing CreateTablet for tablet 110eba2eb7b34ea1b70ed63aca2feda8 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:00:39.973155  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110eba2eb7b34ea1b70ed63aca2feda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:39.993599  1581 tablet_service.cc:1511] Processing CreateTablet for tablet b8e75da90507436ea52296522f44c15e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:39.995146  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e75da90507436ea52296522f44c15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.016434  1783 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:40.017117  1783 ts_tablet_manager.cc:1403] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.105s	user 0.012s	sys 0.004s
I20260709 13:00:40.019585  1653 tablet_service.cc:1511] Processing CreateTablet for tablet b8e75da90507436ea52296522f44c15e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:40.021800  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e75da90507436ea52296522f44c15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.019586  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 162373eeadc2444e87ffd67df6a836e5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:40.024844  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162373eeadc2444e87ffd67df6a836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.027400  1728 tablet_service.cc:1511] Processing CreateTablet for tablet 162373eeadc2444e87ffd67df6a836e5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:40.028920  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162373eeadc2444e87ffd67df6a836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.031032  1727 tablet_service.cc:1511] Processing CreateTablet for tablet b8e75da90507436ea52296522f44c15e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:00:40.032269  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 162373eeadc2444e87ffd67df6a836e5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:00:40.032601  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e75da90507436ea52296522f44c15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.033336  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162373eeadc2444e87ffd67df6a836e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.036403  1783 raft_consensus.cc:359] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.059926  1782 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:40.060539  1782 ts_tablet_manager.cc:1403] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.215s	user 0.008s	sys 0.014s
I20260709 13:00:40.064535  1782 raft_consensus.cc:359] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.037477  1783 raft_consensus.cc:385] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.069227  1783 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.070339  1783 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.065346  1782 raft_consensus.cc:385] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.080411  1581 tablet_service.cc:1511] Processing CreateTablet for tablet f5a2ef920c894818b430473b4809d61d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:40.082907  1784 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:40.083942  1784 ts_tablet_manager.cc:1403] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.239s	user 0.029s	sys 0.023s
I20260709 13:00:40.075925  1782 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.091298  1578 tablet_service.cc:1511] Processing CreateTablet for tablet 9768bad280eb4cdf8f1a090752fc5ddc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:40.091554  1654 tablet_service.cc:1511] Processing CreateTablet for tablet f5a2ef920c894818b430473b4809d61d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:40.092979  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a2ef920c894818b430473b4809d61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.094635  1783 ts_tablet_manager.cc:1434] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.077s	user 0.010s	sys 0.003s
I20260709 13:00:40.095482  1723 tablet_service.cc:1511] Processing CreateTablet for tablet f5a2ef920c894818b430473b4809d61d (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:00:40.097039  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a2ef920c894818b430473b4809d61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.103161  1784 raft_consensus.cc:359] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.104135  1784 raft_consensus.cc:385] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.109076  1783 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:40.110095  1782 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.110136  1784 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.112112  1784 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.113104  1782 ts_tablet_manager.cc:1434] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.052s	user 0.007s	sys 0.002s
I20260709 13:00:40.084462  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a2ef920c894818b430473b4809d61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.114557  1788 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.115141  1782 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:40.115036  1788 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.116811  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 9768bad280eb4cdf8f1a090752fc5ddc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:40.118949  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9768bad280eb4cdf8f1a090752fc5ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.119592  1788 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:40.119433  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:40.121891  1556 leader_election.cc:343] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.119433  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:40.123658  1557 leader_election.cc:343] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.124135  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:40.125177  1784 ts_tablet_manager.cc:1434] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.001s
I20260709 13:00:40.125952  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9768bad280eb4cdf8f1a090752fc5ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.148075  1724 tablet_service.cc:1511] Processing CreateTablet for tablet 9768bad280eb4cdf8f1a090752fc5ddc (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:00:40.149634  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9768bad280eb4cdf8f1a090752fc5ddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.126235  1788 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:40.150810  1583 tablet_service.cc:1511] Processing CreateTablet for tablet c89f1823e773445c99646fed221320df (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:40.149403  1488 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:40.152588  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89f1823e773445c99646fed221320df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.155243  1782 tablet_bootstrap.cc:654] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.159129  1806 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.160275  1806 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.130538  1783 tablet_bootstrap.cc:654] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.166596  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:40.167835  1806 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.168886  1784 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:40.169363  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:40.171358  1790 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.171826  1790 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.174120  1790 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:40.174774  1705 leader_election.cc:343] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.176712  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:40.171366  1592 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:40.177735  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:40.179329  1634 leader_election.cc:343] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.179284  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:40.180815  1806 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:40.181231  1806 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.183270  1806 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.183149  1652 tablet_service.cc:1511] Processing CreateTablet for tablet c89f1823e773445c99646fed221320df (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:40.184777  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89f1823e773445c99646fed221320df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.183229  1788 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.185856  1788 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.188067  1788 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.188392  1728 tablet_service.cc:1511] Processing CreateTablet for tablet c89f1823e773445c99646fed221320df (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:00:40.189816  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c89f1823e773445c99646fed221320df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.189769  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:40.190562  1740 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:40.191922  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:40.194203  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:40.194948  1668 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:40.197294  1790 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.197888  1790 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.200331  1790 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:40.201587  1632 leader_election.cc:343] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.202171  1632 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:40.208879  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:40.209012  1807 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:40.210929  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:40.211879  1593 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:40.213430  1784 tablet_bootstrap.cc:654] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.213799  1809 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:40.219228  1809 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.219622  1809 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.219434  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:40.218561  1790 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:40.221032  1806 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.224623  1790 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.222393  1581 tablet_service.cc:1511] Processing CreateTablet for tablet 93491efa22a449f3aa27354cf02d7047 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:40.226166  1806 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.227607  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:40.234491  1782 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
W20260709 13:00:40.234577  1705 leader_election.cc:343] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:40.235181  1782 ts_tablet_manager.cc:1403] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.120s	user 0.013s	sys 0.013s
I20260709 13:00:40.229560  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:40.235606  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93491efa22a449f3aa27354cf02d7047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.235838  1593 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.240229  1578 tablet_service.cc:1511] Processing CreateTablet for tablet d857fff084d3451d99a39db56ccd9a37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:40.242544  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857fff084d3451d99a39db56ccd9a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.231757  1809 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.227927  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 93491efa22a449f3aa27354cf02d7047 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:40.251250  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93491efa22a449f3aa27354cf02d7047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.262352  1728 tablet_service.cc:1511] Processing CreateTablet for tablet d857fff084d3451d99a39db56ccd9a37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:40.262687  1657 tablet_service.cc:1511] Processing CreateTablet for tablet d857fff084d3451d99a39db56ccd9a37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:00:40.263906  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857fff084d3451d99a39db56ccd9a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.263908  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857fff084d3451d99a39db56ccd9a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.264384  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:40.265071  1740 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.274335  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 0621716cd2fa4c0cb66038edb4352f37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:40.276355  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0621716cd2fa4c0cb66038edb4352f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.275830  1809 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.282938  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:40.284253  1668 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.287809  1806 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:40.290654  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 93491efa22a449f3aa27354cf02d7047 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:00:40.292265  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93491efa22a449f3aa27354cf02d7047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.306139  1576 tablet_service.cc:1511] Processing CreateTablet for tablet dfe67343bf1842888d6176ed2ffe1419 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:40.304934  1653 tablet_service.cc:1511] Processing CreateTablet for tablet 0621716cd2fa4c0cb66038edb4352f37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:40.309731  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0621716cd2fa4c0cb66038edb4352f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.310807  1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe67343bf1842888d6176ed2ffe1419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.311303  1654 tablet_service.cc:1511] Processing CreateTablet for tablet dfe67343bf1842888d6176ed2ffe1419 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:40.311900  1790 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:40.312630  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe67343bf1842888d6176ed2ffe1419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.315577  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 0621716cd2fa4c0cb66038edb4352f37 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:00:40.316907  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0621716cd2fa4c0cb66038edb4352f37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.318797  1783 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:40.319393  1783 ts_tablet_manager.cc:1403] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.213s	user 0.020s	sys 0.007s
I20260709 13:00:40.326718  1740 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:40.328150  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:40.329222  1809 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:40.329612  1809 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:40.330376  1809 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } }
I20260709 13:00:40.333348  1724 tablet_service.cc:1511] Processing CreateTablet for tablet dfe67343bf1842888d6176ed2ffe1419 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:00:40.335047  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe67343bf1842888d6176ed2ffe1419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.309134  1593 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:40.338773  1784 tablet_bootstrap.cc:492] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:40.346609  1784 ts_tablet_manager.cc:1403] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.178s	user 0.013s	sys 0.009s
I20260709 13:00:40.348868  1783 raft_consensus.cc:359] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.350807  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:40.357026  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 422771fe34aa4b29ba2702785d38559a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:40.358806  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422771fe34aa4b29ba2702785d38559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.359759  1807 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:40.359918  1784 raft_consensus.cc:359] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.360594  1807 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:40.360858  1784 raft_consensus.cc:385] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.361193  1784 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.361738  1807 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.362092  1784 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:40.362897  1705 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:40.365041  1784 ts_tablet_manager.cc:1434] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.004s
I20260709 13:00:40.370694  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 422771fe34aa4b29ba2702785d38559a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:40.352723  1783 raft_consensus.cc:385] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.372089  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422771fe34aa4b29ba2702785d38559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.382478  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 4e74a6fd31c14c17a7248fe6eb410c77 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:40.383992  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e74a6fd31c14c17a7248fe6eb410c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.384482  1655 tablet_service.cc:1511] Processing CreateTablet for tablet 7dc3659523cf47fa8918889f5129c8ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:40.385854  1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc3659523cf47fa8918889f5129c8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.389460  1725 tablet_service.cc:1511] Processing CreateTablet for tablet 7dc3659523cf47fa8918889f5129c8ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:40.390914  1725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc3659523cf47fa8918889f5129c8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.396435  1583 tablet_service.cc:1511] Processing CreateTablet for tablet 422771fe34aa4b29ba2702785d38559a (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:00:40.398097  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 422771fe34aa4b29ba2702785d38559a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.367216  1668 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:40.417201  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 4e74a6fd31c14c17a7248fe6eb410c77 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:40.418614  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e74a6fd31c14c17a7248fe6eb410c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.374550  1783 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.419808  1579 tablet_service.cc:1511] Processing CreateTablet for tablet 4e74a6fd31c14c17a7248fe6eb410c77 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:00:40.421236  1579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e74a6fd31c14c17a7248fe6eb410c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.420192  1783 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.423424  1783 ts_tablet_manager.cc:1434] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.104s	user 0.008s	sys 0.000s
I20260709 13:00:40.423249  1806 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:40.424431  1783 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:40.425642  1580 tablet_service.cc:1511] Processing CreateTablet for tablet 88e78eac7b5f4006a15379f62bc8c536 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:40.423156  1574 tablet_service.cc:1511] Processing CreateTablet for tablet 7dc3659523cf47fa8918889f5129c8ba (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:00:40.429387  1654 tablet_service.cc:1511] Processing CreateTablet for tablet 88e78eac7b5f4006a15379f62bc8c536 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:40.434942  1654 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88e78eac7b5f4006a15379f62bc8c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.368543  1782 raft_consensus.cc:359] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.436653  1491 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:40.446538  1580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88e78eac7b5f4006a15379f62bc8c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.451022  1790 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.451619  1790 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.448410  1489 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:40.455174  1809 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.455787  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:40.455947  1809 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.458158  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 88e78eac7b5f4006a15379f62bc8c536 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:00:40.459496  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88e78eac7b5f4006a15379f62bc8c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.459726  1809 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:40.460208  1784 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:40.460675  1591 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:40.463699  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:40.465837  1824 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:40.466475  1824 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.466920  1824 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.473582  1790 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:40.475467  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:40.476164  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:40.478549  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:40.480499  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:40.481180  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:40.482146  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 882fdd7ea10640a78914e0647598ad9b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:40.483424  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fdd7ea10640a78914e0647598ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.467978  1782 raft_consensus.cc:385] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.522680  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:40.501029  1809 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:40.515611  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc3659523cf47fa8918889f5129c8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.526470  1809 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.527892  1809 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.539232  1824 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.540777  1784 tablet_bootstrap.cc:654] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.542423  1668 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:40.542416  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:40.542888  1732 tablet_service.cc:1511] Processing CreateTablet for tablet 882fdd7ea10640a78914e0647598ad9b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:40.544852  1732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fdd7ea10640a78914e0647598ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.544947  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:40.545713  1740 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.516337  1782 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.555060  1578 tablet_service.cc:1511] Processing CreateTablet for tablet 249474541a174b90a099d77784ed4801 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:40.558629  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 249474541a174b90a099d77784ed4801 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:40.561069  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249474541a174b90a099d77784ed4801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.551146  1824 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:40.563963  1782 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.566677  1728 tablet_service.cc:1511] Processing CreateTablet for tablet 249474541a174b90a099d77784ed4801 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:00:40.568225  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249474541a174b90a099d77784ed4801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.592909  1583 tablet_service.cc:1511] Processing CreateTablet for tablet 87a64e3accd242188d2126865daef114 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:40.593008  1657 tablet_service.cc:1511] Processing CreateTablet for tablet 87a64e3accd242188d2126865daef114 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:40.595258  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a64e3accd242188d2126865daef114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.613583  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 9a860695ab4c493b83dfcd08d8bee8de (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:40.616993  1726 tablet_service.cc:1511] Processing CreateTablet for tablet 87a64e3accd242188d2126865daef114 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:00:40.601063  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:40.618817  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 249474541a174b90a099d77784ed4801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.633986  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:40.634164  1724 tablet_service.cc:1511] Processing CreateTablet for tablet 9a860695ab4c493b83dfcd08d8bee8de (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:40.634979  1581 tablet_service.cc:1511] Processing CreateTablet for tablet ce79fa9e7595481c8379fd2bdeca8a6e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:40.635437  1824 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:40.636118  1824 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:40.636013  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 9a860695ab4c493b83dfcd08d8bee8de (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:00:40.636119  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce79fa9e7595481c8379fd2bdeca8a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.637243  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a860695ab4c493b83dfcd08d8bee8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.637960  1653 tablet_service.cc:1511] Processing CreateTablet for tablet ce79fa9e7595481c8379fd2bdeca8a6e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:40.639287  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce79fa9e7595481c8379fd2bdeca8a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.637259  1824 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.648501  1726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a64e3accd242188d2126865daef114. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:00:40.650279  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47894.2us
I20260709 13:00:40.660845  1724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a860695ab4c493b83dfcd08d8bee8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.658823  1583 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a64e3accd242188d2126865daef114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.663766  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a860695ab4c493b83dfcd08d8bee8de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.663986  1727 tablet_service.cc:1511] Processing CreateTablet for tablet ce79fa9e7595481c8379fd2bdeca8a6e (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:00:40.666246  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce79fa9e7595481c8379fd2bdeca8a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.683127  1783 tablet_bootstrap.cc:654] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.706827  1581 tablet_service.cc:1511] Processing CreateTablet for tablet 5ae371f2c53648c2992a835477d85184 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:40.708319  1581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae371f2c53648c2992a835477d85184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.689952  1576 tablet_service.cc:1511] Processing CreateTablet for tablet 882fdd7ea10640a78914e0647598ad9b (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:00:40.719292  1576 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fdd7ea10640a78914e0647598ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.722007  1782 ts_tablet_manager.cc:1434] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.465s	user 0.011s	sys 0.012s
I20260709 13:00:40.721710  1788 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:40.723234  1782 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:40.761061  1784 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:40.761714  1784 ts_tablet_manager.cc:1403] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.328s	user 0.019s	sys 0.011s
I20260709 13:00:40.767320  1784 raft_consensus.cc:359] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.768221  1784 raft_consensus.cc:385] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.768543  1784 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.769161  1809 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.771307  1591 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:40.778654  1655 tablet_service.cc:1511] Processing CreateTablet for tablet 5ae371f2c53648c2992a835477d85184 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:40.780212  1655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae371f2c53648c2992a835477d85184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.812856  1783 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:40.826257  1491 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:40.813875  1657 tablet_service.cc:1511] Processing CreateTablet for tablet 567113a84c1d465597e948caf9276c34 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:40.817406  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:40.819115  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:40.833443  1740 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:40.818946  1723 tablet_service.cc:1511] Processing CreateTablet for tablet 567113a84c1d465597e948caf9276c34 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:40.835439  1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567113a84c1d465597e948caf9276c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.836169  1657 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567113a84c1d465597e948caf9276c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.838502  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:40.839450  1788 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:40.818946  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 5ae371f2c53648c2992a835477d85184 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:00:40.840956  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae371f2c53648c2992a835477d85184. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.810216  1809 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:40.831212  1782 tablet_bootstrap.cc:654] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.831248  1579 tablet_service.cc:1511] Processing CreateTablet for tablet 567113a84c1d465597e948caf9276c34 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:00:40.832271  1574 tablet_service.cc:1511] Processing CreateTablet for tablet ce1eb834927c4d3caa029344d6f69de5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:40.792411  1784 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.848834  1574 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1eb834927c4d3caa029344d6f69de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.849637  1653 tablet_service.cc:1511] Processing CreateTablet for tablet ce1eb834927c4d3caa029344d6f69de5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:40.859355  1652 tablet_service.cc:1511] Processing CreateTablet for tablet 01669b686f2848abb4ec03f67aa748ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:40.863956  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01669b686f2848abb4ec03f67aa748ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.861145  1807 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:40.864087  1577 tablet_service.cc:1511] Processing CreateTablet for tablet 01669b686f2848abb4ec03f67aa748ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:40.865247  1830 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.865809  1830 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.867988  1732 tablet_service.cc:1511] Processing CreateTablet for tablet ce1eb834927c4d3caa029344d6f69de5 (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:00:40.868932  1830 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.870474  1653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1eb834927c4d3caa029344d6f69de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.870949  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:40.871975  1666 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:40.874434  1705 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:40.875612  1830 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:40.876108  1830 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:40.876565  1830 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.878911  1579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 567113a84c1d465597e948caf9276c34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.842195  1783 ts_tablet_manager.cc:1403] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.418s	user 0.016s	sys 0.014s
I20260709 13:00:40.894280  1783 raft_consensus.cc:359] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.895047  1783 raft_consensus.cc:385] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:40.895340  1783 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:40.895550  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:40.896113  1783 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.897008  1593 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:40.908762  1788 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:40.913178  1732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1eb834927c4d3caa029344d6f69de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.897562  1830 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.941485  1577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01669b686f2848abb4ec03f67aa748ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.948201  1784 ts_tablet_manager.cc:1434] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.186s	user 0.007s	sys 0.005s
I20260709 13:00:40.963111  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:40.963822  1784 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:40.963876  1593 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.952688  1727 tablet_service.cc:1511] Processing CreateTablet for tablet 01669b686f2848abb4ec03f67aa748ea (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:00:40.966323  1727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01669b686f2848abb4ec03f67aa748ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:40.970217  1830 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:40.972719  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:40.973013  1593 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:40.973559  1667 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:40.974733  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:40.975881  1830 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:40.977258  1830 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:40.977771  1790 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:40.978440  1784 tablet_bootstrap.cc:654] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:40.978204  1830 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:40.978271  1790 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:40.991267  1705 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:41.002406  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:41.007326  1783 ts_tablet_manager.cc:1434] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.116s	user 0.007s	sys 0.000s
I20260709 13:00:41.008949  1783 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:40.993296  1790 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:41.051049  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:41.059885  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.064476  1578 tablet_service.cc:1511] Processing CreateTablet for tablet e223adb74e71405ea633f364ffa8b6db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:41.067760  1578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e223adb74e71405ea633f364ffa8b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:41.070262  1634 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:41.079406  1652 tablet_service.cc:1511] Processing CreateTablet for tablet e223adb74e71405ea633f364ffa8b6db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:41.086131  1652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e223adb74e71405ea633f364ffa8b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:41.108982  1824 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:41.121472  1790 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:41.122651  1782 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.123597  1782 ts_tablet_manager.cc:1403] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.401s	user 0.034s	sys 0.009s
I20260709 13:00:41.124317  1728 tablet_service.cc:1511] Processing CreateTablet for tablet e223adb74e71405ea633f364ffa8b6db (DEFAULT_TABLE table=test-workload [id=680582cffa864e89afc37eaf28677d7f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:00:41.137620  1790 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:41.145212  1728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e223adb74e71405ea633f364ffa8b6db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:41.154356  1784 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.155067  1784 ts_tablet_manager.cc:1403] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.192s	user 0.012s	sys 0.008s
I20260709 13:00:41.162735  1667 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:41.164433  1790 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.166206  1783 tablet_bootstrap.cc:654] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.165256  1790 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.174409  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:41.176887  1634 leader_election.cc:343] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:41.176824  1784 raft_consensus.cc:359] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.177688  1784 raft_consensus.cc:385] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.177973  1784 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.178170  1790 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.180711  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:41.184924  1632 leader_election.cc:343] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:41.184830  1782 raft_consensus.cc:359] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.185698  1782 raft_consensus.cc:385] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.185563  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:41.186883  1782 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.198119  1784 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.224946  1488 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:41.229005  1790 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.229766  1790 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.238673  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.241024  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:41.241439  1634 leader_election.cc:343] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:41.243237  1632 leader_election.cc:343] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:41.244503  1790 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.247798  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:41.249190  1851 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:41.250299  1851 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:41.251788  1782 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.258961  1782 ts_tablet_manager.cc:1434] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.135s	user 0.010s	sys 0.000s
I20260709 13:00:41.265666  1783 tablet_bootstrap.cc:492] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.268057  1782 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.269774  1783 ts_tablet_manager.cc:1403] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.261s	user 0.013s	sys 0.025s
I20260709 13:00:41.274287  1783 raft_consensus.cc:359] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.277669  1783 raft_consensus.cc:385] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.278614  1782 tablet_bootstrap.cc:654] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.278640  1783 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.284142  1783 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.288343  1783 ts_tablet_manager.cc:1434] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 13:00:41.289883  1783 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.299983  1783 tablet_bootstrap.cc:654] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.318429  1784 ts_tablet_manager.cc:1434] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.163s	user 0.014s	sys 0.037s
I20260709 13:00:41.321327  1782 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.322145  1782 ts_tablet_manager.cc:1403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.054s	user 0.019s	sys 0.019s
I20260709 13:00:41.322698  1784 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.326095  1782 raft_consensus.cc:359] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.327179  1782 raft_consensus.cc:385] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.327543  1782 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.328318  1782 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.330554  1783 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.330780  1784 tablet_bootstrap.cc:654] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.331207  1782 ts_tablet_manager.cc:1434] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 13:00:41.331694  1783 ts_tablet_manager.cc:1403] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.042s	user 0.013s	sys 0.025s
I20260709 13:00:41.332670  1782 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.335793  1783 raft_consensus.cc:359] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.336637  1783 raft_consensus.cc:385] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.336973  1783 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.337929  1783 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.342427  1782 tablet_bootstrap.cc:654] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.345917  1783 ts_tablet_manager.cc:1434] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 13:00:41.348209  1783 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.355109  1783 tablet_bootstrap.cc:654] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.359733  1784 tablet_bootstrap.cc:492] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.360455  1784 ts_tablet_manager.cc:1403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.038s	user 0.021s	sys 0.013s
I20260709 13:00:41.364826  1784 raft_consensus.cc:359] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.365873  1784 raft_consensus.cc:385] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.366246  1784 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.367343  1784 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.369825  1783 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.370456  1783 ts_tablet_manager.cc:1403] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.007s
I20260709 13:00:41.372114  1784 ts_tablet_manager.cc:1434] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.004s
I20260709 13:00:41.373144  1782 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.373339  1784 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.373687  1782 ts_tablet_manager.cc:1403] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.041s	user 0.021s	sys 0.019s
I20260709 13:00:41.374395  1783 raft_consensus.cc:359] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.375605  1783 raft_consensus.cc:385] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.375970  1783 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.376940  1782 raft_consensus.cc:359] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.377712  1782 raft_consensus.cc:385] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.378021  1782 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.378778  1782 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.380573  1667 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.381119  1782 ts_tablet_manager.cc:1434] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:00:41.381471  1784 tablet_bootstrap.cc:654] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.385340  1782 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.392562  1784 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.393119  1784 ts_tablet_manager.cc:1403] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.010s
I20260709 13:00:41.393445  1782 tablet_bootstrap.cc:654] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.376891  1783 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.396210  1784 raft_consensus.cc:359] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.396899  1784 raft_consensus.cc:385] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.397166  1784 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.397979  1784 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.400899  1783 ts_tablet_manager.cc:1434] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.030s	user 0.002s	sys 0.004s
I20260709 13:00:41.402103  1783 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.407526  1784 ts_tablet_manager.cc:1434] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.000s
I20260709 13:00:41.408406  1782 tablet_bootstrap.cc:492] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.408901  1782 ts_tablet_manager.cc:1403] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.024s	user 0.019s	sys 0.002s
I20260709 13:00:41.409041  1784 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.412564  1782 raft_consensus.cc:359] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.413436  1782 raft_consensus.cc:385] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.413924  1782 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.414693  1782 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.417325  1782 ts_tablet_manager.cc:1434] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 13:00:41.418931  1782 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.419642  1783 tablet_bootstrap.cc:654] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.424768  1809 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.425400  1809 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.426147  1782 tablet_bootstrap.cc:654] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.427090  1784 tablet_bootstrap.cc:654] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.428054  1809 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.431068  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.431411  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:41.431800  1666 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:41.431967  1739 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:41.433355  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:41.437579  1809 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.441768  1809 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.442265  1809 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.443213  1782 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.444996  1782 ts_tablet_manager.cc:1403] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.026s	user 0.006s	sys 0.013s
I20260709 13:00:41.446849  1784 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.447543  1784 ts_tablet_manager.cc:1403] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.039s	user 0.018s	sys 0.000s
I20260709 13:00:41.448321  1782 raft_consensus.cc:359] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.449075  1782 raft_consensus.cc:385] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.449357  1782 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.450134  1782 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.452561  1782 ts_tablet_manager.cc:1434] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:41.455529  1784 raft_consensus.cc:359] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.456586  1784 raft_consensus.cc:385] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.456954  1784 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.458371  1782 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.460778  1784 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.458055  1809 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.467856  1784 ts_tablet_manager.cc:1434] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 13:00:41.469224  1784 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.470738  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:41.471616  1666 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.473513  1782 tablet_bootstrap.cc:654] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.475656  1809 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.475831  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:41.476641  1739 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.485059  1739 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:41.486586  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:41.487699  1809 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:41.488230  1809 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:41.490784  1856 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.491797  1856 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.493705  1666 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:41.496888  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.499761  1809 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:41.498994  1634 leader_election.cc:343] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:41.501967  1783 tablet_bootstrap.cc:492] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.501853  1830 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:41.502703  1783 ts_tablet_manager.cc:1403] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.101s	user 0.015s	sys 0.019s
I20260709 13:00:41.505457  1824 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.506524  1824 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.511533  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.512149  1856 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.512614  1593 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:41.521406  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:41.522581  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:41.524787  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:41.527554  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:41.529066  1853 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.529655  1853 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.530086  1853 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.534698  1830 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.543397  1784 tablet_bootstrap.cc:654] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.546831  1830 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.547598  1783 raft_consensus.cc:359] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.548461  1783 raft_consensus.cc:385] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.548766  1783 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.543926  1807 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:00:41.537632  1856 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.550716  1856 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.551157  1856 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.553673  1782 tablet_bootstrap.cc:492] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.554215  1782 ts_tablet_manager.cc:1403] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.096s	user 0.018s	sys 0.026s
I20260709 13:00:41.554167  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:41.550740  1783 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.554939  1739 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:41.556243  1853 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.561936  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.556922  1824 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.562700  1667 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:41.557453  1782 raft_consensus.cc:359] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.559244  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.563699  1592 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:41.564257  1783 ts_tablet_manager.cc:1434] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.061s	user 0.008s	sys 0.002s
I20260709 13:00:41.564512  1782 raft_consensus.cc:385] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.564815  1782 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.565619  1782 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.566550  1834 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.567442  1782 ts_tablet_manager.cc:1434] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 13:00:41.567076  1834 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.568619  1782 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.569666  1667 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.570557  1784 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.571058  1784 ts_tablet_manager.cc:1403] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.102s	user 0.010s	sys 0.005s
I20260709 13:00:41.573876  1834 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:41.565855  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:41.574596  1830 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:41.585567  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:41.587275  1592 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.587558  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:41.588158  1739 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.586263  1783 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.579061  1784 raft_consensus.cc:359] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.594147  1853 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.595485  1784 raft_consensus.cc:385] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.595929  1784 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.597060  1784 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.583894  1488 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:41.601557  1784 ts_tablet_manager.cc:1434] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.001s
I20260709 13:00:41.602255  1592 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:41.602816  1784 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.603696  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:41.604506  1853 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:41.604961  1853 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:41.597731  1830 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.606196  1830 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.595607  1782 tablet_bootstrap.cc:654] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.606599  1830 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.606426  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.605815  1853 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.599022  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:41.612669  1708 leader_election.cc:343] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:41.613812  1705 leader_election.cc:343] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:41.614249  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:41.617821  1807 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:41.609989  1856 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.625387  1830 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.633740  1830 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:41.659865  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:41.682056  1784 tablet_bootstrap.cc:654] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.669395  1856 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.686184  1788 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.686862  1788 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.677304  1783 tablet_bootstrap.cc:654] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.690500  1788 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.699858  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:41.700493  1593 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.701076  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:41.706584  1592 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.707306  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:41.712333  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:41.716073  1740 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:41.717336  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:41.718896  1788 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.719488  1788 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.719923  1593 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:41.719975  1788 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.714892  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.721551  1667 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:41.720235  1488 catalog_manager.cc:5697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:41.729030  1782 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.729766  1782 ts_tablet_manager.cc:1403] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.161s	user 0.016s	sys 0.003s
I20260709 13:00:41.730994  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:41.732810  1856 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:41.734508  1782 raft_consensus.cc:359] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.735126  1782 raft_consensus.cc:385] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.733353  1856 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:41.735419  1782 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.736038  1856 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.736580  1782 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.726223  1739 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.744632  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:41.745914  1667 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.756491  1782 ts_tablet_manager.cc:1434] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260709 13:00:41.759452  1782 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.769088  1783 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.772996  1783 ts_tablet_manager.cc:1403] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.187s	user 0.013s	sys 0.005s
I20260709 13:00:41.769629  1788 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.782547  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:41.783178  1782 tablet_bootstrap.cc:654] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.783258  1666 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.783556  1784 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.784428  1784 ts_tablet_manager.cc:1403] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.182s	user 0.047s	sys 0.053s
I20260709 13:00:41.787259  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:41.787906  1740 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.783465  1788 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:41.783319  1783 raft_consensus.cc:359] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.792902  1783 raft_consensus.cc:385] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.793201  1783 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.790618  1667 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:41.795727  1784 raft_consensus.cc:359] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.798411  1784 raft_consensus.cc:385] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.798717  1784 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.799400  1784 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.801756  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:41.802757  1592 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:41.803356  1783 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.804209  1830 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:41.804811  1830 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:41.807487  1830 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.811177  1489 catalog_manager.cc:5697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:41.813081  1739 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:41.813433  1740 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:41.814627  1784 ts_tablet_manager.cc:1434] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.001s
I20260709 13:00:41.815461  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:41.817956  1788 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:41.818472  1788 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:41.819219  1784 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.821676  1783 ts_tablet_manager.cc:1434] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.003s
I20260709 13:00:41.822929  1666 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:41.823063  1788 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.830439  1783 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.834420  1782 tablet_bootstrap.cc:492] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.835253  1782 ts_tablet_manager.cc:1403] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.078s	user 0.016s	sys 0.002s
I20260709 13:00:41.842468  1782 raft_consensus.cc:359] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.843405  1782 raft_consensus.cc:385] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.843724  1782 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.844056  1784 tablet_bootstrap.cc:654] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.834489  1490 catalog_manager.cc:5697] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:41.848402  1782 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.852005  1782 ts_tablet_manager.cc:1434] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.002s
I20260709 13:00:41.859267  1783 tablet_bootstrap.cc:654] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.858894  1489 catalog_manager.cc:5697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:41.853436  1782 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.866464  1783 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:41.866992  1783 ts_tablet_manager.cc:1403] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.001s
I20260709 13:00:41.871361  1782 tablet_bootstrap.cc:654] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.874231  1783 raft_consensus.cc:359] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.876623  1783 raft_consensus.cc:385] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.877322  1783 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.878618  1783 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.880795  1783 ts_tablet_manager.cc:1434] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 13:00:41.884195  1784 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:41.884824  1784 ts_tablet_manager.cc:1403] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.067s	user 0.013s	sys 0.015s
I20260709 13:00:41.885794  1783 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:41.887398  1782 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:41.887861  1782 ts_tablet_manager.cc:1403] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.035s	user 0.006s	sys 0.011s
I20260709 13:00:41.889444  1830 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.890041  1830 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.890839  1782 raft_consensus.cc:359] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.892002  1782 raft_consensus.cc:385] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.892683  1782 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.892798  1784 raft_consensus.cc:359] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.893838  1784 raft_consensus.cc:385] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:41.894256  1784 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:41.893713  1782 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.924580  1788 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:41.943279  1783 tablet_bootstrap.cc:654] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.943598  1784 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.944278  1782 ts_tablet_manager.cc:1434] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.056s	user 0.014s	sys 0.020s
I20260709 13:00:41.945910  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:41.947032  1592 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:41.947640  1784 ts_tablet_manager.cc:1434] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.062s	user 0.016s	sys 0.011s
I20260709 13:00:41.948841  1830 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:41.950678  1784 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:41.950490  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:41.955325  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.956094  1666 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:41.961335  1788 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:41.961860  1784 tablet_bootstrap.cc:654] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.963472  1834 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:41.964865  1834 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:41.965374  1834 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:41.978636  1835 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:41.979276  1835 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:41.981264  1782 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:41.982368  1835 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:41.984876  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:41.985599  1667 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:41.999444  1782 tablet_bootstrap.cc:654] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:41.994117  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.003118  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.006428  1783 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.006953  1783 ts_tablet_manager.cc:1403] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.121s	user 0.033s	sys 0.075s
I20260709 13:00:42.008047  1830 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.009055  1830 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.009469  1830 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:42.009482  1708 leader_election.cc:343] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.016513  1834 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.020787  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:42.021581  1592 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.022874  1783 raft_consensus.cc:359] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.023732  1783 raft_consensus.cc:385] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.024081  1783 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.024935  1783 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.027549  1783 ts_tablet_manager.cc:1434] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 13:00:42.028837  1783 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.031492  1592 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:42.033846  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:42.034838  1835 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.034796  1830 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.035600  1835 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:42.045995  1830 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.047217  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:42.055778  1856 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.036953  1835 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:42.057003  1708 leader_election.cc:343] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.054226  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.078601  1782 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.079524  1782 ts_tablet_manager.cc:1403] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.099s	user 0.014s	sys 0.007s
I20260709 13:00:42.080861  1666 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.092337  1834 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.116874  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.117995  1668 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.122686  1784 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:42.123554  1784 ts_tablet_manager.cc:1403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.173s	user 0.011s	sys 0.022s
I20260709 13:00:42.139770  1782 raft_consensus.cc:359] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.141877  1782 raft_consensus.cc:385] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.142108  1782 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.143102  1782 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.145536  1782 ts_tablet_manager.cc:1434] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.003s
I20260709 13:00:42.145457  1784 raft_consensus.cc:359] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.151398  1784 raft_consensus.cc:385] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.151944  1784 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
W20260709 13:00:42.154819  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:42.161669  1856 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.162282  1856 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.165611  1856 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.165850  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:42.174307  1782 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.174849  1739 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:42.166750  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.176386  1593 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:42.178146  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.168941  1783 tablet_bootstrap.cc:654] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.172727  1853 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.179394  1856 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.179766  1856 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.184278  1856 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.173990  1784 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.189725  1788 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.174039  1666 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:42.190558  1788 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.194806  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:42.194895  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.195881  1807 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.193413  1788 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.196483  1807 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:42.193396  1853 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:00:42.197608  1807 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.198249  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:42.200908  1784 ts_tablet_manager.cc:1434] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.077s	user 0.009s	sys 0.001s
I20260709 13:00:42.210887  1784 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
W20260709 13:00:42.208092  1556 leader_election.cc:343] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.209197  1740 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:42.264688  1898 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.299212  1898 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.223348  1489 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:42.304768  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.288393  1782 tablet_bootstrap.cc:654] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.306747  1783 tablet_bootstrap.cc:492] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.220829  1856 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.307293  1783 ts_tablet_manager.cc:1403] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.279s	user 0.020s	sys 0.017s
W20260709 13:00:42.308636  1634 leader_election.cc:343] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.309108  1856 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.309751  1856 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:42.310487  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 106593us
I20260709 13:00:42.311038  1783 raft_consensus.cc:359] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.311190  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.319294  1783 raft_consensus.cc:385] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.319643  1783 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.320640  1788 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.321066  1788 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.321460  1788 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.321393  1783 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.325748  1783 ts_tablet_manager.cc:1434] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 13:00:42.321730  1809 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:00:42.326902  1783 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.329572  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:42.329231  1807 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.330190  1742 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.312870  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:42.336808  1898 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.337113  1592 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.317137  1668 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:42.343099  1742 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:42.358784  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.360363  1898 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.361079  1898 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:42.365422  1807 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.362382  1898 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.362318  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:42.380487  1632 leader_election.cc:343] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.377818  1788 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.383834  1809 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:42.381946  1632 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:42.386085  1788 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.387259  1853 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:42.410708  1782 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.411273  1782 ts_tablet_manager.cc:1403] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.260s	user 0.023s	sys 0.026s
I20260709 13:00:42.412379  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.414486  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:42.414382  1488 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:42.415416  1782 raft_consensus.cc:359] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.416071  1782 raft_consensus.cc:385] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.416325  1782 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
W20260709 13:00:42.416653  1556 leader_election.cc:343] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.416972  1782 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.417937  1784 tablet_bootstrap.cc:654] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.415129  1738 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.420526  1782 ts_tablet_manager.cc:1434] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 13:00:42.421693  1782 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.431118  1783 tablet_bootstrap.cc:654] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.431491  1592 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:42.436309  1807 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.437069  1807 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.440814  1807 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.448678  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.442205  1490 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:00:42.451176  1705 leader_election.cc:343] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.455377  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.464570  1782 tablet_bootstrap.cc:654] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:42.466259  1708 leader_election.cc:343] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.467676  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.468415  1784 tablet_bootstrap.cc:492] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:42.471308  1784 ts_tablet_manager.cc:1403] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.261s	user 0.010s	sys 0.008s
I20260709 13:00:42.468873  1807 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:42.475663  1783 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.476470  1783 ts_tablet_manager.cc:1403] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.150s	user 0.015s	sys 0.014s
I20260709 13:00:42.481262  1784 raft_consensus.cc:359] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.482048  1784 raft_consensus.cc:385] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.482359  1784 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.483208  1784 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.483883  1738 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:42.485286  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.486877  1888 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.487401  1888 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:42.487645  1784 ts_tablet_manager.cc:1434] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.004s
I20260709 13:00:42.487980  1783 raft_consensus.cc:359] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.488687  1783 raft_consensus.cc:385] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.488788  1784 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:42.488993  1783 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.488428  1888 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.489699  1783 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.492714  1783 ts_tablet_manager.cc:1434] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.001s
I20260709 13:00:42.494769  1783 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.504151  1783 tablet_bootstrap.cc:654] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.512188  1784 tablet_bootstrap.cc:654] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.515511  1782 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.515728  1783 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.516074  1783 ts_tablet_manager.cc:1403] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.023s	user 0.019s	sys 0.000s
I20260709 13:00:42.516039  1782 ts_tablet_manager.cc:1403] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.095s	user 0.013s	sys 0.007s
I20260709 13:00:42.519969  1782 raft_consensus.cc:359] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.521037  1782 raft_consensus.cc:385] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.521433  1782 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.522485  1782 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.526890  1783 raft_consensus.cc:359] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.527681  1783 raft_consensus.cc:385] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.528122  1783 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.527403  1489 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:42.529129  1783 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.532649  1783 ts_tablet_manager.cc:1434] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:42.534312  1783 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.535717  1834 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.536206  1782 ts_tablet_manager.cc:1434] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.004s
I20260709 13:00:42.537667  1782 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.542271  1783 tablet_bootstrap.cc:654] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.546783  1782 tablet_bootstrap.cc:654] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.549431  1784 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:42.565428  1784 ts_tablet_manager.cc:1403] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.077s	user 0.018s	sys 0.006s
I20260709 13:00:42.578034  1834 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.578675  1834 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.580709  1834 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.576665  1893 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:00:42.584167  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.584933  1592 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:42.594300  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:42.599061  1705 leader_election.cc:343] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.602401  1590 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.603335  1708 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.596891  1784 raft_consensus.cc:359] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.604470  1784 raft_consensus.cc:385] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.604846  1784 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.605674  1782 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.605741  1784 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.606305  1782 ts_tablet_manager.cc:1403] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.069s	user 0.021s	sys 0.007s
I20260709 13:00:42.608246  1784 ts_tablet_manager.cc:1434] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.003s
I20260709 13:00:42.609272  1784 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:42.605815  1807 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.609220  1782 raft_consensus.cc:359] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.609952  1807 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.610255  1782 raft_consensus.cc:385] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.610354  1807 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.610623  1782 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.611342  1782 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.615948  1784 tablet_bootstrap.cc:654] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.627409  1893 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:42.627120  1807 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.644810  1891 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.646684  1891 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.653395  1782 ts_tablet_manager.cc:1434] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.047s	user 0.010s	sys 0.000s
I20260709 13:00:42.660861  1783 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.661924  1783 ts_tablet_manager.cc:1403] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.128s	user 0.028s	sys 0.022s
I20260709 13:00:42.655200  1807 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.670562  1888 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:00:42.671651  1888 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.677284  1782 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.678169  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.679529  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.676892  1891 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:42.684897  1705 leader_election.cc:343] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.685472  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.686195  1666 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
W20260709 13:00:42.685695  1705 leader_election.cc:343] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.687484  1807 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.687955  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.688633  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:42.690106  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.691845  1788 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:42.695608  1788 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:00:42.695736  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:42.696070  1788 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:42.696558  1592 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.697938  1783 raft_consensus.cc:359] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.698627  1783 raft_consensus.cc:385] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.698884  1783 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.701066  1809 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.701889  1809 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:42.702519  1708 leader_election.cc:343] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:42.703033  1708 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.687974  1807 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.705981  1809 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.718653  1788 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.729903  1919 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:42.735948  1783 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.738554  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.731406  1920 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:00:42.739234  1666 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:42.737529  1807 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.694231  1888 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.740921  1853 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:42.742727  1853 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.746423  1853 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.748587  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:42.730340  1592 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:42.749709  1742 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:42.748715  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:42.752417  1783 ts_tablet_manager.cc:1434] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.090s	user 0.009s	sys 0.001s
I20260709 13:00:42.755509  1784 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:42.756214  1784 ts_tablet_manager.cc:1403] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.147s	user 0.011s	sys 0.007s
I20260709 13:00:42.758859  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:42.759640  1666 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:42.759910  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.752846  1738 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:42.761917  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:42.762831  1783 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.763069  1705 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.760003  1784 raft_consensus.cc:359] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.764145  1784 raft_consensus.cc:385] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.764024  1888 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.764442  1784 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.765245  1784 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.767045  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.764523  1888 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.769120  1782 tablet_bootstrap.cc:654] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.804895  1784 ts_tablet_manager.cc:1434] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.038s
I20260709 13:00:42.773316  1788 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.798730  1807 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:42.811368  1888 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.770844  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:42.811859  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:42.815246  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:42.815855  1784 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:42.821940  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.769322  1708 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.823446  1834 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.823670  1853 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.823974  1834 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:42.825987  1834 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.828778  1783 tablet_bootstrap.cc:654] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:42.830684  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67622.7us
I20260709 13:00:42.813419  1807 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:42.833142  1788 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:00:42.836273  1807 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.841200  1590 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:00:42.840659  1888 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.851253  1591 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:42.853195  1784 tablet_bootstrap.cc:654] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.844775  1891 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 13:00:42.837905  1853 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:42.854897  1891 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:42.859838  1891 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:42.861660  1891 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.863022  1891 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:42.867183  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.867864  1665 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:42.875806  1782 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.876364  1782 ts_tablet_manager.cc:1403] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.213s	user 0.010s	sys 0.023s
I20260709 13:00:42.874742  1888 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:42.881951  1807 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.883602  1665 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:42.884057  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:42.884878  1742 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:42.885133  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:42.887512  1591 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:42.885172  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:42.888283  1783 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:42.888856  1783 ts_tablet_manager.cc:1403] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.126s	user 0.006s	sys 0.010s
I20260709 13:00:42.889003  1888 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:42.888387  1782 raft_consensus.cc:359] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.889669  1782 raft_consensus.cc:385] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.889988  1782 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.890833  1782 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.892203  1888 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:42.893093  1888 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.894029  1782 ts_tablet_manager.cc:1434] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:00:42.894351  1783 raft_consensus.cc:359] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.895179  1782 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.895552  1783 raft_consensus.cc:385] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.895910  1783 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.896603  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:42.896739  1783 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.897362  1665 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:42.898358  1807 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:42.899892  1783 ts_tablet_manager.cc:1434] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:00:42.900219  1489 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:42.902141  1783 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:42.907670  1782 tablet_bootstrap.cc:654] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.906852  1705 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:42.911566  1891 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:42.920087  1784 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:42.920673  1784 ts_tablet_manager.cc:1403] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.107s	user 0.017s	sys 0.013s
I20260709 13:00:42.925177  1784 raft_consensus.cc:359] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.925936  1784 raft_consensus.cc:385] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.926206  1784 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.926813  1784 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.929030  1784 ts_tablet_manager.cc:1434] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 13:00:42.931180  1784 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:42.931711  1489 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:42.949651  1888 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:42.950601  1853 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:42.958153  1783 tablet_bootstrap.cc:654] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.960098  1782 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:42.960704  1782 ts_tablet_manager.cc:1403] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.066s	user 0.020s	sys 0.008s
I20260709 13:00:42.962291  1784 tablet_bootstrap.cc:654] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:42.969106  1853 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:42.970144  1888 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:42.969631  1782 raft_consensus.cc:359] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.976120  1782 raft_consensus.cc:385] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:42.976439  1782 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:42.977105  1782 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:42.979355  1782 ts_tablet_manager.cc:1434] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 13:00:42.980813  1782 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:42.990694  1782 tablet_bootstrap.cc:654] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.001266  1888 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.001926  1888 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.005537  1782 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:43.006089  1782 ts_tablet_manager.cc:1403] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.008s
I20260709 13:00:43.010875  1888 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.011361  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.010864  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.015765  1784 tablet_bootstrap.cc:492] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:43.016265  1784 ts_tablet_manager.cc:1403] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.011s
I20260709 13:00:43.018919  1898 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.020366  1898 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:43.021618  1556 leader_election.cc:343] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.026193  1898 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:43.027560  1557 leader_election.cc:343] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.024119  1783 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:43.025081  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.028184  1782 raft_consensus.cc:359] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.028726  1590 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:43.028899  1782 raft_consensus.cc:385] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.029219  1782 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.029759  1783 ts_tablet_manager.cc:1403] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.129s	user 0.017s	sys 0.006s
I20260709 13:00:43.029984  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.029932  1782 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.032584  1782 ts_tablet_manager.cc:1434] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.000s
I20260709 13:00:43.033710  1782 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:43.035279  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.039239  1634 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:43.040131  1788 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.041543  1782 tablet_bootstrap.cc:654] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.041255  1783 raft_consensus.cc:359] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.041332  1784 raft_consensus.cc:359] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.042073  1783 raft_consensus.cc:385] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.042339  1784 raft_consensus.cc:385] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.042307  1856 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.042384  1783 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.042830  1784 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.043428  1783 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.043669  1784 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.045951  1783 ts_tablet_manager.cc:1434] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:00:43.046633  1784 ts_tablet_manager.cc:1434] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 13:00:43.047119  1783 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:43.048167  1784 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:43.062306  1856 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.063129  1853 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.063669  1853 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.067806  1853 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.070218  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.070940  1590 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.072394  1634 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.073257  1782 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:43.073769  1782 ts_tablet_manager.cc:1403] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.040s	user 0.017s	sys 0.008s
I20260709 13:00:43.074579  1784 tablet_bootstrap.cc:654] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.075184  1853 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.075564  1853 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.075903  1853 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.081554  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.085466  1738 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.085901  1783 tablet_bootstrap.cc:654] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.085716  1782 raft_consensus.cc:359] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.086556  1782 raft_consensus.cc:385] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.086845  1782 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.087476  1782 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.094851  1782 ts_tablet_manager.cc:1434] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.001s
I20260709 13:00:43.096611  1782 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:43.097157  1853 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.100203  1784 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:43.100749  1784 ts_tablet_manager.cc:1403] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.053s	user 0.012s	sys 0.003s
I20260709 13:00:43.101156  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.101867  1590 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.104139  1782 tablet_bootstrap.cc:654] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.104789  1784 raft_consensus.cc:359] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.113402  1784 raft_consensus.cc:385] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.113842  1784 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.115355  1784 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.116801  1784 ts_tablet_manager.cc:1434] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:00:43.118618  1784 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:43.112413  1590 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:43.133277  1784 tablet_bootstrap.cc:654] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.132893  1634 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.134277  1898 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.137460  1783 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:43.125881  1856 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.138893  1783 ts_tablet_manager.cc:1403] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.092s	user 0.014s	sys 0.006s
I20260709 13:00:43.139057  1856 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.148051  1783 raft_consensus.cc:359] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.150228  1783 raft_consensus.cc:385] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.150755  1783 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.152240  1783 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.155654  1783 ts_tablet_manager.cc:1434] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.001s
I20260709 13:00:43.158453  1783 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:43.164752  1942 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.177126  1942 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.180145  1942 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.180353  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.181138  1591 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.181851  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.182549  1738 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.182476  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.183259  1783 tablet_bootstrap.cc:654] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.188920  1942 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.189391  1942 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.191324  1856 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.191653  1942 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.192189  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.192550  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.193490  1807 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.202316  1591 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.202461  1807 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.204480  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.207449  1738 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.210011  1856 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.210501  1856 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.210922  1856 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.212270  1807 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.214123  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.214015  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.221526  1898 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:43.223526  1853 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.223199  1898 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.223809  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:43.220363  1590 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:43.224448  1738 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.225080  1783 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:43.228741  1783 ts_tablet_manager.cc:1403] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.071s	user 0.014s	sys 0.008s
I20260709 13:00:43.230082  1708 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:43.231218  1807 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.231690  1807 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.232080  1807 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.234632  1783 raft_consensus.cc:359] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.238276  1783 raft_consensus.cc:385] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.239099  1783 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.240377  1783 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.247890  1783 ts_tablet_manager.cc:1434] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.004s
I20260709 13:00:43.238977  1856 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.227752  1942 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.242623  1788 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.244740  1784 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:43.251135  1788 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.253369  1942 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.252609  1784 ts_tablet_manager.cc:1403] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.134s	user 0.023s	sys 0.023s
W20260709 13:00:43.248143  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:43.258821  1788 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.258610  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.263499  1590 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.264245  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:43.265722  1668 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:43.265548  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.267078  1741 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:43.267841  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.271894  1782 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:43.271077  1784 raft_consensus.cc:359] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.272680  1782 ts_tablet_manager.cc:1403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.176s	user 0.043s	sys 0.054s
I20260709 13:00:43.272843  1784 raft_consensus.cc:385] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.273334  1784 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.277249  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.277990  1591 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.281955  1782 raft_consensus.cc:359] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.283264  1920 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.268381  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:43.283813  1920 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.284596  1788 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.285033  1788 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.285395  1788 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.286792  1920 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.291042  1856 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.292338  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:43.274243  1784 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.287328  1782 raft_consensus.cc:385] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.259672  1738 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:43.294368  1782 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
W20260709 13:00:43.296501  1556 leader_election.cc:343] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.296716  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.296571  1782 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.297461  1592 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:43.297740  1807 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.300146  1741 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:43.300284  1807 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.266320  1742 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.302597  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:43.302737  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.302685  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.303645  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:43.304530  1665 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:43.305300  1784 ts_tablet_manager.cc:1434] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.003s
I20260709 13:00:43.306494  1783 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:43.307906  1891 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.314277  1891 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.314812  1891 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:43.304953  1705 leader_election.cc:343] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.328727  1590 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:43.329962  1705 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.330902  1807 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.330789  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.333256  1856 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.333598  1788 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.334777  1856 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:43.335808  1856 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.339538  1782 ts_tablet_manager.cc:1434] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.066s	user 0.011s	sys 0.004s
I20260709 13:00:43.341595  1782 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:43.335811  1788 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.346755  1784 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:43.350399  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:43.350800  1742 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:43.352316  1489 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.331346  1591 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:43.356271  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:43.357452  1898 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.357981  1898 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:43.358872  1898 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:43.361479  1556 leader_election.cc:343] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.367633  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:43.368664  1742 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.368687  1783 tablet_bootstrap.cc:654] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.353111  1891 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.372530  1891 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.373551  1834 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.374675  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:43.377452  1784 tablet_bootstrap.cc:654] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.379935  1834 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.381404  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.383930  1591 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:43.388391  1705 leader_election.cc:343] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.388870  1834 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.395748  1788 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:43.399183  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:43.403832  1705 leader_election.cc:343] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.405555  1856 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.410075  1856 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.416819  1782 tablet_bootstrap.cc:654] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.423879  1856 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.424723  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.429889  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:43.430280  1634 leader_election.cc:343] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.430382  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.432734  1834 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:00:43.432718  1632 leader_election.cc:343] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.433907  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:00:43.432719  1708 leader_election.cc:343] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.434697  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.435146  1856 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.421301  1742 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:43.440333  1591 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:43.441900  1807 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:43.436640  1834 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.457190  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.461417  1809 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.466926  1809 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:43.468420  1809 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.469975  1784 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:43.470505  1784 ts_tablet_manager.cc:1403] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.148s	user 0.017s	sys 0.009s
I20260709 13:00:43.474362  1783 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:43.473937  1784 raft_consensus.cc:359] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.474819  1784 raft_consensus.cc:385] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.474864  1783 ts_tablet_manager.cc:1403] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.188s	user 0.006s	sys 0.020s
I20260709 13:00:43.475260  1784 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.476011  1489 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.476076  1784 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.478189  1489 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.478965  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.480225  1834 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.480574  1783 raft_consensus.cc:359] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.482743  1783 raft_consensus.cc:385] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.483110  1783 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.483930  1783 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.487241  1782 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:43.487746  1782 ts_tablet_manager.cc:1403] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.146s	user 0.010s	sys 0.013s
I20260709 13:00:43.480896  1834 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:43.493647  1783 ts_tablet_manager.cc:1434] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 13:00:43.496095  1784 ts_tablet_manager.cc:1434] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 13:00:43.500260  1782 raft_consensus.cc:359] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.501271  1782 raft_consensus.cc:385] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.501642  1782 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.502436  1782 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.505786  1783 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:43.509368  1782 ts_tablet_manager.cc:1434] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.005s
I20260709 13:00:43.509699  1888 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.510210  1888 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.512068  1888 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.515750  1784 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:43.522461  1834 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.531306  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.543347  1782 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:43.542950  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:43.549181  1557 leader_election.cc:343] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.549365  1888 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:43.562465  1784 tablet_bootstrap.cc:654] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:43.564749  1556 leader_election.cc:343] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.575639  1783 tablet_bootstrap.cc:654] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.573477  1488 catalog_manager.cc:5697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:43.580091  1782 tablet_bootstrap.cc:654] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.575464  1556 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:43.589794  1788 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:43.596853  1489 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:43.620510  1856 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 13:00:43.640828  1834 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.641549  1834 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.765512  1856 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.766314  1856 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.790891  1783 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:43.791481  1783 ts_tablet_manager.cc:1403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.290s	user 0.036s	sys 0.026s
I20260709 13:00:43.798338  1782 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:43.798840  1782 ts_tablet_manager.cc:1403] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.256s	user 0.016s	sys 0.023s
I20260709 13:00:43.801442  1784 tablet_bootstrap.cc:492] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:43.802076  1784 ts_tablet_manager.cc:1403] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.295s	user 0.048s	sys 0.048s
I20260709 13:00:43.805298  1784 raft_consensus.cc:359] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.807716  1784 raft_consensus.cc:385] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.808507  1784 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.809602  1784 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.811723  1784 ts_tablet_manager.cc:1434] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:43.813242  1784 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:43.814533  1856 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:43.808332  1782 raft_consensus.cc:359] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.818101  1782 raft_consensus.cc:385] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.818480  1782 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.823807  1782 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.827795  1782 ts_tablet_manager.cc:1434] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.001s
I20260709 13:00:43.828778  1782 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:43.825938  1807 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.833009  1807 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.834051  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.835340  1807 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.840893  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.825178  1834 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:43.844986  1708 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.854645  1783 raft_consensus.cc:359] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.859934  1783 raft_consensus.cc:385] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:43.860229  1783 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:43.860798  1783 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.859429  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.859197  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:43.862735  1668 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:43.863198  1665 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:43.863564  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:43.864678  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:43.864779  1589 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:43.866381  1705 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:43.866514  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.867172  1807 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.867533  1807 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.867623  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:43.867934  1807 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.871050  1963 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:43.872818  1634 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:43.874162  1834 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.874580  1834 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.875674  1898 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:43.874876  1963 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:43.877377  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:43.878849  1807 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.880249  1898 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:43.880628  1898 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:43.880970  1898 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.881198  1807 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.875090  1834 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.889808  1964 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:43.891419  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.895294  1783 ts_tablet_manager.cc:1434] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.103s	user 0.004s	sys 0.004s
I20260709 13:00:43.898350  1783 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:43.899188  1593 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.906773  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:43.907506  1665 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.911420  1963 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:43.911903  1963 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.931180  1784 tablet_bootstrap.cc:654] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:43.947621  1668 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:43.940377  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:43.949872  1591 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:43.955381  1665 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:43.956965  1834 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.966199  1593 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:43.967120  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:43.968317  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:43.968776  1834 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:43.969336  1891 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:43.960381  1898 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.145866  1782 tablet_bootstrap.cc:654] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.166934  1834 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:44.043866  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44333.6us
I20260709 13:00:44.199872  1834 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:43.969964  1891 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:43.941562  1963 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.202028  1891 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.202770  1834 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.062456  1807 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.141s
I20260709 13:00:44.205624  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:44.142746  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.157754  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:44.207957  1668 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:00:44.207584  1634 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.209864  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:44.210963  1963 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:44.211908  1784 tablet_bootstrap.cc:492] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:44.212466  1784 ts_tablet_manager.cc:1403] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.400s	user 0.015s	sys 0.003s
I20260709 13:00:44.215137  1898 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.176033  1970 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260709 13:00:44.219663  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:44.250598  1742 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:44.254635  1783 tablet_bootstrap.cc:700] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent opening tablet: real 0.102s	user 0.004s	sys 0.000s
I20260709 13:00:44.278414  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:44.279448  1942 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:44.280330  1783 tablet_bootstrap.cc:654] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.282581  1971 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:44.286666  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.293114  1784 raft_consensus.cc:359] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.297149  1782 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.297729  1782 ts_tablet_manager.cc:1403] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.469s	user 0.037s	sys 0.033s
I20260709 13:00:44.293996  1784 raft_consensus.cc:385] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.299676  1784 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.300616  1784 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:43.969559  1708 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:44.308784  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 339301us
I20260709 13:00:44.309443  1784 ts_tablet_manager.cc:1434] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.097s	user 0.008s	sys 0.000s
I20260709 13:00:44.310698  1784 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:44.316164  1782 raft_consensus.cc:359] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.319074  1782 raft_consensus.cc:385] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.319476  1782 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.320480  1782 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.322570  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:44.323508  1782 ts_tablet_manager.cc:1434] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.003s
I20260709 13:00:44.323519  1834 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:00:44.324352  1705 leader_election.cc:343] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.326093  1782 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:44.328989  1783 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:44.331768  1783 ts_tablet_manager.cc:1403] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.434s	user 0.016s	sys 0.004s
I20260709 13:00:44.347165  1784 tablet_bootstrap.cc:654] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.349403  1970 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:00:44.350703  1970 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.354521  1970 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.355489  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.357203  1589 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:44.357057  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:44.360188  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:00:44.362394  1782 tablet_bootstrap.cc:654] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.367450  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:44.368353  1970 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:44.367672  1783 raft_consensus.cc:359] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.368790  1783 raft_consensus.cc:385] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.369092  1783 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.368729  1970 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:00:44.369608  1970 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:44.371157  1972 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.371721  1972 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.371274  1783 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.374719  1972 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.376269  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:44.377112  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:00:44.381762  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:44.382725  1972 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:00:44.382834  1556 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:44.383250  1972 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:00:44.383296  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 99468us
I20260709 13:00:44.384398  1972 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:44.390556  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.392063  1784 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
W20260709 13:00:44.392128  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:44.392673  1784 ts_tablet_manager.cc:1403] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.082s	user 0.011s	sys 0.003s
I20260709 13:00:44.395774  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.396565  1589 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:44.399915  1489 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:44.395771  1784 raft_consensus.cc:359] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.405130  1784 raft_consensus.cc:385] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.405428  1784 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.407294  1783 ts_tablet_manager.cc:1434] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.072s	user 0.009s	sys 0.001s
I20260709 13:00:44.408567  1783 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:44.412114  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:44.413031  1807 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:44.413414  1807 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:44.413771  1807 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.414764  1970 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.415977  1834 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:44.418905  1664 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:00:44.420701  1972 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.412514  1784 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.427529  1972 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.427937  1784 ts_tablet_manager.cc:1434] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.035s	user 0.004s	sys 0.004s
I20260709 13:00:44.428962  1784 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:44.429023  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:00:44.431669  1694 tablet.cc:2406] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:00:44.423080  1807 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.423687  1971 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.433817  1971 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.435309  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:44.436170  1589 raft_consensus.cc:2368] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:00:44.436683  1971 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.424188  1782 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.425439  1970 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.437539  1782 ts_tablet_manager.cc:1403] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.112s	user 0.013s	sys 0.002s
I20260709 13:00:44.437731  1634 leader_election.cc:400] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:00:44.439476  1634 leader_election.cc:403] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:00:44.440074  1970 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:44.440752  1807 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.443305  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:44.443987  1782 raft_consensus.cc:359] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.444666  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:44.445763  1742 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:44.446909  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:44.447777  1971 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:44.448161  1971 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:44.448490  1971 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.452627  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:00:44.453979  1970 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:00:44.455348  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.444638  1782 raft_consensus.cc:385] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.457221  1782 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
W20260709 13:00:44.457417  1705 leader_election.cc:343] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.458243  1782 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.460831  1782 ts_tablet_manager.cc:1434] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.002s
I20260709 13:00:44.464260  1783 tablet_bootstrap.cc:654] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.468466  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.470064  1782 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:44.472646  1971 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.467453  1784 tablet_bootstrap.cc:654] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.476002  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.477111  1667 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.468137  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.470029  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:44.484164  1591 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.485888  1971 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.486557  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:44.487190  1738 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.492914  1667 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:44.494844  1556 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:44.497092  1972 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:44.501662  1972 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:44.502965  1972 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.493940  1591 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:44.506266  1667 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.506759  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:44.510002  1807 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:44.510648  1807 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:44.511768  1807 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.505837  1664 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:00:44.515861  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:44.522720  1971 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:44.524669  1971 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:44.526754  1784 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:44.527637  1784 ts_tablet_manager.cc:1403] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.099s	user 0.013s	sys 0.005s
I20260709 13:00:44.530202  1782 tablet_bootstrap.cc:654] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.514858  1738 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:44.537318  1783 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:44.538034  1783 ts_tablet_manager.cc:1403] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.130s	user 0.005s	sys 0.013s
I20260709 13:00:44.534646  1971 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.547024  1783 raft_consensus.cc:359] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.547955  1783 raft_consensus.cc:385] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.548236  1783 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.549039  1783 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.550796  1783 ts_tablet_manager.cc:1434] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.002s
I20260709 13:00:44.552626  1783 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:44.572932  1784 raft_consensus.cc:359] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.574386  1784 raft_consensus.cc:385] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.574867  1784 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.575753  1784 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.578224  1784 ts_tablet_manager.cc:1434] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.050s	user 0.007s	sys 0.002s
I20260709 13:00:44.584403  1784 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:44.593677  1784 tablet_bootstrap.cc:654] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.606370  1782 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.606909  1782 ts_tablet_manager.cc:1403] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.137s	user 0.009s	sys 0.010s
I20260709 13:00:44.609228  1782 raft_consensus.cc:359] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.609849  1782 raft_consensus.cc:385] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.610137  1782 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.611706  1782 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.614078  1783 tablet_bootstrap.cc:654] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.613049  1489 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:44.615530  1489 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:44.621598  1782 ts_tablet_manager.cc:1434] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:44.623524  1782 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:44.629696  1807 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.630263  1807 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.632172  1834 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.632609  1834 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.632910  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.633649  1591 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:44.633785  1807 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.635317  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:44.635280  1834 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.639508  1784 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:44.640293  1784 ts_tablet_manager.cc:1403] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.056s	user 0.008s	sys 0.004s
I20260709 13:00:44.641333  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.642266  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.642551  1666 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:44.644358  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.645233  1668 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
W20260709 13:00:44.646478  1708 leader_election.cc:343] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.655027  1807 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:44.655566  1807 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:44.655915  1807 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.660120  1705 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:44.661923  1784 raft_consensus.cc:359] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.662722  1784 raft_consensus.cc:385] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.663086  1784 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.664995  1834 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:44.665879  1834 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:44.666230  1834 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.667475  1784 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.666661  1488 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:44.674571  1782 tablet_bootstrap.cc:654] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.675848  1784 ts_tablet_manager.cc:1434] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.035s	user 0.009s	sys 0.001s
I20260709 13:00:44.679248  1784 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:44.679052  1834 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.676681  1783 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:44.681984  1783 ts_tablet_manager.cc:1403] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.130s	user 0.030s	sys 0.011s
I20260709 13:00:44.682335  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:44.682647  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.677749  1807 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.683249  1668 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:44.684507  1708 leader_election.cc:343] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.684931  1783 raft_consensus.cc:359] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.685626  1783 raft_consensus.cc:385] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.685896  1783 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.686643  1783 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.688633  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:44.689296  1591 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.691931  1668 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:44.694164  1705 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:44.694463  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:44.695178  1667 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.702951  1964 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.703480  1964 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.703989  1667 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:44.705281  1891 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:44.706432  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.706853  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:44.707284  1589 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:44.708720  1632 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:44.709911  1970 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:44.714749  1782 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.717638  1964 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:44.723467  1782 ts_tablet_manager.cc:1403] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.004s
I20260709 13:00:44.714771  1834 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.726423  1970 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:44.723378  1591 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:44.728662  1783 ts_tablet_manager.cc:1434] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.046s	user 0.001s	sys 0.005s
I20260709 13:00:44.721602  1807 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.729890  1783 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:44.734016  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:44.738857  1807 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:44.739347  1807 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:44.740155  1807 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.720917  1784 tablet_bootstrap.cc:654] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.727692  1997 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:44.736219  1782 raft_consensus.cc:359] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.743592  1782 raft_consensus.cc:385] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.743893  1782 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.744407  1782 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.753473  1782 ts_tablet_manager.cc:1434] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.028s	user 0.006s	sys 0.000s
I20260709 13:00:44.754292  1891 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:44.755686  1891 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.759941  1834 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:44.769438  1782 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:44.782598  1783 tablet_bootstrap.cc:654] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:44.781409  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62202.6us
I20260709 13:00:44.811378  1998 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.811949  1998 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.823064  1782 tablet_bootstrap.cc:654] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.831091  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:44.814054  1998 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:44.836210  1488 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:00:44.850024  1705 leader_election.cc:343] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.855346  1782 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.856025  1782 ts_tablet_manager.cc:1403] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.087s	user 0.010s	sys 0.007s
I20260709 13:00:44.856055  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.860687  1783 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:44.861369  1783 ts_tablet_manager.cc:1403] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.132s	user 0.033s	sys 0.032s
I20260709 13:00:44.860749  1782 raft_consensus.cc:359] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.861914  1782 raft_consensus.cc:385] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.862263  1782 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.863061  1782 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.869108  1783 raft_consensus.cc:359] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.871871  1783 raft_consensus.cc:385] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.872268  1783 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.871454  1782 ts_tablet_manager.cc:1434] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.001s
I20260709 13:00:44.873303  1783 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.876569  1782 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
W20260709 13:00:44.878628  1708 leader_election.cc:343] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.878966  1784 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:44.879487  1784 ts_tablet_manager.cc:1403] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.201s	user 0.013s	sys 0.034s
I20260709 13:00:44.885622  1784 raft_consensus.cc:359] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.886463  1784 raft_consensus.cc:385] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.886754  1784 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.887570  1784 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.890499  1783 ts_tablet_manager.cc:1434] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.000s
I20260709 13:00:44.890707  1489 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:44.891628  1783 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:44.882323  1708 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:44.897094  1998 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:44.898083  1784 ts_tablet_manager.cc:1434] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 13:00:44.899178  1784 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:44.914671  1782 tablet_bootstrap.cc:654] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.918481  1783 tablet_bootstrap.cc:654] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.923686  1784 tablet_bootstrap.cc:654] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.936278  1782 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.936837  1782 ts_tablet_manager.cc:1403] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.061s	user 0.013s	sys 0.003s
I20260709 13:00:44.939889  1782 raft_consensus.cc:359] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.940646  1782 raft_consensus.cc:385] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.940948  1782 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:44.941735  1782 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.944666  1782 ts_tablet_manager.cc:1434] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:44.945845  1782 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:44.946527  1667 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:44.952296  1942 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:44.953272  1942 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.956487  1971 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:44.961396  1782 tablet_bootstrap.cc:654] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:44.963650  1971 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:44.983763  1782 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:44.984346  1782 ts_tablet_manager.cc:1403] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.039s	user 0.007s	sys 0.008s
I20260709 13:00:44.984822  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:44.987434  1782 raft_consensus.cc:359] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:44.988457  1782 raft_consensus.cc:385] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:44.988759  1782 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
W20260709 13:00:44.988785  1634 leader_election.cc:343] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:44.996800  1782 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.001137  1782 ts_tablet_manager.cc:1434] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.000s
I20260709 13:00:45.001993  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.002945  1742 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:45.004323  1782 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:45.005770  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:45.007280  1964 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.008144  1964 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.008795  1964 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.014927  1998 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:45.021205  1782 tablet_bootstrap.cc:654] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.032112  1782 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:45.032696  1782 ts_tablet_manager.cc:1403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.003s
I20260709 13:00:45.041769  1998 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.042915  1782 raft_consensus.cc:359] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.043120  1998 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.043637  1782 raft_consensus.cc:385] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.044027  1782 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.052943  1782 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.041167  1942 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.055851  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.055648  1998 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:45.075886  1708 leader_election.cc:343] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.076488  1972 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.082195  1782 ts_tablet_manager.cc:1434] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.004s
I20260709 13:00:45.079108  1971 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:00:45.079831  1972 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.092159  1972 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.092540  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.093232  1738 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:45.097827  1891 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.097966  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:45.098464  1891 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.105991  1891 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:45.107789  1782 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:45.107108  1972 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.113762  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.115207  1972 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.116796  1972 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:45.118484  1708 leader_election.cc:343] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.122391  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.123972  1964 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.129874  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:45.131181  1634 leader_election.cc:343] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.132876  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.131949  1783 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:45.133723  1783 ts_tablet_manager.cc:1403] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.242s	user 0.077s	sys 0.065s
I20260709 13:00:45.134198  1784 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:45.303288  1784 ts_tablet_manager.cc:1403] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.404s	user 0.038s	sys 0.056s
I20260709 13:00:45.303117  1783 raft_consensus.cc:359] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.304020  1783 raft_consensus.cc:385] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.157141  1782 tablet_bootstrap.cc:654] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.304391  1783 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.312067  1783 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.137136  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.317636  1996 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:45.152863  1738 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:45.178668  1955 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.146005  1964 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.177865  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:00:45.325347  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 140403us
I20260709 13:00:45.324376  1783 ts_tablet_manager.cc:1434] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.190s	user 0.009s	sys 0.008s
I20260709 13:00:45.331918  1784 raft_consensus.cc:359] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.332190  1996 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:45.332691  1784 raft_consensus.cc:385] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.333025  1784 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
W20260709 13:00:45.333999  1705 leader_election.cc:343] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.334415  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:45.335381  1997 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:45.340327  1556 leader_election.cc:343] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.340667  1742 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:45.311234  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 114425us
I20260709 13:00:45.325078  1955 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.347142  1784 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.349536  1705 leader_election.cc:343] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.350016  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:45.350973  1784 ts_tablet_manager.cc:1434] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.047s	user 0.003s	sys 0.003s
I20260709 13:00:45.351764  1783 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:45.351541  1807 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:45.349035  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47802.5us
I20260709 13:00:45.345631  1972 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.379032  1955 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.382341  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.384907  1891 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:45.389588  1742 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:45.393818  1972 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.396338  1784 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:45.436160  1738 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:45.451764  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.457460  2015 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:00:45.457479  1662 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:45.410168  1891 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20260709 13:00:45.432493  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:45.490585  1556 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:45.491917  2015 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.488116  1742 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.499611  1891 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.501174  1891 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.509743  1782 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:45.510291  1782 ts_tablet_manager.cc:1403] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.411s	user 0.020s	sys 0.027s
I20260709 13:00:45.510555  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.514580  1782 raft_consensus.cc:359] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.524184  1782 raft_consensus.cc:385] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.524524  1782 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.518697  1891 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:45.494871  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:45.528925  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:45.533033  1705 leader_election.cc:343] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.534488  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:45.539021  1782 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.543490  1708 leader_election.cc:343] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.544039  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:45.544651  1964 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.545158  1964 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:45.546056  1964 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.549268  1556 leader_election.cc:343] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.550700  1891 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.562001  1742 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:45.563385  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:45.563772  2015 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.564714  1955 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.565245  1955 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:00:45.550093  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61373.1us
I20260709 13:00:45.566972  1955 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.575686  1782 ts_tablet_manager.cc:1434] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.065s	user 0.005s	sys 0.003s
I20260709 13:00:45.585909  1891 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:45.600817  1784 tablet_bootstrap.cc:654] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.599359  1782 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:45.606278  1783 tablet_bootstrap.cc:654] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.639107  1488 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.644188  1490 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.646970  1784 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:45.648020  1784 ts_tablet_manager.cc:1403] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.252s	user 0.021s	sys 0.017s
I20260709 13:00:45.649286  1782 tablet_bootstrap.cc:654] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:45.643848  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:45.663754  1784 raft_consensus.cc:359] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.665642  1784 raft_consensus.cc:385] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.666031  1784 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.666973  1784 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.669207  1784 ts_tablet_manager.cc:1434] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 13:00:45.671890  1784 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:45.675535  1783 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:45.676059  1783 ts_tablet_manager.cc:1403] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.324s	user 0.032s	sys 0.030s
I20260709 13:00:45.682497  1784 tablet_bootstrap.cc:654] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.686980  1783 raft_consensus.cc:359] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.687559  1783 raft_consensus.cc:385] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.687855  1783 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.688593  1783 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.691520  1783 ts_tablet_manager.cc:1434] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.001s
I20260709 13:00:45.692152  1784 tablet_bootstrap.cc:492] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:45.692584  1783 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:45.692679  1784 ts_tablet_manager.cc:1403] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.000s
I20260709 13:00:45.694180  1782 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:45.694798  1782 ts_tablet_manager.cc:1403] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.113s	user 0.013s	sys 0.006s
I20260709 13:00:45.696084  1784 raft_consensus.cc:359] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.696790  1784 raft_consensus.cc:385] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.697055  1784 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.697311  1782 raft_consensus.cc:359] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.698161  1782 raft_consensus.cc:385] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.698437  1782 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.699214  1782 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.701656  1782 ts_tablet_manager.cc:1434] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 13:00:45.703083  1783 tablet_bootstrap.cc:654] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.703233  1782 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:45.697767  1784 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.706642  1784 ts_tablet_manager.cc:1434] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.001s
I20260709 13:00:45.707684  1784 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:45.717633  2019 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.718228  2019 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.722088  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:45.724552  1634 leader_election.cc:343] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.726608  1784 tablet_bootstrap.cc:654] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.726466  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.727231  1742 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:45.728950  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:45.730188  1942 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.730700  1942 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.731352  1942 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.744704  2019 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.768524  1942 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.784073  1971 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.781129  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:45.784919  1971 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.786419  1634 leader_election.cc:343] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.789209  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.790190  1664 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:45.792227  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:45.793543  2015 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.794067  2015 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.794492  2015 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.796196  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:45.796922  1742 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.802258  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.803167  1971 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.809237  2015 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.803263  1738 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:45.847601  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:45.848508  1664 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.863760  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:45.865813  1738 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.869688  2015 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.873728  2019 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.874615  2019 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.880678  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.876739  1664 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:45.882644  1741 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:45.886003  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:45.887166  2015 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.893882  2015 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:45.896438  1942 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.900759  2015 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.903402  1783 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:45.905648  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.906553  1783 ts_tablet_manager.cc:1403] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.214s	user 0.054s	sys 0.143s
I20260709 13:00:45.909802  2019 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.910362  1783 raft_consensus.cc:359] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.911337  1783 raft_consensus.cc:385] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.911700  1783 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.912540  1783 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.913640  1784 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:45.914315  1784 ts_tablet_manager.cc:1403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.207s	user 0.066s	sys 0.096s
I20260709 13:00:45.915850  1783 ts_tablet_manager.cc:1434] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:45.917248  1783 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:45.905746  1738 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:45.920224  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:45.918546  1784 raft_consensus.cc:359] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.893009  1782 tablet_bootstrap.cc:654] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.922122  2019 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.840301  1742 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:45.922528  1784 raft_consensus.cc:385] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:45.925915  1784 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:45.926517  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:45.932556  1783 tablet_bootstrap.cc:654] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.926659  1784 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.936055  2020 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.936517  2020 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.938081  1971 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.938578  1971 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.939544  2020 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.940603  1971 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.941577  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.927757  1942 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:45.944386  1593 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.944537  1942 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:45.943812  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.943090  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.945673  1942 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.947158  1634 leader_election.cc:343] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.942233  1742 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:45.950736  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:45.951889  1964 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:45.952239  1964 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.952553  1964 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.954751  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:45.955129  1784 ts_tablet_manager.cc:1434] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.040s	user 0.005s	sys 0.003s
I20260709 13:00:45.964673  1784 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:45.931716  2019 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:45.960680  1964 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.971745  1971 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:45.972213  1488 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:45.971783  1784 tablet_bootstrap.cc:654] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:45.976125  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:45.976960  1741 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:45.979732  1997 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.981663  1964 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:45.962241  2020 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:45.984517  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:45.985566  1807 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:45.986040  1807 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:45.986526  1634 leader_election.cc:343] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.980494  1997 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:45.988097  1807 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:45.989660  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.990330  1664 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:45.991420  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:45.993176  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.963469  1738 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:45.994717  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:45.997844  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:45.999003  1807 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:00:45.999131  1705 leader_election.cc:343] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:45.999423  1807 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:45.999949  1807 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.000357  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.001384  1971 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:46.002190  1708 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.002534  1784 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.002988  1784 ts_tablet_manager.cc:1403] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.039s	user 0.003s	sys 0.018s
W20260709 13:00:46.003118  1708 leader_election.cc:343] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.003486  1708 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.004175  1891 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.005926  1784 raft_consensus.cc:359] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.006599  1784 raft_consensus.cc:385] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.006875  1784 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.007582  1784 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.009766  1784 ts_tablet_manager.cc:1434] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:00:46.013847  1741 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.012763  1997 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.013489  1807 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.018323  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.023015  1782 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.023504  1782 ts_tablet_manager.cc:1403] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.321s	user 0.008s	sys 0.006s
I20260709 13:00:46.025220  1807 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.026211  1782 raft_consensus.cc:359] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.026906  1782 raft_consensus.cc:385] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.027184  1782 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.027938  1782 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.029935  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:46.030072  1782 ts_tablet_manager.cc:1434] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 13:00:46.030613  1665 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.031657  1782 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:46.034932  2020 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.035024  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:46.036504  1708 leader_election.cc:343] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.037461  1997 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.040345  1782 tablet_bootstrap.cc:654] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.043447  1784 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.044173  1942 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.049892  1942 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.052241  1942 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.058132  1783 tablet_bootstrap.cc:492] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.058800  1783 ts_tablet_manager.cc:1403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.142s	user 0.015s	sys 0.022s
I20260709 13:00:46.052079  1997 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.061641  1784 tablet_bootstrap.cc:654] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.063632  1665 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:00:46.065551  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.065832  2049 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:46.067101  1708 leader_election.cc:343] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.074542  1782 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.080509  1782 ts_tablet_manager.cc:1403] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.049s	user 0.012s	sys 0.005s
I20260709 13:00:46.075183  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.076224  1997 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.076462  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.084178  1593 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.084753  1742 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.085599  1634 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:46.086534  1964 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.069104  1490 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.074332  1783 raft_consensus.cc:359] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.087028  1964 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.087399  1783 raft_consensus.cc:385] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.087747  1783 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.087612  1964 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.088552  1783 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.091130  1783 ts_tablet_manager.cc:1434] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 13:00:46.092164  1783 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.100091  1783 tablet_bootstrap.cc:654] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.103442  2020 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:46.105126  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.104523  2020 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.106359  1997 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:46.112918  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.098131  1782 raft_consensus.cc:359] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.114526  1782 raft_consensus.cc:385] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.114814  1782 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.116911  1784 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.117394  1784 ts_tablet_manager.cc:1403] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.074s	user 0.029s	sys 0.018s
I20260709 13:00:46.117767  1964 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.119925  1964 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.120673  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:46.121248  1593 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.121636  1782 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.123900  1782 ts_tablet_manager.cc:1434] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.035s	user 0.005s	sys 0.000s
I20260709 13:00:46.124099  2015 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.126709  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:46.127351  1738 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.129113  1593 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.130474  1634 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:46.131994  2019 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.132490  2019 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:46.133356  2019 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.110116  1997 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:46.138669  1738 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.138342  1997 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.141047  1784 raft_consensus.cc:359] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.141794  1784 raft_consensus.cc:385] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.142107  1784 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.143186  1784 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.146517  1782 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
W20260709 13:00:46.148861  1705 leader_election.cc:343] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.150300  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.151809  1998 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.155548  1782 tablet_bootstrap.cc:654] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.151015  1784 ts_tablet_manager.cc:1434] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260709 13:00:46.159698  1784 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.167619  1488 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.181401  1783 tablet_bootstrap.cc:492] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.182291  1783 ts_tablet_manager.cc:1403] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.090s	user 0.011s	sys 0.004s
I20260709 13:00:46.188946  1784 tablet_bootstrap.cc:654] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.190764  1488 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 2, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:46.199222  1782 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.199790  1782 ts_tablet_manager.cc:1403] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.022s
I20260709 13:00:46.202786  1782 raft_consensus.cc:359] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.203560  1782 raft_consensus.cc:385] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.203923  1782 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.204306  1784 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.204782  1782 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.202724  1490 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.207000  1782 ts_tablet_manager.cc:1434] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:46.204792  1784 ts_tablet_manager.cc:1403] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.002s
I20260709 13:00:46.210013  1784 raft_consensus.cc:359] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.210621  1784 raft_consensus.cc:385] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.210875  1784 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.201987  1783 raft_consensus.cc:359] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.215309  1783 raft_consensus.cc:385] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.215616  1783 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.216423  1783 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.211476  1784 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.219190  1783 ts_tablet_manager.cc:1434] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.034s	user 0.005s	sys 0.000s
I20260709 13:00:46.221755  1782 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:46.226014  1784 ts_tablet_manager.cc:1434] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.001s
I20260709 13:00:46.227089  1784 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.235437  1782 tablet_bootstrap.cc:654] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.234596  1784 tablet_bootstrap.cc:654] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.240274  1998 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.240806  1998 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.242550  1783 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.242961  1998 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.244480  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:46.246723  1708 leader_election.cc:343] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.247859  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.248811  1664 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:46.250394  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.251464  1997 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.251832  1997 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.252184  1997 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.259577  1782 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.260056  1782 ts_tablet_manager.cc:1403] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.012s
I20260709 13:00:46.261757  1783 tablet_bootstrap.cc:654] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.263386  1782 raft_consensus.cc:359] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.263968  1782 raft_consensus.cc:385] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.264461  1782 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.263962  1997 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.265196  1782 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.265794  1784 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.266355  1784 ts_tablet_manager.cc:1403] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.040s	user 0.016s	sys 0.003s
I20260709 13:00:46.267798  1782 ts_tablet_manager.cc:1434] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 13:00:46.267776  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:46.269342  1708 leader_election.cc:343] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.269124  1784 raft_consensus.cc:359] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.270044  1784 raft_consensus.cc:385] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.270375  1784 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.270602  1782 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:46.271135  1784 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.271467  1997 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.273336  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:46.274061  1664 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.277199  1783 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.277865  1783 ts_tablet_manager.cc:1403] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.036s	user 0.006s	sys 0.010s
I20260709 13:00:46.278551  1782 tablet_bootstrap.cc:654] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.278903  1784 ts_tablet_manager.cc:1434] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:00:46.282536  1784 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.283883  1664 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:46.285173  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.285311  1783 raft_consensus.cc:359] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.286547  1783 raft_consensus.cc:385] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.286981  1783 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.287720  1783 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.289937  1783 ts_tablet_manager.cc:1434] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 13:00:46.286609  1997 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.291028  1997 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:46.291821  1783 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.291914  1997 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.307346  2015 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.310408  1998 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.311003  1998 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.316753  1783 tablet_bootstrap.cc:654] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.313328  1998 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.321794  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.324687  1782 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.325176  1782 ts_tablet_manager.cc:1403] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.055s	user 0.009s	sys 0.014s
W20260709 13:00:46.325227  1705 leader_election.cc:343] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.327031  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:46.329996  1708 leader_election.cc:343] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.330401  1708 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.330874  1784 tablet_bootstrap.cc:654] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.331900  1998 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.336697  1593 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.329056  1782 raft_consensus.cc:359] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.345291  1782 raft_consensus.cc:385] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.345655  1782 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.343863  1488 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:46.345438  2020 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.350328  1784 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.350805  1784 ts_tablet_manager.cc:1403] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.068s	user 0.021s	sys 0.011s
I20260709 13:00:46.354050  1784 raft_consensus.cc:359] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.354758  1784 raft_consensus.cc:385] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.355018  1784 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.355664  1784 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.356987  2015 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.359844  1784 ts_tablet_manager.cc:1434] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 13:00:46.360884  1784 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.362656  1782 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.364982  1782 ts_tablet_manager.cc:1434] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20260709 13:00:46.367076  1784 tablet_bootstrap.cc:654] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.371913  1782 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:46.385980  1782 tablet_bootstrap.cc:654] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.390377  1784 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.390965  1784 ts_tablet_manager.cc:1403] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.000s
I20260709 13:00:46.393712  2020 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.394243  1784 raft_consensus.cc:359] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.397382  1784 raft_consensus.cc:385] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.397771  1784 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.398432  1784 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.400388  1784 ts_tablet_manager.cc:1434] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:00:46.403595  1784 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.404273  2020 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.416204  1783 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.416781  1783 ts_tablet_manager.cc:1403] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.125s	user 0.020s	sys 0.028s
I20260709 13:00:46.417115  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:46.418402  1634 leader_election.cc:343] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.419310  2020 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.419673  1783 raft_consensus.cc:359] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.420351  1783 raft_consensus.cc:385] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.420624  1783 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.421275  1783 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.423341  1783 ts_tablet_manager.cc:1434] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.003s
I20260709 13:00:46.424207  1784 tablet_bootstrap.cc:654] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.424309  1783 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.432257  1783 tablet_bootstrap.cc:654] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.433916  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.434684  1741 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.437991  2048 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.440781  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.442394  2048 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.442764  2048 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.443132  2048 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.443667  1783 tablet_bootstrap.cc:492] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.444149  1783 ts_tablet_manager.cc:1403] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.020s	user 0.004s	sys 0.012s
I20260709 13:00:46.447647  1783 raft_consensus.cc:359] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.448350  1783 raft_consensus.cc:385] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.448618  1783 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.449527  1783 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.452059  1782 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.459271  1782 ts_tablet_manager.cc:1403] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.088s	user 0.014s	sys 0.001s
I20260709 13:00:46.453258  1783 ts_tablet_manager.cc:1434] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:46.463200  1783 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.470818  2048 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.472628  1783 tablet_bootstrap.cc:654] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.459437  1784 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.475031  1784 ts_tablet_manager.cc:1403] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.072s	user 0.009s	sys 0.008s
I20260709 13:00:46.474931  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:46.475477  1782 raft_consensus.cc:359] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.479501  1591 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.479923  1782 raft_consensus.cc:385] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.480074  2015 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.480269  1782 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.480595  2015 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.481065  1782 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.483139  1784 raft_consensus.cc:359] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.483783  1784 raft_consensus.cc:385] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.483981  1782 ts_tablet_manager.cc:1434] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.001s
I20260709 13:00:46.484035  1784 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.487226  1784 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.556155  1782 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:46.524262  1784 ts_tablet_manager.cc:1434] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.001s
I20260709 13:00:46.498589  1971 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.565076  1784 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.556550  2020 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.498881  2048 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.534053  2019 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.521637  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.531795  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.500933  1591 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.502403  2015 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.577913  1665 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:46.571744  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:46.597957  1807 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:46.564554  1971 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.592890  1998 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.575521  2019 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.604291  1783 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:46.597645  1740 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.605039  1783 ts_tablet_manager.cc:1403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.142s	user 0.007s	sys 0.014s
I20260709 13:00:46.573446  2020 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.607048  1997 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.605229  1998 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.627341  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:00:46.610957  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51386us
I20260709 13:00:46.634248  1593 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 2
I20260709 13:00:46.618340  1997 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.638039  2047 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.644063  2047 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:46.644753  1998 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.645493  2020 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.645089  2047 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.646474  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.646548  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.647224  1738 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
W20260709 13:00:46.647599  1708 leader_election.cc:343] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.650204  1998 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.650776  1998 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.651801  1997 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.653920  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.658056  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:46.658345  1593 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:46.659255  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.659372  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.662617  1942 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.664724  1942 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.666013  1942 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.671895  1740 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.675885  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.677040  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:46.677680  1708 leader_election.cc:343] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.678557  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.678895  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.677708  1668 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:46.679615  1666 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
W20260709 13:00:46.678896  1708 leader_election.cc:343] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.682327  2019 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.682461  1997 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.686775  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.657385  1998 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.702533  1784 tablet_bootstrap.cc:654] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.704896  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.720983  1740 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.734820  1783 raft_consensus.cc:359] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.746030  1783 raft_consensus.cc:385] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.747781  1971 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.754652  1942 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.756853  1942 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:46.758090  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74685.1us
I20260709 13:00:46.746506  1783 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
W20260709 13:00:46.764281  1705 leader_election.cc:343] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.762251  1783 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.765100  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.766130  1807 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.763098  2079 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.767481  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.768456  1807 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.768985  1807 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.769380  1807 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.773211  1782 tablet_bootstrap.cc:654] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.773315  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.774195  1592 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.777858  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.780082  1998 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.780483  1998 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.780941  1998 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.782496  1783 ts_tablet_manager.cc:1434] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.177s	user 0.009s	sys 0.001s
I20260709 13:00:46.780452  1807 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.789376  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.790508  1589 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.794760  1740 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:46.793973  1488 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:46.789965  2020 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.768443  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.796612  1783 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:46.796701  2020 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.800855  2020 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.826406  1807 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:46.827971  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59616.5us
I20260709 13:00:46.828253  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.830308  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:46.802727  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:46.832422  1708 leader_election.cc:343] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.831118  1592 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.834506  1784 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:46.836094  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:46.836730  1664 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.838259  2079 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.808789  2020 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.839797  2020 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.840616  1784 ts_tablet_manager.cc:1403] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.276s	user 0.017s	sys 0.005s
I20260709 13:00:46.840976  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:46.842341  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.845645  1942 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:00:46.840200  2020 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.851543  1964 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:46.815357  1998 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.874089  1998 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.875248  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:46.876299  2082 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:46.877782  1708 leader_election.cc:343] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.877190  2082 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.883594  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:46.901199  1664 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:46.904484  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:46.907063  1784 raft_consensus.cc:359] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.907804  1784 raft_consensus.cc:385] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.913654  1784 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.914408  1784 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.920045  1784 ts_tablet_manager.cc:1434] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.077s	user 0.008s	sys 0.002s
I20260709 13:00:46.914992  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:46.921605  1740 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.922222  2082 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:46.924628  1783 tablet_bootstrap.cc:654] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:46.927671  1592 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:46.927686  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:46.906121  1998 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:46.928866  1738 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:46.922117  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:46.932416  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:46.935794  2088 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:00:46.937270  1708 leader_election.cc:343] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.940043  2048 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:46.940531  2048 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.940965  1784 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:46.942510  1807 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:00:46.919893  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69585.9us
I20260709 13:00:46.945539  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:46.937745  1998 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:46.949143  2047 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:46.949569  2047 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:46.949991  2047 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:46.960137  1782 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:46.960734  1782 ts_tablet_manager.cc:1403] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.405s	user 0.009s	sys 0.025s
I20260709 13:00:46.949254  1998 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:46.952168  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:46.967383  1782 raft_consensus.cc:359] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:46.967890  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:46.970129  1782 raft_consensus.cc:385] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:46.970870  1782 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:46.945179  2088 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 13:00:46.971920  1782 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:46.975773  1705 leader_election.cc:343] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:46.976241  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:46.977237  1807 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:46.981573  1782 ts_tablet_manager.cc:1434] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
I20260709 13:00:46.990093  2048 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:46.989998  1740 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:47.006889  1784 tablet_bootstrap.cc:654] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.016812  1782 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:47.021596  1634 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.024962  2088 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:47.025481  2088 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:47.002475  2020 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.026338  2088 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.027014  1666 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:47.034435  1783 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.034948  1783 ts_tablet_manager.cc:1403] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.239s	user 0.018s	sys 0.014s
I20260709 13:00:47.037927  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.039835  2087 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:47.055819  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.061437  1740 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:47.063970  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.065764  1942 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.065598  1783 raft_consensus.cc:359] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.066242  1942 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.066743  1942 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.066350  1783 raft_consensus.cc:385] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.068768  1783 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.069696  1783 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.070254  1490 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:47.076804  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.082510  1783 ts_tablet_manager.cc:1434] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.047s	user 0.006s	sys 0.002s
I20260709 13:00:47.049399  2047 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.093098  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.118449  2020 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.120831  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:47.122311  1740 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:00:47.130687  1634 leader_election.cc:343] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.139055  1488 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:47.153658  2047 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.157493  1782 tablet_bootstrap.cc:654] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.157404  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:47.164922  1783 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
W20260709 13:00:47.137148  1634 leader_election.cc:343] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.175770  1740 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.198582  1784 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.199263  1784 ts_tablet_manager.cc:1403] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.259s	user 0.040s	sys 0.033s
I20260709 13:00:47.179737  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:47.203742  2047 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 13:00:47.203904  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72050.7us
I20260709 13:00:47.207547  1782 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.208236  1784 raft_consensus.cc:359] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.209973  1784 raft_consensus.cc:385] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.210059  1782 ts_tablet_manager.cc:1403] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.194s	user 0.015s	sys 0.014s
I20260709 13:00:47.210258  1784 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.211892  1740 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:47.210976  1784 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.213155  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.213797  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.214067  1592 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.214572  2100 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:47.215180  2100 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:47.216079  2100 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.211942  1942 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.223337  1782 raft_consensus.cc:359] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.237674  1782 raft_consensus.cc:385] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.238119  1782 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.224479  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:47.225526  1784 ts_tablet_manager.cc:1434] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.003s
I20260709 13:00:47.226423  1592 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:47.242245  1740 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.242679  1784 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.228519  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.250995  1784 tablet_bootstrap.cc:654] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.238951  1782 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.240554  1783 tablet_bootstrap.cc:654] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.252868  1782 ts_tablet_manager.cc:1434] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.042s	user 0.003s	sys 0.002s
I20260709 13:00:47.253904  1782 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
W20260709 13:00:47.255398  1634 leader_election.cc:343] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.255437  2088 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:47.259575  2088 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.261161  1782 tablet_bootstrap.cc:654] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.250888  1740 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:47.223412  1942 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.269987  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:47.270932  2100 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:47.271929  2100 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:47.273226  2100 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.278724  1783 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.279343  1783 ts_tablet_manager.cc:1403] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.115s	user 0.007s	sys 0.013s
I20260709 13:00:47.282447  1783 raft_consensus.cc:359] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.283156  1783 raft_consensus.cc:385] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.283442  1783 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.284108  1783 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.286737  1783 ts_tablet_manager.cc:1434] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:47.287878  1783 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.296690  1783 tablet_bootstrap.cc:654] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.328675  1784 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.329341  1784 ts_tablet_manager.cc:1403] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.087s	user 0.029s	sys 0.014s
I20260709 13:00:47.332674  1490 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:47.359078  1783 tablet_bootstrap.cc:492] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.359642  1783 ts_tablet_manager.cc:1403] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.072s	user 0.016s	sys 0.006s
I20260709 13:00:47.363513  1783 raft_consensus.cc:359] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.364178  1783 raft_consensus.cc:385] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.364454  1783 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.365131  1783 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.368774  1784 raft_consensus.cc:359] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.369334  1784 raft_consensus.cc:385] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.369616  1784 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.370520  1784 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.374102  1784 ts_tablet_manager.cc:1434] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.043s	user 0.002s	sys 0.007s
I20260709 13:00:47.375859  1783 ts_tablet_manager.cc:1434] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 13:00:47.377068  1783 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.377847  1784 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.379743  1782 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.380182  1782 ts_tablet_manager.cc:1403] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.127s	user 0.012s	sys 0.027s
I20260709 13:00:47.382256  1782 raft_consensus.cc:359] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.382786  1782 raft_consensus.cc:385] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.383001  1782 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.383599  1782 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.384545  1784 tablet_bootstrap.cc:654] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.387220  1783 tablet_bootstrap.cc:654] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.389765  1782 ts_tablet_manager.cc:1434] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:47.391079  1784 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.391177  1782 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:47.391582  1784 ts_tablet_manager.cc:1403] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.015s	user 0.000s	sys 0.012s
I20260709 13:00:47.399550  1782 tablet_bootstrap.cc:654] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.405233  1784 raft_consensus.cc:359] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.406278  1784 raft_consensus.cc:385] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.406605  1784 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.409029  1784 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.411608  1783 tablet_bootstrap.cc:492] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.411958  1784 ts_tablet_manager.cc:1434] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.020s	user 0.010s	sys 0.000s
I20260709 13:00:47.412125  1783 ts_tablet_manager.cc:1403] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.035s	user 0.019s	sys 0.000s
I20260709 13:00:47.412153  1589 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.413071  1784 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.414963  1783 raft_consensus.cc:359] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.415669  1783 raft_consensus.cc:385] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.415931  1783 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.416694  1783 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.419807  1783 ts_tablet_manager.cc:1434] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:00:47.420769  1783 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.421099  1784 tablet_bootstrap.cc:654] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.427763  1782 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.428326  1783 tablet_bootstrap.cc:654] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.428349  1782 ts_tablet_manager.cc:1403] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.038s	user 0.013s	sys 0.009s
I20260709 13:00:47.430763  2100 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.432658  1782 raft_consensus.cc:359] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.433414  1782 raft_consensus.cc:385] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.433758  1782 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.434551  1782 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.437311  1782 ts_tablet_manager.cc:1434] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:00:47.438453  1782 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:47.441326  1491 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:47.445830  1782 tablet_bootstrap.cc:654] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.450497  1592 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.463224  1784 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.463864  1784 ts_tablet_manager.cc:1403] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.006s
I20260709 13:00:47.467377  1784 raft_consensus.cc:359] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.468407  2088 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:47.473316  1784 raft_consensus.cc:385] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.471942  1998 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:47.474164  1784 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.475239  1784 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.475664  1998 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.477285  1998 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.477914  1998 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.480283  1998 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.472398  1782 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.480996  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.481040  1782 ts_tablet_manager.cc:1403] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.043s	user 0.002s	sys 0.015s
I20260709 13:00:47.483879  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.484647  1668 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:47.484788  1782 raft_consensus.cc:359] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.485464  1782 raft_consensus.cc:385] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.486564  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.509191  1782 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.511080  1998 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.511536  1998 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.511981  1998 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.510493  1782 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.515092  1782 ts_tablet_manager.cc:1434] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 13:00:47.516255  1782 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
W20260709 13:00:47.519404  1708 leader_election.cc:343] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.520386  2088 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.500485  1783 tablet_bootstrap.cc:492] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.521528  1783 ts_tablet_manager.cc:1403] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.101s	user 0.002s	sys 0.033s
I20260709 13:00:47.499894  1997 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.522271  1997 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.520905  2088 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.524345  1783 raft_consensus.cc:359] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.525879  1783 raft_consensus.cc:385] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.526255  1783 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.526387  2088 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.526953  1783 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.527163  1998 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.524344  1997 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.491910  1784 ts_tablet_manager.cc:1434] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 13:00:47.529417  1783 ts_tablet_manager.cc:1434] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:00:47.529771  1784 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.530799  1783 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.532006  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.532860  1998 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:47.536294  1634 leader_election.cc:343] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.539124  1782 tablet_bootstrap.cc:654] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.540534  1783 tablet_bootstrap.cc:654] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.541728  1784 tablet_bootstrap.cc:654] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.543445  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:47.544119  1665 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.547935  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.548691  1666 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
W20260709 13:00:47.556336  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64398.1us
I20260709 13:00:47.557791  1782 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.557673  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.558383  1782 ts_tablet_manager.cc:1403] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.006s
I20260709 13:00:47.558444  1742 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:47.558825  1783 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.560067  1783 ts_tablet_manager.cc:1403] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.006s
I20260709 13:00:47.562181  1782 raft_consensus.cc:359] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.562903  1782 raft_consensus.cc:385] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.563262  1782 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.562964  1783 raft_consensus.cc:359] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.563645  1783 raft_consensus.cc:385] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.563889  1783 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.563925  1782 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.566185  1782 ts_tablet_manager.cc:1434] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:47.566331  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:47.567200  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.567476  1783 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.567958  2082 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.568001  2047 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.568497  2082 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.568615  2047 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.568905  2082 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.568969  2047 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.569705  1783 ts_tablet_manager.cc:1434] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 13:00:47.571123  1783 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.580035  1997 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.581403  1665 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:47.581199  1997 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.582122  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:47.587194  1708 leader_election.cc:343] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.600255  1807 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.600776  1807 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.602351  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.603132  1807 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.604228  2047 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.606433  2047 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:47.606808  1708 leader_election.cc:343] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.608214  1998 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.608814  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.608729  1998 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.611908  1998 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.612175  2082 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.612469  1998 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:47.592382  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.590478  1783 tablet_bootstrap.cc:654] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.612560  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.596787  1997 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.615250  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:47.616068  1738 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:00:47.616118  1705 leader_election.cc:343] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.617868  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:47.620779  1634 leader_election.cc:343] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.626485  2082 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.623852  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.623801  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.628108  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:47.615312  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.629247  1662 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:47.628571  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:47.617803  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.630707  1666 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
W20260709 13:00:47.631551  1705 leader_election.cc:343] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:47.626020  1708 leader_election.cc:343] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.629894  1784 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.632196  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.626339  1998 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:47.632786  1784 ts_tablet_manager.cc:1403] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.103s	user 0.011s	sys 0.008s
I20260709 13:00:47.633705  2082 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:47.634733  1708 leader_election.cc:343] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:47.635505  1708 leader_election.cc:343] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.636173  1708 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.567219  1782 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:47.633401  1998 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.637677  2082 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:47.637897  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:47.639056  2082 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.639456  2082 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.639863  2082 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.644409  2047 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:47.645853  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:47.650530  1708 leader_election.cc:343] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.648844  1784 raft_consensus.cc:359] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.651841  1784 raft_consensus.cc:385] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.652213  1784 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.651187  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.653155  1807 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:47.655479  1784 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.657289  2082 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.659582  1784 ts_tablet_manager.cc:1434] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.005s
I20260709 13:00:47.661330  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.663296  1784 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.661437  2082 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.662639  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:47.664769  1666 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.662644  1783 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
W20260709 13:00:47.664161  1708 leader_election.cc:343] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.671375  1783 ts_tablet_manager.cc:1403] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.101s	user 0.029s	sys 0.008s
I20260709 13:00:47.672904  1782 tablet_bootstrap.cc:654] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.674680  1783 raft_consensus.cc:359] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.675460  1783 raft_consensus.cc:385] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.675776  1783 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.676492  1783 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.678844  1666 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:47.679165  1783 ts_tablet_manager.cc:1434] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:00:47.681061  1783 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.684204  1491 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:47.685400  1784 tablet_bootstrap.cc:654] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.687585  2047 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.690137  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:47.691026  1998 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:47.691515  1998 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:47.692400  1998 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.702853  1592 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.719261  1783 tablet_bootstrap.cc:654] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.719602  1782 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.720167  1782 ts_tablet_manager.cc:1403] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.153s	user 0.015s	sys 0.007s
I20260709 13:00:47.722954  2088 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.724080  1784 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:47.723616  2088 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.723946  1782 raft_consensus.cc:359] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.724557  1784 ts_tablet_manager.cc:1403] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.061s	user 0.009s	sys 0.005s
I20260709 13:00:47.724637  1782 raft_consensus.cc:385] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.724943  1782 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.726080  1782 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.727572  1784 raft_consensus.cc:359] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.728194  1784 raft_consensus.cc:385] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.728453  1784 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.728454  1782 ts_tablet_manager.cc:1434] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.002s
I20260709 13:00:47.730012  1782 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:47.729832  1784 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.730903  1998 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.731374  1998 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.731755  1783 tablet_bootstrap.cc:492] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.732321  1783 ts_tablet_manager.cc:1403] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.052s	user 0.010s	sys 0.009s
I20260709 13:00:47.733249  1942 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.733866  1942 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.735356  1783 raft_consensus.cc:359] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.736235  1998 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.736868  1942 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.739284  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.740002  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:47.741883  1705 leader_election.cc:343] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:47.742348  1708 leader_election.cc:343] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.742764  1708 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.746438  1782 tablet_bootstrap.cc:654] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.747120  2088 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.748523  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.749269  1591 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:47.736270  1783 raft_consensus.cc:385] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.749891  1783 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.750541  1783 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.752389  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.752620  1783 ts_tablet_manager.cc:1434] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I20260709 13:00:47.753070  1738 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:47.753664  1783 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:47.755220  1807 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:47.761430  1783 tablet_bootstrap.cc:654] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.763067  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.763876  2088 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.765296  2088 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.766697  2088 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.767208  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.767975  1741 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:47.764846  2082 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.769923  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:47.770920  1964 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.771747  1964 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.772167  1964 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.778127  2077 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.778713  2077 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.782619  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.784083  2100 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.769827  2082 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.794497  1784 ts_tablet_manager.cc:1434] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.070s	user 0.009s	sys 0.000s
I20260709 13:00:47.795236  2077 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:47.799770  1634 leader_election.cc:343] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.803134  2082 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:47.803450  1784 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:47.806197  1971 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.792520  1491 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:47.806746  1971 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.813589  1971 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.809528  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.819147  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.819648  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:47.819819  1738 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:47.820233  1589 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:47.815936  2077 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.821450  2077 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.822161  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:47.822978  2079 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:47.823324  2079 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.823674  2079 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:47.823758  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.824440  1741 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:47.825718  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:00:47.827934  1556 leader_election.cc:343] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.831282  2077 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.837670  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:47.839736  1997 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.840160  1997 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.840494  1997 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.841454  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.842386  1972 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:47.842783  1972 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:47.843129  1972 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.845641  2100 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.836920  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.846570  2088 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.849429  1668 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:47.856029  1972 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.850544  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:47.871258  1784 tablet_bootstrap.cc:654] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:47.865996  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:47.885157  1741 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:47.886044  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:47.887630  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:47.888691  2077 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.876924  1972 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:47.931123  1782 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:47.931708  1782 ts_tablet_manager.cc:1403] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.202s	user 0.028s	sys 0.031s
I20260709 13:00:47.946164  1783 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:47.946796  1783 ts_tablet_manager.cc:1403] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.193s	user 0.031s	sys 0.045s
W20260709 13:00:47.949155  1705 leader_election.cc:343] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.954360  1971 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:47.955025  1971 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.954456  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:00:47.966869  1556 leader_election.cc:343] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:47.967229  2047 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:47.975485  1782 raft_consensus.cc:359] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:47.976421  1782 raft_consensus.cc:385] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:47.976701  1782 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:47.983543  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:47.984254  1591 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:47.999639  1971 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:48.004904  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:48.010085  2077 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.011271  2088 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.008958  1783 raft_consensus.cc:359] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.013368  1783 raft_consensus.cc:385] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.014258  1783 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.014930  1783 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.015398  1591 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:47.977463  1782 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.023312  1964 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.024338  1782 ts_tablet_manager.cc:1434] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.092s	user 0.006s	sys 0.002s
I20260709 13:00:47.994827  2079 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.034878  2079 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.035976  1782 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.041831  1592 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:00:48.079583  1782 tablet_bootstrap.cc:654] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.082010  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:48.087292  2047 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:48.087862  2047 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:48.089120  2047 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.114987  2136 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.115684  2136 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.121697  1964 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:48.115048  2135 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.185s	user 0.002s	sys 0.000s
I20260709 13:00:48.126046  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.131136  1738 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:48.129290  2136 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.132074  1964 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:48.140808  1964 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:00:48.141229  1964 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:48.142009  1997 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.142165  1964 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.144315  1964 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:48.144996  1997 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:48.146260  1740 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:48.150420  2019 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.150969  2019 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.155879  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:48.156123  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61973.7us
I20260709 13:00:48.162320  1783 ts_tablet_manager.cc:1434] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.215s	user 0.013s	sys 0.008s
I20260709 13:00:48.163744  1783 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:48.168344  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:48.173803  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:48.178251  1742 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.178556  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.181782  1662 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:48.182649  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:48.183334  1737 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.183223  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:48.184494  1664 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260709 13:00:48.201334  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.202327  1663 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:48.204276  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:48.205142  1593 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:00:48.206408  1634 leader_election.cc:343] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.196485  2019 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.208491  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:48.262902  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.273413  1783 tablet_bootstrap.cc:654] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.283170  1784 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:48.283753  1784 ts_tablet_manager.cc:1403] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.488s	user 0.014s	sys 0.018s
I20260709 13:00:48.287214  1737 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:48.294281  1742 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
W20260709 13:00:48.295667  1705 leader_election.cc:343] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.296963  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.281030  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.298554  2136 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.302351  1784 raft_consensus.cc:359] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.303211  1784 raft_consensus.cc:385] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.303519  1784 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.304261  1784 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.304311  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.306205  1592 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:48.307536  1782 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:48.308194  2086 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.312505  1591 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.315866  1782 ts_tablet_manager.cc:1403] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.290s	user 0.019s	sys 0.007s
I20260709 13:00:48.317770  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.318483  1741 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:48.318965  1782 raft_consensus.cc:359] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.318900  1489 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:48.319844  1782 raft_consensus.cc:385] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.320202  1782 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.320873  1782 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.322570  1782 ts_tablet_manager.cc:1434] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 13:00:48.324898  2136 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.330394  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.331820  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.333214  1971 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.333953  1971 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:00:48.334501  1784 ts_tablet_manager.cc:1434] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.002s
I20260709 13:00:48.336139  1784 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:48.340343  1634 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:48.341243  1782 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.343024  2019 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:48.344291  2019 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:48.344659  2019 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.345494  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.351641  2139 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.349105  1782 tablet_bootstrap.cc:654] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:48.354749  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:00:48.358798  2019 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.360488  1784 tablet_bootstrap.cc:654] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.363121  2019 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.367414  1998 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:48.369565  1807 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:48.382745  2082 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:00:48.407338  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 70278.4us
W20260709 13:00:48.409341  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67873.2us
I20260709 13:00:48.408934  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:48.426148  1782 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:48.430356  1782 ts_tablet_manager.cc:1403] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.107s	user 0.007s	sys 0.046s
I20260709 13:00:48.434671  1783 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:48.435493  1783 ts_tablet_manager.cc:1403] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.272s	user 0.012s	sys 0.014s
W20260709 13:00:48.438588  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58156us
W20260709 13:00:48.444310  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:48.443842  1783 raft_consensus.cc:359] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.444864  1783 raft_consensus.cc:385] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.445292  1783 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.422150  2088 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:48.456125  1783 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.458981  2088 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:48.484954  1784 tablet_bootstrap.cc:492] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:48.485571  1784 ts_tablet_manager.cc:1403] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.150s	user 0.027s	sys 0.007s
I20260709 13:00:48.486645  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:48.487275  1593 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.487509  1783 ts_tablet_manager.cc:1434] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.000s
I20260709 13:00:48.488258  2088 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.492199  1782 raft_consensus.cc:359] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.493743  1782 raft_consensus.cc:385] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.494036  1782 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.488755  1783 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:48.495651  1784 raft_consensus.cc:359] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.498093  1784 raft_consensus.cc:385] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.498451  1784 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.499306  1784 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.504632  1593 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:48.506883  1782 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.508279  1784 ts_tablet_manager.cc:1434] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.001s
W20260709 13:00:48.515442  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59683.5us
I20260709 13:00:48.517603  1634 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:48.524878  1784 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:48.527861  2152 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:48.530761  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:48.531759  1737 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.535926  2152 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:48.544824  1782 ts_tablet_manager.cc:1434] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.114s	user 0.010s	sys 0.000s
I20260709 13:00:48.559119  2137 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.559824  2137 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.561961  1782 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.563949  2137 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:48.565789  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.573241  1592 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:48.567746  2151 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.568310  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.575443  2151 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.605428  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:48.610152  2082 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:48.610711  2082 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:48.611156  2082 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.613547  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.617342  1592 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:48.551110  2152 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.625042  1784 tablet_bootstrap.cc:654] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.627043  2082 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.631138  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:48.631742  1705 leader_election.cc:343] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.633731  1593 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.644397  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:48.639400  2047 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:00:48.650254  2154 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:00:48.654846  1593 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:48.655913  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:48.650521  2047 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.662485  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.663854  1740 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:48.664044  2047 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:48.669862  1705 leader_election.cc:343] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.641427  2151 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.674103  1782 tablet_bootstrap.cc:654] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.675356  2148 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:48.678129  2148 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:48.679340  1737 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:48.646869  2082 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:48.684319  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.690644  1783 tablet_bootstrap.cc:654] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.679246  2148 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:48.696254  1634 leader_election.cc:343] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.699630  2087 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.702685  2137 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.703176  2137 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.705258  2137 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:48.706961  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.700443  2087 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.707677  1666 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:48.709647  2087 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:48.709794  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:48.712812  2086 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:48.713217  2086 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:48.714401  2086 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.720551  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.722512  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:48.722620  1708 leader_election.cc:343] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:48.724102  1708 leader_election.cc:343] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.725775  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:48.725216  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.725844  2145 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:00:48.728488  1705 leader_election.cc:343] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.729046  1942 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.729657  1784 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:48.730175  1784 ts_tablet_manager.cc:1403] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.208s	user 0.018s	sys 0.009s
I20260709 13:00:48.729046  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:48.738987  2087 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.741478  1784 raft_consensus.cc:359] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.743203  1784 raft_consensus.cc:385] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.743513  1784 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.744428  1784 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.751715  1782 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:48.752468  1782 ts_tablet_manager.cc:1403] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.191s	user 0.026s	sys 0.011s
I20260709 13:00:48.758044  1784 ts_tablet_manager.cc:1434] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.028s	user 0.010s	sys 0.000s
I20260709 13:00:48.764183  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.766846  1782 raft_consensus.cc:359] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.767609  1782 raft_consensus.cc:385] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.768054  1782 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.768801  1782 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.774140  1782 ts_tablet_manager.cc:1434] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 13:00:48.775698  1784 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:48.786484  1782 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.786082  2088 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.796043  2088 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.798784  1942 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.799410  2088 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.811105  2086 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.812077  2088 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 13:00:48.809641  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.818775  1741 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 1: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:48.819657  2086 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:48.819834  1634 leader_election.cc:343] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.823586  1783 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:48.823464  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:48.824215  1783 ts_tablet_manager.cc:1403] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.336s	user 0.024s	sys 0.006s
I20260709 13:00:48.825805  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:48.826488  1663 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.827018  1783 raft_consensus.cc:359] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.827675  1783 raft_consensus.cc:385] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.827934  1783 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
W20260709 13:00:48.832268  1708 leader_election.cc:343] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.832350  1783 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.837350  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:48.842021  1632 leader_election.cc:343] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.842530  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:48.843793  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:48.846575  1783 ts_tablet_manager.cc:1434] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 13:00:48.850293  1942 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.854094  1783 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:48.861143  1782 tablet_bootstrap.cc:654] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.873412  1663 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:48.874146  2152 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:48.876943  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:48.878388  2086 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:48.879415  2086 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:48.880409  2086 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.885829  1782 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:48.887140  1782 ts_tablet_manager.cc:1403] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.101s	user 0.017s	sys 0.002s
I20260709 13:00:48.878368  1783 tablet_bootstrap.cc:654] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.886371  1784 tablet_bootstrap.cc:654] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.896217  1782 raft_consensus.cc:359] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.899096  1782 raft_consensus.cc:385] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.900379  1782 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.902244  1782 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.909122  1782 ts_tablet_manager.cc:1434] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.002s
I20260709 13:00:48.913862  1782 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.918867  1783 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:48.918071  1489 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:48.919508  1783 ts_tablet_manager.cc:1403] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.066s	user 0.012s	sys 0.006s
I20260709 13:00:48.925302  1782 tablet_bootstrap.cc:654] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.928313  1783 raft_consensus.cc:359] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.929026  1784 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:48.929174  1783 raft_consensus.cc:385] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.929574  1784 ts_tablet_manager.cc:1403] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.154s	user 0.015s	sys 0.010s
I20260709 13:00:48.929579  1783 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.933568  1784 raft_consensus.cc:359] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.933902  1783 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.934508  1784 raft_consensus.cc:385] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.934868  1782 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:48.934816  1784 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.935443  1782 ts_tablet_manager.cc:1403] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.000s
I20260709 13:00:48.935901  1784 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.937085  1783 ts_tablet_manager.cc:1434] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.004s
I20260709 13:00:48.938603  1783 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:48.938453  1782 raft_consensus.cc:359] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.939145  1782 raft_consensus.cc:385] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:48.938601  1784 ts_tablet_manager.cc:1434] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:00:48.939432  1782 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:48.940547  1784 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:48.940228  1782 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.943234  2138 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.943805  2138 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.945246  1782 ts_tablet_manager.cc:1434] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:00:48.947732  1783 tablet_bootstrap.cc:654] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.954695  2138 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:48.956689  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:48.957520  1741 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 0.
I20260709 13:00:48.958783  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:48.959708  2137 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.960181  2137 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.960620  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.946278  1782 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:48.961763  2138 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:48.962211  2138 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:48.962545  2138 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.970045  1782 tablet_bootstrap.cc:654] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
W20260709 13:00:48.972594  1556 leader_election.cc:343] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.974259  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:48.977439  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:48.977174  1488 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:48.978302  1663 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:48.979480  2138 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.979663  1488 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:48.981051  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:48.982040  2082 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:48.982380  1784 tablet_bootstrap.cc:654] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:48.982524  2082 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:48.983039  2082 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.986126  2138 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:48.990857  1708 leader_election.cc:343] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:48.991541  2087 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:48.992678  2087 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:48.996249  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:49.000442  1741 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.008227  1587 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:48.999792  2137 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.019194  2087 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.022964  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.024031  1591 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:49.035524  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:49.036764  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.036973  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.037668  1665 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:49.039654  1784 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.040135  1784 ts_tablet_manager.cc:1403] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.100s	user 0.013s	sys 0.005s
I20260709 13:00:49.042105  2088 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.043781  2149 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:49.044456  2149 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.044827  2149 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:00:49.049311  1556 leader_election.cc:343] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.040467  2082 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.059473  1784 raft_consensus.cc:359] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.060314  1784 raft_consensus.cc:385] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.060600  1784 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.061287  1784 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.066250  1784 ts_tablet_manager.cc:1434] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.007s
I20260709 13:00:49.071615  1782 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:49.072160  1782 ts_tablet_manager.cc:1403] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.126s	user 0.028s	sys 0.031s
I20260709 13:00:49.075675  1784 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:49.076602  1489 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.080590  1782 raft_consensus.cc:359] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.089392  1782 raft_consensus.cc:385] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.089794  1782 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.089982  1741 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:49.090523  1782 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.094218  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.101461  1782 ts_tablet_manager.cc:1434] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 13:00:49.102813  1782 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:49.103677  1783 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:49.103833  2082 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.104475  1972 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.104954  1972 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.097888  2149 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.110658  2145 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:00:49.112078  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.112901  1666 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.104195  1783 ts_tablet_manager.cc:1403] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.166s	user 0.013s	sys 0.010s
W20260709 13:00:49.106405  1708 leader_election.cc:343] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.109362  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:49.117936  2139 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:49.119439  2139 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:49.120469  2139 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.136464  2152 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.140542  1972 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.143618  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:49.146845  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44547.6us
W20260709 13:00:49.147486  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:49.147498  1783 raft_consensus.cc:359] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.148481  1783 raft_consensus.cc:385] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.148782  1783 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.144049  1666 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:49.149833  2149 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.152061  1783 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.155723  1782 tablet_bootstrap.cc:654] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.157775  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.183050  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.183408  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:49.185012  2087 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:49.185665  2087 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:49.186578  2087 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.189812  1591 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.192615  1784 tablet_bootstrap.cc:654] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.201885  1783 ts_tablet_manager.cc:1434] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.086s	user 0.008s	sys 0.000s
I20260709 13:00:49.202436  1942 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.203081  1783 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:49.202945  1942 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.205025  1942 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.207454  2151 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:49.208343  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:49.222739  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:49.222800  2148 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.224520  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.225211  1663 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.234131  2136 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:00:49.248021  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.267699  1634 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:49.268586  2145 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:49.233487  1591 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:00:49.270320  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:49.286092  1783 tablet_bootstrap.cc:654] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.258153  2137 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.290917  2137 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.292837  2137 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.293067  1784 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.293767  1784 ts_tablet_manager.cc:1403] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.218s	user 0.030s	sys 0.013s
I20260709 13:00:49.293071  1489 catalog_manager.cc:5697] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:49.297699  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.298426  1662 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:49.299258  2145 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.299777  2145 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.275692  2087 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:49.298951  1488 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.301025  2087 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:49.301954  2087 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.256582  1663 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:00:49.303300  2145 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.305940  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.306635  1592 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:00:49.306690  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:49.312664  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:49.313844  2148 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:49.314208  2148 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.314556  2148 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.316701  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:49.340101  1784 raft_consensus.cc:359] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.340886  1784 raft_consensus.cc:385] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.341181  1784 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.341918  1784 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.344206  2145 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:00:49.350381  2145 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.350909  2145 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:49.320855  2137 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.359820  1784 ts_tablet_manager.cc:1434] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.001s
I20260709 13:00:49.363860  1663 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.366586  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.357816  2137 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.370558  1784 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
W20260709 13:00:49.377748  1708 leader_election.cc:343] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.377967  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:49.383244  1708 leader_election.cc:343] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.393218  1782 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:49.393942  1782 ts_tablet_manager.cc:1403] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.291s	user 0.015s	sys 0.023s
W20260709 13:00:49.395260  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51751.4us
I20260709 13:00:49.361884  2082 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.397691  2137 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.397316  2082 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.399506  2145 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.357419  2148 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.401257  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.402434  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.402984  1587 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:00:49.403087  2148 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.406473  2145 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.409888  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:49.410714  1488 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 2, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.412726  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.413319  1592 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
W20260709 13:00:49.413363  1705 leader_election.cc:343] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.413862  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:49.414088  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.414486  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:49.401301  2082 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.415724  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.416412  1662 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
W20260709 13:00:49.417428  1708 leader_election.cc:343] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.417994  2148 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:49.418444  2148 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.418824  2148 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.418876  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.423038  2082 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:49.423316  1590 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.428326  1782 raft_consensus.cc:359] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.429065  1782 raft_consensus.cc:385] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.429276  1782 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.430871  1782 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.432390  2148 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.434186  1784 tablet_bootstrap.cc:654] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.436973  2148 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.437672  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.438325  1592 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.441663  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.442308  1665 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:49.445518  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:00:49.419521  1662 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.447358  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:00:49.449120  2145 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:00:49.449712  2145 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:49.450668  2145 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.454778  1592 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:00:49.458415  1665 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:00:49.461130  2148 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.489820  1782 ts_tablet_manager.cc:1434] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.095s	user 0.008s	sys 0.000s
I20260709 13:00:49.491426  1782 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:49.504522  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:00:49.511689  2087 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:49.520859  2087 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:49.521903  2087 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.557914  2082 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.558890  1662 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:49.561672  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:49.564371  1783 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:49.564873  1783 ts_tablet_manager.cc:1403] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.362s	user 0.040s	sys 0.026s
I20260709 13:00:49.565444  2082 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:49.567974  2082 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:49.568838  2082 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.579893  1784 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.588165  1782 tablet_bootstrap.cc:654] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.584825  1783 raft_consensus.cc:359] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.593721  1783 raft_consensus.cc:385] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.594049  1783 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.593994  1488 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:49.594766  1783 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.598865  1784 ts_tablet_manager.cc:1403] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.229s	user 0.020s	sys 0.021s
I20260709 13:00:49.602723  1783 ts_tablet_manager.cc:1434] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:00:49.603864  1783 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
W20260709 13:00:49.611563  1708 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:49.617985  1784 raft_consensus.cc:359] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:49.619401  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:49.619694  1784 raft_consensus.cc:385] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.620000  1784 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.621014  1784 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.626564  1784 ts_tablet_manager.cc:1434] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 13:00:49.627687  1784 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:49.628769  1782 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:49.629272  1782 ts_tablet_manager.cc:1403] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.138s	user 0.016s	sys 0.000s
I20260709 13:00:49.632200  1782 raft_consensus.cc:359] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.633088  1782 raft_consensus.cc:385] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.636142  1784 tablet_bootstrap.cc:654] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.638823  1783 tablet_bootstrap.cc:654] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.644016  1972 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.633445  1782 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.653543  1782 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.662058  1782 ts_tablet_manager.cc:1434] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 13:00:49.663184  1782 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Bootstrap starting.
I20260709 13:00:49.669327  2149 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:00:49.669960  2149 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.672075  2149 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.679373  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.680476  1587 raft_consensus.cc:2368] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:00:49.683511  1708 leader_election.cc:400] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:00:49.683820  1708 leader_election.cc:403] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:00:49.687916  2137 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:00:49.688241  1782 tablet_bootstrap.cc:654] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.690546  2149 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.685724  1784 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.693034  2149 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.693717  1784 ts_tablet_manager.cc:1403] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.066s	user 0.016s	sys 0.007s
I20260709 13:00:49.701407  2149 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.701252  1784 raft_consensus.cc:359] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.702147  1784 raft_consensus.cc:385] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.702442  1784 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.703106  1784 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.705808  1784 ts_tablet_manager.cc:1434] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 13:00:49.706091  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.706777  1784 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
W20260709 13:00:49.709074  1708 leader_election.cc:343] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:49.710780  1783 tablet_bootstrap.cc:492] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:49.711400  1783 ts_tablet_manager.cc:1403] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.108s	user 0.016s	sys 0.004s
I20260709 13:00:49.716662  1783 raft_consensus.cc:359] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.719079  1784 tablet_bootstrap.cc:654] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.719465  1783 raft_consensus.cc:385] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.719836  1783 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.720799  1783 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.706125  2137 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:00:49.721781  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.723829  1783 ts_tablet_manager.cc:1434] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.004s
I20260709 13:00:49.724933  1783 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:49.731357  1783 tablet_bootstrap.cc:654] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.748704  2020 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.749250  2020 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.751350  2020 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.753568  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.754321  1587 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:49.755483  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:49.757620  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:49.758306  1739 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:49.759048  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.759922  1662 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:49.760044  2190 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:49.760766  2190 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.761132  2190 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.762063  1662 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:49.773608  1784 tablet_bootstrap.cc:492] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.774231  1784 ts_tablet_manager.cc:1403] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.068s	user 0.018s	sys 0.020s
I20260709 13:00:49.781605  1784 raft_consensus.cc:359] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.782439  1784 raft_consensus.cc:385] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.782779  1784 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.783502  1784 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.785452  1662 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.785822  1784 ts_tablet_manager.cc:1434] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.004s
I20260709 13:00:49.787106  1784 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:49.793821  1784 tablet_bootstrap.cc:654] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.797119  2190 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.801541  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.802394  1590 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.806416  1784 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:49.806981  1784 ts_tablet_manager.cc:1403] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.011s
I20260709 13:00:49.810753  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:49.811378  1741 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.812212  2086 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.816009  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:49.820075  2086 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:49.829927  2086 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:49.827391  1784 raft_consensus.cc:359] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.830333  2086 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.830461  1784 raft_consensus.cc:385] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.831223  1784 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.832001  1784 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.834600  1784 ts_tablet_manager.cc:1434] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.000s
I20260709 13:00:49.823294  2149 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:00:49.838986  1784 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:49.810084  2190 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.847791  1782 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: No bootstrap required, opened a new log
I20260709 13:00:49.847107  1489 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 2, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.848268  1782 ts_tablet_manager.cc:1403] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent bootstrapping tablet: real 0.185s	user 0.014s	sys 0.001s
I20260709 13:00:49.849367  1489 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:49.850737  1783 tablet_bootstrap.cc:492] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:49.851209  1783 ts_tablet_manager.cc:1403] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.126s	user 0.041s	sys 0.048s
I20260709 13:00:49.854339  1741 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:49.856774  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:49.862622  1972 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:49.864775  2151 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:49.866443  2151 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:49.867645  2151 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.886286  1782 raft_consensus.cc:359] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.887220  1782 raft_consensus.cc:385] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.887709  1782 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.889019  1782 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.892513  1782 ts_tablet_manager.cc:1434] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent starting tablet: real 0.044s	user 0.006s	sys 0.000s
I20260709 13:00:49.893947  1590 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:49.909162  1783 raft_consensus.cc:359] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.910061  1783 raft_consensus.cc:385] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:49.910411  1783 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:49.916150  2086 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.919090  2086 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:00:49.919701  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56454us
I20260709 13:00:49.923386  2191 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.923913  2191 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.927587  2191 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:49.937546  1783 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.940531  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:49.941412  1668 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:49.920317  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:49.945799  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:49.957183  1667 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 0.
I20260709 13:00:49.964560  1783 ts_tablet_manager.cc:1434] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.113s	user 0.010s	sys 0.000s
I20260709 13:00:49.969812  1783 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:49.971210  1668 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:49.964819  1590 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:49.965423  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:49.980062  1784 tablet_bootstrap.cc:654] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:49.986037  1705 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:49.990132  1739 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:49.991964  2138 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:49.992483  2138 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:49.994635  2138 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:49.996294  2145 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:49.999796  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:00:49.999981  1708 leader_election.cc:343] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:00:50.001282  1708 leader_election.cc:343] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.001576  2148 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:50.002084  2148 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.002445  2148 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.003883  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.006865  2149 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:50.003238  2082 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 13:00:50.020102  1587 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:00:50.026866  1488 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.028949  1783 tablet_bootstrap.cc:654] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.037978  2152 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:50.050019  2152 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.050566  2152 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.052719  2152 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.053853  2149 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:50.055173  2149 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.056561  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:50.057281  1586 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:50.059558  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.060385  1737 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:50.062439  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.068351  2152 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:50.068907  2152 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.069603  2152 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:50.073316  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59531.2us
I20260709 13:00:50.076143  1784 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:50.076817  1784 ts_tablet_manager.cc:1403] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.238s	user 0.023s	sys 0.010s
I20260709 13:00:50.048789  2148 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.081063  2148 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:50.082080  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:50.082975  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:00:50.084138  1708 leader_election.cc:343] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.087067  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:50.087666  1668 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.088070  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.088876  2136 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:00:50.099650  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:50.100414  2136 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.100980  2136 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.103035  2136 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.105211  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.105952  1668 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:50.106031  1664 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:50.106839  2219 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.107365  2219 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.108484  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.109373  2136 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:50.110199  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.110884  2219 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:50.110778  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:50.116001  1742 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:00:50.115792  2136 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.116539  1590 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:50.116714  2136 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.110927  1705 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:50.118412  2219 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:50.119771  2219 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:00:50.121263  2219 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.119014  2148 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:50.124346  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.124991  1668 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:00:50.126219  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.132143  2148 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:00:50.138774  2148 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.139246  2148 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.138322  1783 tablet_bootstrap.cc:492] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.145185  1783 ts_tablet_manager.cc:1403] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.176s	user 0.029s	sys 0.002s
I20260709 13:00:50.121440  1784 raft_consensus.cc:359] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.145804  1784 raft_consensus.cc:385] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.146111  1784 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.150116  1784 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.152696  1668 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:50.155241  2136 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.159363  1593 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:50.159039  2137 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:50.166320  1593 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.166419  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:50.167069  1664 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:00:50.171010  1783 raft_consensus.cc:359] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.171763  1783 raft_consensus.cc:385] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.172025  1783 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.187614  2136 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.191367  1784 ts_tablet_manager.cc:1434] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.114s	user 0.003s	sys 0.007s
I20260709 13:00:50.195011  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:50.172674  1783 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.197799  1664 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:00:50.192462  1784 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Bootstrap starting.
I20260709 13:00:50.199397  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.202137  1488 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:50.208835  1972 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:00:50.209389  1972 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:00:50.211889  1783 ts_tablet_manager.cc:1434] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.066s	user 0.006s	sys 0.001s
I20260709 13:00:50.214287  1783 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:50.210281  1972 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.211380  2148 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.208076  2152 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.235118  2152 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.239286  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:50.239984  1737 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.207929  2219 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 13:00:50.252668  1784 tablet_bootstrap.cc:654] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.277956  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:50.279072  1593 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.296175  2136 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.297225  2136 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.309965  1739 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:00:50.270990  1737 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:50.347031  2136 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.347430  2087 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.353420  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.354372  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.379732  1593 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:50.396947  1783 tablet_bootstrap.cc:654] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.397235  2148 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:50.374286  2087 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.406865  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:00:50.406888  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.407819  1668 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:00:50.408248  1972 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:50.424646  2087 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:50.430097  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.430864  1784 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: No bootstrap required, opened a new log
I20260709 13:00:50.431368  1784 ts_tablet_manager.cc:1403] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent bootstrapping tablet: real 0.239s	user 0.015s	sys 0.031s
I20260709 13:00:50.433993  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:00:50.434638  1705 leader_election.cc:343] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.439927  1783 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.440439  1783 ts_tablet_manager.cc:1403] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.226s	user 0.006s	sys 0.021s
I20260709 13:00:50.443233  1783 raft_consensus.cc:359] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.443949  1783 raft_consensus.cc:385] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.444236  1783 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.444408  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.445941  2210 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:50.446442  2210 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:50.447285  2210 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.448056  1784 raft_consensus.cc:359] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.448745  1784 raft_consensus.cc:385] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.449003  1784 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.449631  1784 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.451869  1784 ts_tablet_manager.cc:1434] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent starting tablet: real 0.020s	user 0.004s	sys 0.000s
I20260709 13:00:50.444963  1783 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.454525  1783 ts_tablet_manager.cc:1434] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.002s
I20260709 13:00:50.455724  1783 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
W20260709 13:00:50.452953  1708 leader_election.cc:343] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.457809  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.458630  2221 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:50.458348  1491 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 0 to 1, leader changed from <none> to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:00:50.462800  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:50.469067  1972 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.469708  1972 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.472247  1972 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.476644  1783 tablet_bootstrap.cc:654] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.476894  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.492595  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.493857  2219 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:00:50.501703  1593 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.507547  1489 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 0 to 2, leader changed from <none> to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:00:50.519578  1539 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:00:50.522993  2210 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.540343  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.541193  2190 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:50.544406  2190 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:50.523629  2210 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.548101  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:50.548791  1592 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:50.549264  1634 leader_election.cc:343] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.554546  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.555233  1739 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:50.556017  2210 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.557119  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:50.558149  2151 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:50.558632  2151 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.558979  2151 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:00:50.565659  1708 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:50.592428  1556 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.597425  2219 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:50.586162  2151 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.609977  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:00:50.612071  1634 leader_election.cc:343] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:00:50.621740  1491 catalog_manager.cc:5697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.626540  1783 tablet_bootstrap.cc:492] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.627097  1783 ts_tablet_manager.cc:1403] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.172s	user 0.037s	sys 0.053s
I20260709 13:00:50.630108  1783 raft_consensus.cc:359] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.630847  2151 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.630223  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:50.631623  1740 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.630875  1783 raft_consensus.cc:385] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.634366  1783 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.636102  1783 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:00:50.642848  1708 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:50.642923  1740 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:50.645011  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:00:50.646605  2210 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:50.647131  2210 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:50.647926  2210 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.650156  1783 ts_tablet_manager.cc:1434] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260709 13:00:50.651309  1783 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:50.663457  1783 tablet_bootstrap.cc:654] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.666218  2087 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:50.669684  1972 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:50.678153  1783 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.678766  1783 ts_tablet_manager.cc:1403] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.028s	user 0.011s	sys 0.008s
I20260709 13:00:50.681876  1783 raft_consensus.cc:359] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.682613  1783 raft_consensus.cc:385] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.682881  1783 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.683698  1664 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:50.683758  1783 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.685662  1783 ts_tablet_manager.cc:1434] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:50.687961  1972 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:50.689813  1783 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:50.706369  1783 tablet_bootstrap.cc:654] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.726109  1783 tablet_bootstrap.cc:492] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.726642  1783 ts_tablet_manager.cc:1403] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.000s
I20260709 13:00:50.728868  1740 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 13:00:50.733378  1972 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:50.737929  1783 raft_consensus.cc:359] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.738777  1783 raft_consensus.cc:385] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.739064  1783 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.739885  1783 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.743692  1783 ts_tablet_manager.cc:1434] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:00:50.744629  1783 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Bootstrap starting.
I20260709 13:00:50.744709  1488 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:50.752426  1783 tablet_bootstrap.cc:654] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Neither blocks nor log segments found. Creating new log.
I20260709 13:00:50.766820  1783 tablet_bootstrap.cc:492] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: No bootstrap required, opened a new log
I20260709 13:00:50.767278  1783 ts_tablet_manager.cc:1403] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.008s
I20260709 13:00:50.769765  1783 raft_consensus.cc:359] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.770468  1783 raft_consensus.cc:385] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:00:50.770742  1783 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Initialized, Role: FOLLOWER
I20260709 13:00:50.771517  1783 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.774248  1783 ts_tablet_manager.cc:1434] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:00:50.785446  2196 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.786042  2196 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.788491  2196 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.790742  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.793463  2151 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:50.794066  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:50.794008  2151 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.797731  1556 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:50.798501  2197 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:50.803633  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:00:50.804865  1592 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:50.805492  2151 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.806238  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.806970  2151 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:00:50.807337  2151 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:00:50.807705  2151 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.806263  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:50.812603  1740 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 0.
I20260709 13:00:50.816982  2151 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.822276  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:00:50.823222  1592 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.823553  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:00:50.824162  1740 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:50.825721  2151 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:50.833246  1740 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:50.834616  1592 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:00:50.834484  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:00:50.835741  2151 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:00:50.836275  2151 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:00:50.837083  2151 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:50.848074  1489 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:00:51.011582  2197 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:00:51.012267  2197 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:00:51.015203  2151 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:51.015755  2197 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:51.015857  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:00:51.016625  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:00:51.016888  2211 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:51.017477  1740 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 1: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:00:51.018687  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:00:51.020382  2197 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:51.029893  2197 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:00:51.109402  1593 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:51.155716  2211 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:51.171459  2190 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:00:51.178470  1592 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:00:51.411056  2151 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 13:00:51.422938  2211 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:00:52.721758   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:00:52.744853  2271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:52.746733  2274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:00:52.747339  2272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:00:52.749594   431 server_base.cc:1061] running on GCE node
I20260709 13:00:52.751358   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:00:52.751587   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:00:52.751739   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602052751727 us; error 0 us; skew 500 ppm
I20260709 13:00:52.752250   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:00:52.755671   431 webserver.cc:533] Webserver started at http://127.0.107.196:43131/ using document root <none> and password file <none>
I20260709 13:00:52.756228   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:00:52.756421   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:00:52.756693   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:00:52.758396   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-3-root/instance:
uuid: "e6adef9da7da44f6af0af5695cf9b704"
format_stamp: "Formatted at 2026-07-09 13:00:52 on dist-test-slave-p8rg"
I20260709 13:00:52.763902   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.003s
I20260709 13:00:52.770118  2279 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:52.771592   431 fs_manager.cc:730] Time spent opening block manager: real 0.005s	user 0.003s	sys 0.000s
I20260709 13:00:52.772050   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-3-root
uuid: "e6adef9da7da44f6af0af5695cf9b704"
format_stamp: "Formatted at 2026-07-09 13:00:52 on dist-test-slave-p8rg"
I20260709 13:00:52.772505   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:00:52.805380   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:00:52.807986   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:00:52.810791   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:00:52.816581   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:00:52.816923   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:00:52.817257   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:00:52.817559   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 13:00:52.932107   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.196:35615
I20260709 13:00:52.932199  2342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.196:35615 every 8 connection(s)
I20260709 13:00:52.941885   431 auto_leader_rebalancer-test.cc:384] add a tserver: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:52.970151  2343 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39071
I20260709 13:00:52.970657  2343 heartbeater.cc:461] Registering TS with master...
I20260709 13:00:52.971936  2343 heartbeater.cc:507] Master 127.0.107.254:39071 requested a full tablet report, sending...
I20260709 13:00:52.975982  1490 ts_manager.cc:194] Registered new tserver with Master: e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:00:52.978108  1490 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49282
I20260709 13:00:53.763605  1667 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:53.779825  1587 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:53.780386  1742 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:53.782264  2151 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:53.783164  2266 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:53.803644  2269 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.811261  1740 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.813446  1587 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:00:53.827481  1632 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a9c383ed9b7c4624b8fa546326e129ff. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:53.839816  1489 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.855497  1663 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:53.877621  1742 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:53.880367  1591 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:00:53.880539  1632 consensus_peers.cc:597] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9e456da7a864a1b97fb4c468e75980d. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:53.882215  2355 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:53.883210  2151 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:53.896580  2355 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.898200  1591 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.904451  1740 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.939404  1491 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:53.967391  1740 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:53.983325  2343 heartbeater.cc:499] Master 127.0.107.254:39071 was elected leader, sending a full tablet report...
I20260709 13:00:54.001725  1663 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:54.002460  1591 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:54.003399  2363 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:54.004825  2364 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:00:54.023166  1706 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f90c9079e24cf2a4c9ab3dd55172ea. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:54.022243  1591 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.022737  1663 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.022372  2364 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.043154  1664 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.049737  1488 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.057276  1742 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:00:54.059141  1632 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 337669b46a6b4f75a35716542d1e20ae. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:54.059628  1587 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:54.060304  2266 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:54.065742  2151 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:54.092162  2266 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.102088  1740 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.112614  1591 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.124231  1488 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:54.338474  2373 ts_tablet_manager.cc:933] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:54.340806  2373 tablet_copy_client.cc:323] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:00:54.391901  2377 ts_tablet_manager.cc:933] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:54.413666  1678 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet c9e456da7a864a1b97fb4c468e75980d from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:00:54.414794  1678 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet c9e456da7a864a1b97fb4c468e75980d from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-c9e456da7a864a1b97fb4c468e75980d
I20260709 13:00:54.418677  2377 tablet_copy_client.cc:323] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:00:54.421415  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet a9c383ed9b7c4624b8fa546326e129ff from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:00:54.422082  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet a9c383ed9b7c4624b8fa546326e129ff from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-a9c383ed9b7c4624b8fa546326e129ff
I20260709 13:00:54.439129  1677 tablet_copy_source_session.cc:215] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:54.441407  1678 tablet_copy_source_session.cc:215] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:54.444458  2377 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9c383ed9b7c4624b8fa546326e129ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:54.446416  2373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e456da7a864a1b97fb4c468e75980d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:54.466651  2379 ts_tablet_manager.cc:933] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:54.475930  2379 tablet_copy_client.cc:323] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:00:54.478114  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 337669b46a6b4f75a35716542d1e20ae from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:00:54.478678  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 337669b46a6b4f75a35716542d1e20ae from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-337669b46a6b4f75a35716542d1e20ae
I20260709 13:00:54.482388  2377 tablet_copy_client.cc:806] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:54.477370  2373 tablet_copy_client.cc:806] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:54.484958  2373 tablet_copy_client.cc:670] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:54.485357  2377 tablet_copy_client.cc:670] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:54.493067  2377 tablet_copy_client.cc:538] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:54.490381  1677 tablet_copy_source_session.cc:215] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:54.493527  2373 tablet_copy_client.cc:538] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:54.509897  2377 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:54.511616  2379 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 337669b46a6b4f75a35716542d1e20ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:54.537688  2373 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:54.542908  2381 ts_tablet_manager.cc:933] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:54.560185  2379 tablet_copy_client.cc:806] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:54.562636  2379 tablet_copy_client.cc:670] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:54.568408  2379 tablet_copy_client.cc:538] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:54.590368  2379 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:54.593998  2381 tablet_copy_client.cc:323] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:00:54.645813  2377 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:54.647166  2377 tablet_bootstrap.cc:492] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:54.648134  2377 ts_tablet_manager.cc:1403] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.139s	user 0.020s	sys 0.012s
I20260709 13:00:54.659395  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet e7f90c9079e24cf2a4c9ab3dd55172ea from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:00:54.660188  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet e7f90c9079e24cf2a4c9ab3dd55172ea from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-e7f90c9079e24cf2a4c9ab3dd55172ea
I20260709 13:00:54.679793  2377 raft_consensus.cc:359] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.691490  2377 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:54.699069  2377 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.701436  1752 tablet_copy_source_session.cc:215] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:54.704133  2379 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:54.706166  2381 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f90c9079e24cf2a4c9ab3dd55172ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:54.706768  2379 tablet_bootstrap.cc:492] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:54.707669  2379 ts_tablet_manager.cc:1403] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.118s	user 0.023s	sys 0.021s
I20260709 13:00:54.711793  2379 raft_consensus.cc:359] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.715034  2377 ts_tablet_manager.cc:1434] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.066s	user 0.002s	sys 0.006s
I20260709 13:00:54.720971  2379 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:54.722012  2379 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.723423  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-a9c383ed9b7c4624b8fa546326e129ff received from {username='slave'} at 127.0.0.1:56492
I20260709 13:00:54.724098  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-a9c383ed9b7c4624b8fa546326e129ff on tablet a9c383ed9b7c4624b8fa546326e129ff with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:54.733767  2373 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:54.735569  2373 tablet_bootstrap.cc:492] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:54.736661  2373 ts_tablet_manager.cc:1403] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.199s	user 0.043s	sys 0.027s
I20260709 13:00:54.739401  2379 ts_tablet_manager.cc:1434] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.007s
I20260709 13:00:54.741570  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-337669b46a6b4f75a35716542d1e20ae received from {username='slave'} at 127.0.0.1:56492
I20260709 13:00:54.741144  2373 raft_consensus.cc:359] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.742041  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-337669b46a6b4f75a35716542d1e20ae on tablet 337669b46a6b4f75a35716542d1e20ae with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:54.742192  2373 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:54.742944  2373 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.746658  2373 ts_tablet_manager.cc:1434] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:00:54.759369  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-c9e456da7a864a1b97fb4c468e75980d received from {username='slave'} at 127.0.0.1:56492
I20260709 13:00:54.759907  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-c9e456da7a864a1b97fb4c468e75980d on tablet c9e456da7a864a1b97fb4c468e75980d with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:54.758903  2381 tablet_copy_client.cc:806] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:54.760788  2381 tablet_copy_client.cc:670] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:54.766604  2381 tablet_copy_client.cc:538] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:54.778349  2318 raft_consensus.cc:1217] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:54.788341  2381 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:54.812467  2381 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:54.813588  2381 tablet_bootstrap.cc:492] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:54.814392  2381 ts_tablet_manager.cc:1403] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.009s
I20260709 13:00:54.817574  2381 raft_consensus.cc:359] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.818548  2381 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:54.819221  2381 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:54.821638  2381 ts_tablet_manager.cc:1434] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:00:54.824554  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-e7f90c9079e24cf2a4c9ab3dd55172ea received from {username='slave'} at 127.0.0.1:42028
I20260709 13:00:54.825230  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-e7f90c9079e24cf2a4c9ab3dd55172ea on tablet e7f90c9079e24cf2a4c9ab3dd55172ea with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:54.942013  2318 raft_consensus.cc:1217] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:54.962729  2318 raft_consensus.cc:1217] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:55.098938  2318 raft_consensus.cc:1217] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:55.210456  2151 raft_consensus.cc:1064] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:55.213305  2151 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.222389  2318 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.222817  1592 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.222389  1742 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.223834  2269 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:55.224840  2266 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:55.225726  2355 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:55.242563  1742 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.240327  2151 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.268785  2318 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.287449  1593 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.288553  1488 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.340974  1664 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.354877  2318 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:55.354920  1742 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:55.358547  2151 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:55.360455  2355 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:55.367213  2151 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.369948  2318 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.378675  1740 raft_consensus.cc:2955] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.384105  2266 raft_consensus.cc:1064] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:55.387327  2266 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.414920  1490 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.435302  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet c9e456da7a864a1b97fb4c468e75980d with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:00:55.456076  2403 tablet_replica.cc:333] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:00:55.458899  1592 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.477121  2403 raft_consensus.cc:2243] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:55.479032  2266 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:00:55.456636  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 49025.5us
I20260709 13:00:55.493520  2317 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.495409  2403 raft_consensus.cc:2272] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:55.500535  2403 ts_tablet_manager.cc:1916] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:55.500902  1740 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.518553  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c9e456da7a864a1b97fb4c468e75980d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:55.528414  2355 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:00:55.549916  2403 ts_tablet_manager.cc:1929] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:55.550637  2403 log.cc:1199] T c9e456da7a864a1b97fb4c468e75980d P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/c9e456da7a864a1b97fb4c468e75980d
I20260709 13:00:55.553471  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet c9e456da7a864a1b97fb4c468e75980d (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:00:55.567381  2355 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.569200  2151 raft_consensus.cc:1064] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:55.573491  2151 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.575110  2352 raft_consensus.cc:1064] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:55.620313  2355 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.606463  2352 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.628093  1742 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:00:55.729619  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70356.9us
I20260709 13:00:55.713044  2318 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.779301  1490 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:55.787719  1592 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.823186  2317 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.828029  1667 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:55.833382  2364 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.845958  1667 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.868142  1593 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.870172  2355 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.874390  2316 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.875160  1739 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:00:55.876684  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73955.2us
I20260709 13:00:55.880668  2411 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 13:00:55.884730  1591 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.892670  2317 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:55.901414  2413 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.914613  2404 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.921072  2404 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:55.934005  2404 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:55.938238  1741 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:55.946151  2355 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:55.965771  2151 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.967420  1742 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.964123  2317 raft_consensus.cc:2955] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:55.990686  2266 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.009100  2318 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.011600  1491 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:00:56.033020  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 132776us
I20260709 13:00:56.048079  1667 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.043936  2363 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.069728  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet 337669b46a6b4f75a35716542d1e20ae with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:00:56.086920  1489 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:00:56.101040  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62620.1us
W20260709 13:00:56.113031  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 108498us
W20260709 13:00:56.116029  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57297us
I20260709 13:00:56.119395  2424 tablet_replica.cc:333] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
W20260709 13:00:56.130656  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 125534us
I20260709 13:00:56.132918  1741 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.135069  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 337669b46a6b4f75a35716542d1e20ae with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:56.134879  2318 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.138975  1491 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:56.146956  1592 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:00:56.203884  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67822.6us
I20260709 13:00:56.248587  2424 raft_consensus.cc:2243] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:56.250065  2424 raft_consensus.cc:2272] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:56.258713  1587 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.261196  2424 ts_tablet_manager.cc:1916] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 13:00:56.282315  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83338.1us
I20260709 13:00:56.284073  2424 ts_tablet_manager.cc:1929] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:56.284696  2424 log.cc:1199] T 337669b46a6b4f75a35716542d1e20ae P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/337669b46a6b4f75a35716542d1e20ae
I20260709 13:00:56.309473  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet 337669b46a6b4f75a35716542d1e20ae (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:00:56.336457  1667 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:56.346741  2318 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:56.348430  1591 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:56.350690  2355 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:56.354157  2422 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:56.366255  2421 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.373282  1591 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.384019  1737 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:56.394955  2318 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:56.394202  2317 raft_consensus.cc:2955] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.396570  2413 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:56.406256  1592 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:56.408293  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a9c383ed9b7c4624b8fa546326e129ff with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:56.411510  1491 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.415118  2414 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.416783  2316 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.427120  2429 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:56.436034  1588 raft_consensus.cc:2955] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.446609  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet a9c383ed9b7c4624b8fa546326e129ff with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:00:56.467082  1489 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:56.478750  2431 tablet_replica.cc:333] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:00:56.480865  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e7f90c9079e24cf2a4c9ab3dd55172ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:56.489897  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet e7f90c9079e24cf2a4c9ab3dd55172ea with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:00:56.500875  2432 tablet_replica.cc:333] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:00:56.502426  2432 raft_consensus.cc:2243] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:56.508386  2432 raft_consensus.cc:2272] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:56.513983  2431 raft_consensus.cc:2243] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:56.516331  2431 raft_consensus.cc:2272] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:56.524101  2432 ts_tablet_manager.cc:1916] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:56.524305  2431 ts_tablet_manager.cc:1916] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:56.548218  2432 ts_tablet_manager.cc:1929] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:56.548828  2432 log.cc:1199] T e7f90c9079e24cf2a4c9ab3dd55172ea P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/e7f90c9079e24cf2a4c9ab3dd55172ea
I20260709 13:00:56.550575  2431 ts_tablet_manager.cc:1929] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:56.551926  2431 log.cc:1199] T a9c383ed9b7c4624b8fa546326e129ff P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/a9c383ed9b7c4624b8fa546326e129ff
I20260709 13:00:56.552768  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet e7f90c9079e24cf2a4c9ab3dd55172ea (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:00:56.555193  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet a9c383ed9b7c4624b8fa546326e129ff (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:00:57.576145  1737 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:00:57.584852  1706 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a5331c1cc5e4cbf8847232f0d5835c7. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:57.584947  1588 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.586333  1662 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.587013  2413 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:57.588138  2414 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:57.634764  2436 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.636734  1591 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.666509  1491 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.677023  1662 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.693023  1667 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:57.705600  1591 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.707664  2355 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:00:57.710851  1632 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88e78eac7b5f4006a15379f62bc8c536. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:57.707525  1741 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.714570  2434 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:57.732873  2422 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.736037  1591 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.739823  1737 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.775913  1489 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.802256  1591 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:57.813256  1667 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.815521  2427 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:57.816571  1741 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.818362  2427 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:00:57.845928  2430 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.848529  1740 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.852659  1667 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.874637  1490 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.880761  1740 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:57.895413  1667 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:00:57.895877  1706 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 882fdd7ea10640a78914e0647598ad9b. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:57.898674  2414 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:00:57.906117  1591 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:00:57.908381  2414 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:00:57.910751  1557 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e165279fd0cb4e84a8c9e244d087f8db. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:00:57.925742  2436 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.933303  1662 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:57.974514  1490 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:00:58.008841  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58358.3us
I20260709 13:00:58.040095  2463 ts_tablet_manager.cc:933] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:00:58.041538  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 105199us
I20260709 13:00:58.049226  2463 tablet_copy_client.cc:323] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:00:58.051505  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:00:58.052187  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 7a5331c1cc5e4cbf8847232f0d5835c7 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-7a5331c1cc5e4cbf8847232f0d5835c7
I20260709 13:00:58.061198  1752 tablet_copy_source_session.cc:215] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:58.065809  2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5331c1cc5e4cbf8847232f0d5835c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:58.086463  1588 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:00:58.118781  2463 tablet_copy_client.cc:806] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:58.119550  2463 tablet_copy_client.cc:670] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:58.123502  2463 tablet_copy_client.cc:538] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:58.134531  2463 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:58.160877  2463 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:58.162094  2463 tablet_bootstrap.cc:492] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:58.162878  2463 ts_tablet_manager.cc:1403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.029s	user 0.018s	sys 0.008s
I20260709 13:00:58.165490  2463 raft_consensus.cc:359] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.166261  2463 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:58.166844  2463 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.170149  2463 ts_tablet_manager.cc:1434] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 13:00:58.172160  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-7a5331c1cc5e4cbf8847232f0d5835c7 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:00:58.172683  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-7a5331c1cc5e4cbf8847232f0d5835c7 on tablet 7a5331c1cc5e4cbf8847232f0d5835c7 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:58.234751  2463 ts_tablet_manager.cc:933] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:00:58.237427  2463 tablet_copy_client.cc:323] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:00:58.241043  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 88e78eac7b5f4006a15379f62bc8c536 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:00:58.241652  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 88e78eac7b5f4006a15379f62bc8c536 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-88e78eac7b5f4006a15379f62bc8c536
I20260709 13:00:58.249292  1677 tablet_copy_source_session.cc:215] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:58.252656  2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88e78eac7b5f4006a15379f62bc8c536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:58.268517  2463 tablet_copy_client.cc:806] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:58.269429  2463 tablet_copy_client.cc:670] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:58.275079  2463 tablet_copy_client.cc:538] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:58.286111  2463 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:58.339776  2463 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:58.341190  2463 tablet_bootstrap.cc:492] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:58.341993  2463 ts_tablet_manager.cc:1403] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.056s	user 0.026s	sys 0.018s
I20260709 13:00:58.345860  2463 raft_consensus.cc:359] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.346796  2463 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:58.347509  2463 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.350477  2463 ts_tablet_manager.cc:1434] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:00:58.353178  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-88e78eac7b5f4006a15379f62bc8c536 received from {username='slave'} at 127.0.0.1:56492
I20260709 13:00:58.354794  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-88e78eac7b5f4006a15379f62bc8c536 on tablet 88e78eac7b5f4006a15379f62bc8c536 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:58.357277  2469 ts_tablet_manager.cc:933] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:00:58.360404  2469 tablet_copy_client.cc:323] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:41267
I20260709 13:00:58.378089  2463 ts_tablet_manager.cc:933] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:00:58.397147  2463 tablet_copy_client.cc:323] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:00:58.399217  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 882fdd7ea10640a78914e0647598ad9b from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:00:58.399838  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 882fdd7ea10640a78914e0647598ad9b from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-882fdd7ea10640a78914e0647598ad9b
I20260709 13:00:58.409101  1752 tablet_copy_source_session.cc:215] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:58.410969  1603 tablet_copy_service.cc:140] P 316f3bf18ed94e32b238130a6bfedf1a: Received BeginTabletCopySession request for tablet e165279fd0cb4e84a8c9e244d087f8db from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:46322)
I20260709 13:00:58.411652  1603 tablet_copy_service.cc:161] P 316f3bf18ed94e32b238130a6bfedf1a: Beginning new tablet copy session on tablet e165279fd0cb4e84a8c9e244d087f8db from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:46322: session id = e6adef9da7da44f6af0af5695cf9b704-e165279fd0cb4e84a8c9e244d087f8db
I20260709 13:00:58.413270  2463 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882fdd7ea10640a78914e0647598ad9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:58.419062  1603 tablet_copy_source_session.cc:215] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:00:58.423054  2469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e165279fd0cb4e84a8c9e244d087f8db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:00:58.439946  2463 tablet_copy_client.cc:806] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:58.440755  2463 tablet_copy_client.cc:670] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:58.444712  2469 tablet_copy_client.cc:806] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:00:58.445148  2463 tablet_copy_client.cc:538] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:58.445429  2469 tablet_copy_client.cc:670] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:00:58.450986  2469 tablet_copy_client.cc:538] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:00:58.456959  2463 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:58.469347  2469 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:00:58.480161  2463 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:58.481168  2463 tablet_bootstrap.cc:492] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:58.481910  2463 ts_tablet_manager.cc:1403] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.025s	user 0.026s	sys 0.000s
I20260709 13:00:58.487979  2463 raft_consensus.cc:359] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.489037  2463 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:58.489846  2463 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.498153  2463 ts_tablet_manager.cc:1434] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.000s
I20260709 13:00:58.501386  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-882fdd7ea10640a78914e0647598ad9b received from {username='slave'} at 127.0.0.1:42028
I20260709 13:00:58.501631  2469 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:00:58.501942  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-882fdd7ea10640a78914e0647598ad9b on tablet 882fdd7ea10640a78914e0647598ad9b with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:58.502758  2469 tablet_bootstrap.cc:492] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:00:58.503599  2469 ts_tablet_manager.cc:1403] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.035s	user 0.024s	sys 0.008s
I20260709 13:00:58.507088  2469 raft_consensus.cc:359] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.507961  2469 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:00:58.508628  2469 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:00:58.511399  2469 ts_tablet_manager.cc:1434] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:00:58.513180  1603 tablet_copy_service.cc:342] P 316f3bf18ed94e32b238130a6bfedf1a: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-e165279fd0cb4e84a8c9e244d087f8db received from {username='slave'} at 127.0.0.1:46322
I20260709 13:00:58.513664  1603 tablet_copy_service.cc:434] P 316f3bf18ed94e32b238130a6bfedf1a: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-e165279fd0cb4e84a8c9e244d087f8db on tablet e165279fd0cb4e84a8c9e244d087f8db with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:00:58.538703  2316 raft_consensus.cc:1217] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:58.725181  2316 raft_consensus.cc:1217] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:58.757854  2316 raft_consensus.cc:1217] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:58.976233  2316 raft_consensus.cc:1217] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:00:59.067893  2476 raft_consensus.cc:1064] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:59.071789  2476 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.092142  2316 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.095245  1662 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.096064  2459 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:59.099314  2461 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:59.101956  1588 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.106683  2461 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.156188  2446 raft_consensus.cc:1064] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:59.163472  2446 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.182607  2461 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.194419  1662 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.215850  2316 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.232003  1490 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.265518  2316 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.270011  1741 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.269923  2434 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:59.271788  2481 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.293648  1592 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.295377  2481 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.278512  1588 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.339118  2481 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.340916  1592 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.350019  2430 raft_consensus.cc:1064] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:59.352952  2430 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.375038  2317 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.390369  1741 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.410379  1741 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.411130  2316 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.413578  2427 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.416173  2490 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.419497  1662 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.421176  2491 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:00:59.420596  1488 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.432140  1737 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.440478  1592 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.444353  2317 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.447402  2413 raft_consensus.cc:1064] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:00:59.442866  2492 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 13:00:59.451460  2461 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:59.451716  2413 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.475001  2491 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.481081  1662 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.483492  1667 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.488523  1741 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.490037  1588 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.489290  2459 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.500073  1664 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.501060  1592 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.503332  2317 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:00:59.504312  2318 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.506013  2413 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:00:59.505391  2316 raft_consensus.cc:2955] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.512941  2492 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:59.519320  2486 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:00:59.516288  1491 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.531535  2314 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.531517  1592 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.534487  2481 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:59.538158  2422 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:59.544373  2434 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:00:59.549814  2500 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:00:59.552961  1587 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.568769  1592 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.569988  2314 raft_consensus.cc:2955] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.589776  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:59.593447  1491 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.615548  2315 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.618135  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 88e78eac7b5f4006a15379f62bc8c536 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:59.620628  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a5331c1cc5e4cbf8847232f0d5835c7 with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:00:59.624405  2491 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:59.635167  2485 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.637382  1662 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.640936  2430 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.646371  1665 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.648809  2317 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.640411  1489 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.679042  2503 tablet_replica.cc:333] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:00:59.686480  2484 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:00:59.687103  1587 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.691938  1478 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e165279fd0cb4e84a8c9e244d087f8db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:59.724169  1665 raft_consensus.cc:2955] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.730943  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet 88e78eac7b5f4006a15379f62bc8c536 with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:00:59.730545  2316 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.722731  2503 raft_consensus.cc:2243] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:00:59.725903  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86098.1us
I20260709 13:00:59.750069  1489 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.769867  2509 tablet_replica.cc:333] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:00:59.771225  2509 raft_consensus.cc:2243] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:59.772189  2509 raft_consensus.cc:2272] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:59.773911  2503 raft_consensus.cc:2272] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:59.781853  2509 ts_tablet_manager.cc:1916] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:59.782421  1488 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.792495  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet e165279fd0cb4e84a8c9e244d087f8db with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:00:59.782166  2503 ts_tablet_manager.cc:1916] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:59.841949  1740 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:00:59.852447  2317 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.854379  2485 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:00:59.864815  2413 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.879093  2317 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.895128  1662 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:00:59.893327  2509 ts_tablet_manager.cc:1929] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:59.898561  2509 log.cc:1199] T 88e78eac7b5f4006a15379f62bc8c536 P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/88e78eac7b5f4006a15379f62bc8c536
I20260709 13:00:59.901188  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet 88e78eac7b5f4006a15379f62bc8c536 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:00:59.907271  2509 tablet_replica.cc:333] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:00:59.909088  2509 raft_consensus.cc:2243] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:00:59.910662  2509 raft_consensus.cc:2272] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:00:59.913995  2492 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:00:59.914942  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 882fdd7ea10640a78914e0647598ad9b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:00:59.916502  2509 ts_tablet_manager.cc:1916] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:00:59.929212  1665 raft_consensus.cc:2955] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:00:59.932636  1490 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:00:59.970044  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet 882fdd7ea10640a78914e0647598ad9b with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:00:59.972268  2503 ts_tablet_manager.cc:1929] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:00:59.972817  2503 log.cc:1199] T 7a5331c1cc5e4cbf8847232f0d5835c7 P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/7a5331c1cc5e4cbf8847232f0d5835c7
W20260709 13:00:59.982743  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 192392us
I20260709 13:01:00.025996  2509 ts_tablet_manager.cc:1929] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:00.028692  2509 log.cc:1199] T e165279fd0cb4e84a8c9e244d087f8db P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/e165279fd0cb4e84a8c9e244d087f8db
I20260709 13:01:00.050516  2514 tablet_replica.cc:333] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:00.052058  2514 raft_consensus.cc:2243] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:00.053092  2514 raft_consensus.cc:2272] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:00.062345  2514 ts_tablet_manager.cc:1916] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:00.064054  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet e165279fd0cb4e84a8c9e244d087f8db (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:00.074400  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet 7a5331c1cc5e4cbf8847232f0d5835c7 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:00.090354  2514 ts_tablet_manager.cc:1929] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:00.090992  2514 log.cc:1199] T 882fdd7ea10640a78914e0647598ad9b P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/882fdd7ea10640a78914e0647598ad9b
I20260709 13:01:00.092826  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet 882fdd7ea10640a78914e0647598ad9b (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:01.051664  1737 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:01.058909  1706 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce79fa9e7595481c8379fd2bdeca8a6e. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:01.059063  1587 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.059685  1662 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.060372  2476 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:01.061232  2515 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:01.070214  2515 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.071470  1587 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.071388  1662 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.087713  1592 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.092413  1490 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.109104  1662 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:01.110190  1557 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbe322e4447d486680550d1490266b17. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:01.110610  1737 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.111614  2430 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:01.113113  2427 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:01.129920  2427 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.133106  1662 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.145311  1741 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.155611  1665 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.162722  1489 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:01:01.174636  1632 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d857fff084d3451d99a39db56ccd9a37. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:01.175413  1737 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.175603  1587 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.177139  2516 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:01.179905  2517 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:01.189877  2517 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.191747  1737 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.199762  1592 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.210556  1737 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.214186  1491 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.223661  1665 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:01.224898  1706 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 219129ac643e4650b611a85f565f5990. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:01.224933  1592 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:01.226806  2486 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:01.227923  2515 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:01.237905  2486 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.286842  1489 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:01.254070  1592 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.295300  1665 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:01.608954  2539 ts_tablet_manager.cc:933] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:01.624948  2539 tablet_copy_client.cc:323] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:01.631809  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 219129ac643e4650b611a85f565f5990 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:01.632635  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 219129ac643e4650b611a85f565f5990 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-219129ac643e4650b611a85f565f5990
I20260709 13:01:01.643877  2541 ts_tablet_manager.cc:933] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:01.644160  1752 tablet_copy_source_session.cc:215] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:01.647444  2541 tablet_copy_client.cc:323] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:01.649441  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet ce79fa9e7595481c8379fd2bdeca8a6e from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:01.649554  2539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 219129ac643e4650b611a85f565f5990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:01.650182  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet ce79fa9e7595481c8379fd2bdeca8a6e from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-ce79fa9e7595481c8379fd2bdeca8a6e
I20260709 13:01:01.660075  1752 tablet_copy_source_session.cc:215] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:01.663990  2541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce79fa9e7595481c8379fd2bdeca8a6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:01.667868  2539 tablet_copy_client.cc:806] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:01.668744  2539 tablet_copy_client.cc:670] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:01.681156  2539 tablet_copy_client.cc:538] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:01.681142  2543 ts_tablet_manager.cc:933] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:01.698408  2543 tablet_copy_client.cc:323] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:01:01.701309  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet d857fff084d3451d99a39db56ccd9a37 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:01.701905  2541 tablet_copy_client.cc:806] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:01.702100  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet d857fff084d3451d99a39db56ccd9a37 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-d857fff084d3451d99a39db56ccd9a37
I20260709 13:01:01.702687  2541 tablet_copy_client.cc:670] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:01.703454  2539 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:01.707159  2541 tablet_copy_client.cc:538] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:01.723083  2541 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:01.752068  1677 tablet_copy_source_session.cc:215] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:01.757115  2543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d857fff084d3451d99a39db56ccd9a37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:01.757198  2545 ts_tablet_manager.cc:933] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:01.780267  2539 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:01.781635  2539 tablet_bootstrap.cc:492] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:01.787366  2539 ts_tablet_manager.cc:1403] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.084s	user 0.026s	sys 0.007s
I20260709 13:01:01.797410  2541 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:01.801367  2543 tablet_copy_client.cc:806] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:01.802446  2543 tablet_copy_client.cc:670] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:01.797847  2545 tablet_copy_client.cc:323] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:41267
I20260709 13:01:01.804487  2541 tablet_bootstrap.cc:492] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:01.810871  2543 tablet_copy_client.cc:538] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:01.816591  2541 ts_tablet_manager.cc:1403] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.094s	user 0.029s	sys 0.009s
I20260709 13:01:01.818130  1603 tablet_copy_service.cc:140] P 316f3bf18ed94e32b238130a6bfedf1a: Received BeginTabletCopySession request for tablet fbe322e4447d486680550d1490266b17 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:46322)
I20260709 13:01:01.818864  1603 tablet_copy_service.cc:161] P 316f3bf18ed94e32b238130a6bfedf1a: Beginning new tablet copy session on tablet fbe322e4447d486680550d1490266b17 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:46322: session id = e6adef9da7da44f6af0af5695cf9b704-fbe322e4447d486680550d1490266b17
I20260709 13:01:01.823469  2541 raft_consensus.cc:359] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.824733  2541 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:01.825785  2541 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.829977  1603 tablet_copy_source_session.cc:215] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:01.841930  2545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbe322e4447d486680550d1490266b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:01.846480  2541 ts_tablet_manager.cc:1434] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.029s	user 0.014s	sys 0.000s
I20260709 13:01:01.814802  2539 raft_consensus.cc:359] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.848568  2539 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:01.849340  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-ce79fa9e7595481c8379fd2bdeca8a6e received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:01.849918  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-ce79fa9e7595481c8379fd2bdeca8a6e on tablet ce79fa9e7595481c8379fd2bdeca8a6e with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:01.856763  2539 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.859314  2539 ts_tablet_manager.cc:1434] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.071s	user 0.004s	sys 0.004s
I20260709 13:01:01.861140  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-219129ac643e4650b611a85f565f5990 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:01.861626  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-219129ac643e4650b611a85f565f5990 on tablet 219129ac643e4650b611a85f565f5990 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:01.868151  2543 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:01.921466  2543 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:01.922575  2543 tablet_bootstrap.cc:492] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:01.923341  2543 ts_tablet_manager.cc:1403] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.055s	user 0.022s	sys 0.011s
I20260709 13:01:01.926790  2543 raft_consensus.cc:359] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.927657  2543 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:01.928330  2543 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.930971  2543 ts_tablet_manager.cc:1434] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:01:01.933557  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-d857fff084d3451d99a39db56ccd9a37 received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:01.934006  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-d857fff084d3451d99a39db56ccd9a37 on tablet d857fff084d3451d99a39db56ccd9a37 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:01.946470  2545 tablet_copy_client.cc:806] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:01.947355  2545 tablet_copy_client.cc:670] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:01.952407  2545 tablet_copy_client.cc:538] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:01.964785  2545 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:01.988989  2545 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:01.990453  2545 tablet_bootstrap.cc:492] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:01.991408  2545 ts_tablet_manager.cc:1403] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.015s
I20260709 13:01:01.994309  2545 raft_consensus.cc:359] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.995060  2545 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:01.995885  2545 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:01.998852  2545 ts_tablet_manager.cc:1434] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:01:02.001458  1603 tablet_copy_service.cc:342] P 316f3bf18ed94e32b238130a6bfedf1a: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-fbe322e4447d486680550d1490266b17 received from {username='slave'} at 127.0.0.1:46322
I20260709 13:01:02.002215  1603 tablet_copy_service.cc:434] P 316f3bf18ed94e32b238130a6bfedf1a: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-fbe322e4447d486680550d1490266b17 on tablet fbe322e4447d486680550d1490266b17 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:02.055707  2316 raft_consensus.cc:1217] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:02.085548  2316 raft_consensus.cc:1217] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:02.224380  2316 raft_consensus.cc:1217] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:02.305966  2316 raft_consensus.cc:1217] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:02.581889  2476 raft_consensus.cc:1064] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:02.584916  2476 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:02.596033  2316 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.597816  2535 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.607311  1661 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.613749  1589 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.616014  2486 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.618808  2486 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:02.641288  2476 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.643270  1665 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.650560  2535 raft_consensus.cc:1064] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:02.654440  2535 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:02.650326  1589 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.674070  1491 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.676824  2507 raft_consensus.cc:1064] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:02.681043  2316 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.683763  2507 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:02.696820  2317 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.699916  1661 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.701916  2554 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.707571  2554 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:02.773608  1589 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.794554  2554 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:02.795284  1737 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:02.803455  2317 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:02.805757  2554 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:02.810215  1592 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:02.814656  2559 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:02.816311  2476 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.820451  2314 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.837450  1592 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.839182  1737 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.845839  2570 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.856839  1490 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:02.865763  2556 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.867532  2528 raft_consensus.cc:1064] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:02.872108  2528 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:02.858349  2317 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:02.882014  2571 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:02.888473  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ce79fa9e7595481c8379fd2bdeca8a6e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:02.896073  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet ce79fa9e7595481c8379fd2bdeca8a6e with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:02.965690  1741 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.968122  2576 tablet_replica.cc:333] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:02.961410  2517 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:02.980166  1592 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.025596  2576 raft_consensus.cc:2243] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:03.026605  2576 raft_consensus.cc:2272] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:03.033859  2576 ts_tablet_manager.cc:1916] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:03.086021  1665 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:03.090618  2314 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.099748  2317 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:03.110860  2578 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260709 13:01:03.116185  1490 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.117102  2427 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 13:01:03.159662  2314 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.185310  1741 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:03.156190  2476 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.245571  2427 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:03.262832  2576 ts_tablet_manager.cc:1929] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:03.263447  2576 log.cc:1199] T ce79fa9e7595481c8379fd2bdeca8a6e P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/ce79fa9e7595481c8379fd2bdeca8a6e
I20260709 13:01:03.269150  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet ce79fa9e7595481c8379fd2bdeca8a6e (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
W20260709 13:01:03.281288  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 159269us
I20260709 13:01:03.287295  1665 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:03.315403  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69155.5us
I20260709 13:01:03.318156  2590 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.319242  2316 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.331537  1592 raft_consensus.cc:2955] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.347167  1661 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.360364  1665 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:03.370067  2316 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.374327  1739 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.387621  1591 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.392925  2563 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:03.395853  2585 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:03.396639  1488 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.401965  1490 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:03.440568  1591 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:03.447374  2563 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.459638  2316 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.465651  2590 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:03.470181  1739 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.476097  2587 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.499204  2316 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.507367  2314 raft_consensus.cc:2955] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.524680  1665 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.543648  1478 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fbe322e4447d486680550d1490266b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:03.545921  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d857fff084d3451d99a39db56ccd9a37 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:03.545305  1489 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.570171  2587 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:03.581943  1665 raft_consensus.cc:2955] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.582302  1491 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:03.593914  1741 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:03.619233  2316 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.622807  2560 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 13:01:03.627317  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet d857fff084d3451d99a39db56ccd9a37 with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:01:03.644801  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet fbe322e4447d486680550d1490266b17 with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:01:03.650403  1589 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:03.656858  2560 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:03.666168  2476 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.672448  2316 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.680029  2594 tablet_replica.cc:333] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:01:03.683135  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 219129ac643e4650b611a85f565f5990 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:03.682309  1589 raft_consensus.cc:2955] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:03.698812  2594 raft_consensus.cc:2243] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:03.702775  2593 tablet_replica.cc:333] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:03.706779  1490 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:03.717823  2594 pending_rounds.cc:70] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Trying to abort 1 pending ops.
I20260709 13:01:03.718384  2594 pending_rounds.cc:77] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7305634049510588416 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "fbe322e4447d486680550d1490266b17" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } } }
I20260709 13:01:03.719961  2594 raft_consensus.cc:2917] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260709 13:01:03.720507  2594 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 13:01:03.720938  2594 raft_consensus.cc:2272] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:03.731245  2594 ts_tablet_manager.cc:1916] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:03.731837  2593 raft_consensus.cc:2243] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:03.732789  2593 raft_consensus.cc:2272] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:03.736156  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet 219129ac643e4650b611a85f565f5990 with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:03.737317  2576 tablet_replica.cc:333] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:03.738487  2576 raft_consensus.cc:2243] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:03.739033  2593 ts_tablet_manager.cc:1916] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:03.739387  2576 raft_consensus.cc:2272] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:03.744024  2576 ts_tablet_manager.cc:1916] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:03.778602  2593 ts_tablet_manager.cc:1929] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:03.779112  2593 log.cc:1199] T d857fff084d3451d99a39db56ccd9a37 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/d857fff084d3451d99a39db56ccd9a37
I20260709 13:01:03.781065  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet d857fff084d3451d99a39db56ccd9a37 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:03.781731  2576 ts_tablet_manager.cc:1929] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:03.782194  2576 log.cc:1199] T 219129ac643e4650b611a85f565f5990 P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/219129ac643e4650b611a85f565f5990
I20260709 13:01:03.784554  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet 219129ac643e4650b611a85f565f5990 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:03.786787  2594 ts_tablet_manager.cc:1929] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:03.787238  2594 log.cc:1199] T fbe322e4447d486680550d1490266b17 P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/fbe322e4447d486680550d1490266b17
I20260709 13:01:03.788970  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet fbe322e4447d486680550d1490266b17 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:04.823257  1661 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:04.832279  1737 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:04.832050  1589 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:04.834559  1632 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8e75da90507436ea52296522f44c15e. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:04.835479  2585 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:04.837054  2597 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:04.852942  2585 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:04.855574  1741 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:04.863389  1587 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:04.878955  1490 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:04.885336  1589 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:04.942108  1557 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 873c3a32540d4c2cb6d882298cb1eedc. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:04.942873  1661 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:04.943203  1741 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:04.944633  2427 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:04.945622  2595 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:04.961903  2595 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:04.968876  1741 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.016245  1491 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.021354  1741 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.042018  1665 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:01:05.048282  1706 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9c6fca959014bb6a785fac033e5fe47. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:05.052351  1661 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:05.069926  1589 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:05.072316  2559 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:05.073731  2610 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:05.085413  2610 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.087879  1589 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.089874  1661 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.113251  1741 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.116659  1491 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.135768  1661 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:05.136466  1706 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce1eb834927c4d3caa029344d6f69de5. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:05.137964  2559 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:05.139654  1589 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:05.144001  2609 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:05.158600  2610 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.161278  1665 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.161249  1587 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:05.182065  1489 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:05.233531  2618 ts_tablet_manager.cc:933] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:05.237834  2618 tablet_copy_client.cc:323] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:01:05.239817  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet b8e75da90507436ea52296522f44c15e from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:05.240561  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet b8e75da90507436ea52296522f44c15e from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-b8e75da90507436ea52296522f44c15e
I20260709 13:01:05.250365  1677 tablet_copy_source_session.cc:215] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:05.255474  2618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8e75da90507436ea52296522f44c15e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:05.285367  2618 tablet_copy_client.cc:806] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:05.286106  2618 tablet_copy_client.cc:670] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:05.291095  2618 tablet_copy_client.cc:538] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:05.301849  2618 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:05.322898  2618 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:05.323975  2618 tablet_bootstrap.cc:492] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:05.324999  2618 ts_tablet_manager.cc:1403] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.023s	user 0.018s	sys 0.004s
I20260709 13:01:05.327844  2618 raft_consensus.cc:359] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.328936  2618 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:05.330070  2618 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.350529  2618 ts_tablet_manager.cc:1434] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 13:01:05.352620  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-b8e75da90507436ea52296522f44c15e received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:05.353091  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-b8e75da90507436ea52296522f44c15e on tablet b8e75da90507436ea52296522f44c15e with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:05.499655  2618 ts_tablet_manager.cc:933] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:05.501575  2618 tablet_copy_client.cc:323] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:05.503249  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet b9c6fca959014bb6a785fac033e5fe47 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:05.503798  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet b9c6fca959014bb6a785fac033e5fe47 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-b9c6fca959014bb6a785fac033e5fe47
I20260709 13:01:05.510017  2623 ts_tablet_manager.cc:933] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:05.512298  2623 tablet_copy_client.cc:323] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.193:41267
I20260709 13:01:05.512323  1752 tablet_copy_source_session.cc:215] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:05.514042  1603 tablet_copy_service.cc:140] P 316f3bf18ed94e32b238130a6bfedf1a: Received BeginTabletCopySession request for tablet 873c3a32540d4c2cb6d882298cb1eedc from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:46322)
I20260709 13:01:05.514559  1603 tablet_copy_service.cc:161] P 316f3bf18ed94e32b238130a6bfedf1a: Beginning new tablet copy session on tablet 873c3a32540d4c2cb6d882298cb1eedc from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:46322: session id = e6adef9da7da44f6af0af5695cf9b704-873c3a32540d4c2cb6d882298cb1eedc
I20260709 13:01:05.516950  2618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c6fca959014bb6a785fac033e5fe47. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:05.526537  1603 tablet_copy_source_session.cc:215] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:05.534161  2623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873c3a32540d4c2cb6d882298cb1eedc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:05.552481  2623 tablet_copy_client.cc:806] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:05.553465  2623 tablet_copy_client.cc:670] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:05.553762  2618 tablet_copy_client.cc:806] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:05.554382  2618 tablet_copy_client.cc:670] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:05.559577  2618 tablet_copy_client.cc:538] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:05.565626  2623 tablet_copy_client.cc:538] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:05.570413  2618 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:05.579522  2625 ts_tablet_manager.cc:933] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:05.588146  2623 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:05.588553  2625 tablet_copy_client.cc:323] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:05.592280  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet ce1eb834927c4d3caa029344d6f69de5 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:05.592826  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet ce1eb834927c4d3caa029344d6f69de5 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-ce1eb834927c4d3caa029344d6f69de5
I20260709 13:01:05.622711  2316 raft_consensus.cc:1217] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:05.629206  2618 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:05.657545  1752 tablet_copy_source_session.cc:215] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:05.630702  2618 tablet_bootstrap.cc:492] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:05.670814  2618 ts_tablet_manager.cc:1403] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.101s	user 0.018s	sys 0.024s
I20260709 13:01:05.678313  2625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce1eb834927c4d3caa029344d6f69de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:05.681445  2618 raft_consensus.cc:359] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.695778  2618 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:05.700745  2623 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:05.701850  2623 tablet_bootstrap.cc:492] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:05.702615  2623 ts_tablet_manager.cc:1403] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.115s	user 0.007s	sys 0.024s
I20260709 13:01:05.699126  2618 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.709035  2618 ts_tablet_manager.cc:1434] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 13:01:05.710191  2623 raft_consensus.cc:359] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.711330  2623 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:05.711464  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-b9c6fca959014bb6a785fac033e5fe47 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:05.712333  2623 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.713130  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-b9c6fca959014bb6a785fac033e5fe47 on tablet b9c6fca959014bb6a785fac033e5fe47 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:05.715592  2623 ts_tablet_manager.cc:1434] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 13:01:05.719666  1603 tablet_copy_service.cc:342] P 316f3bf18ed94e32b238130a6bfedf1a: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-873c3a32540d4c2cb6d882298cb1eedc received from {username='slave'} at 127.0.0.1:46322
I20260709 13:01:05.720209  1603 tablet_copy_service.cc:434] P 316f3bf18ed94e32b238130a6bfedf1a: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-873c3a32540d4c2cb6d882298cb1eedc on tablet 873c3a32540d4c2cb6d882298cb1eedc with peer e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:05.724560  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49771.8us
I20260709 13:01:05.733918  2625 tablet_copy_client.cc:806] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:05.734630  2625 tablet_copy_client.cc:670] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:05.745208  2625 tablet_copy_client.cc:538] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:05.768239  2625 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:05.794298  2625 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:05.795630  2625 tablet_bootstrap.cc:492] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:05.796244  2625 ts_tablet_manager.cc:1403] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.028s	user 0.026s	sys 0.001s
I20260709 13:01:05.799779  2625 raft_consensus.cc:359] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.800804  2625 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:05.801401  2625 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:05.803870  2625 ts_tablet_manager.cc:1434] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:01:05.805778  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-ce1eb834927c4d3caa029344d6f69de5 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:05.806231  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-ce1eb834927c4d3caa029344d6f69de5 on tablet ce1eb834927c4d3caa029344d6f69de5 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:05.887751  2316 raft_consensus.cc:1217] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:05.966837  2316 raft_consensus.cc:1217] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:05.984854  2316 raft_consensus.cc:1217] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:06.252848  2597 raft_consensus.cc:1064] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:06.256533  2597 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.285140  2316 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.288552  1587 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.290844  2628 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:06.288676  1741 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.293776  2585 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:06.295424  2634 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:06.333345  2597 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.338584  2316 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.345721  2633 raft_consensus.cc:1064] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:06.347908  1739 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.357338  1491 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.349362  2633 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.361158  1591 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.394383  2314 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.396615  2427 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:06.396888  1741 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.409238  1661 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.418586  2611 raft_consensus.cc:1064] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:06.418708  2604 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:06.421459  2611 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.445233  1661 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:06.446599  2641 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:01:06.447765  2611 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 13:01:06.448680  2316 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:06.454259  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 57954.9us
I20260709 13:01:06.449023  1587 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.461548  2611 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:06.463263  2629 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:06.476444  2629 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.478744  2317 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.486591  2427 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:06.495920  2610 raft_consensus.cc:1064] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:06.500662  2610 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.513185  2642 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.520021  2314 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.561023  2317 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.563073  2559 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:06.562124  1661 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.572927  1667 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.556066  1665 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.583276  2657 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 13:01:06.609139  1489 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.611193  1489 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.623055  1739 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:06.636050  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90049.6us
I20260709 13:01:06.642277  1662 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.649302  1589 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:06.652179  1589 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.663367  1591 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:06.663480  2314 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:06.667824  2657 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:06.672016  1588 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.673702  2635 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:06.695641  2585 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:06.727599  1587 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.738729  1739 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:06.733125  2635 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.747547  2633 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:06.750494  2317 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:01:06.756008  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 71480.6us
I20260709 13:01:06.770424  2604 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 13:01:06.778127  2611 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.794353  1741 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:06.800578  1489 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.803884  2633 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.817157  2314 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.834441  2315 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 13:01:06.834981  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47774.6us
I20260709 13:01:06.823179  2317 raft_consensus.cc:2955] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.843751  2653 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:06.851804  1667 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.855479  2655 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.859781  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b8e75da90507436ea52296522f44c15e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:06.867410  2316 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.887923  1478 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 873c3a32540d4c2cb6d882298cb1eedc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:06.888118  1491 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.906508  1488 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:06.911500  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet ce1eb834927c4d3caa029344d6f69de5 with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:06.916113  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet b8e75da90507436ea52296522f44c15e with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:01:06.920836  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ce1eb834927c4d3caa029344d6f69de5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:06.926687  1589 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:06.934999  1741 raft_consensus.cc:2955] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.947535  2629 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:06.953969  1652 tablet_service.cc:1558] Processing DeleteTablet for tablet 873c3a32540d4c2cb6d882298cb1eedc with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:06.958510  2664 tablet_replica.cc:333] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:06.959628  2664 raft_consensus.cc:2243] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:06.958881  1588 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.960496  2664 raft_consensus.cc:2272] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:06.962909  2665 tablet_replica.cc:333] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:01:06.964129  2665 raft_consensus.cc:2243] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:06.965010  2665 raft_consensus.cc:2272] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:06.967398  2317 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.972512  1592 raft_consensus.cc:2955] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:06.980690  2665 ts_tablet_manager.cc:1916] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:06.984947  2664 ts_tablet_manager.cc:1916] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:07.015079  1489 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:07.047657  2664 ts_tablet_manager.cc:1929] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:07.055958  2664 log.cc:1199] T ce1eb834927c4d3caa029344d6f69de5 P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/ce1eb834927c4d3caa029344d6f69de5
I20260709 13:01:07.060503  2664 tablet_replica.cc:333] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:07.061722  2664 raft_consensus.cc:2243] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:07.063334  2664 raft_consensus.cc:2272] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:07.067893  2664 ts_tablet_manager.cc:1916] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:07.062099  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet ce1eb834927c4d3caa029344d6f69de5 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:07.081770  2665 ts_tablet_manager.cc:1929] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:07.082562  2665 log.cc:1199] T b8e75da90507436ea52296522f44c15e P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/b8e75da90507436ea52296522f44c15e
I20260709 13:01:07.084354  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet b8e75da90507436ea52296522f44c15e (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:07.097146  2664 ts_tablet_manager.cc:1929] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:07.097889  2664 log.cc:1199] T 873c3a32540d4c2cb6d882298cb1eedc P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/873c3a32540d4c2cb6d882298cb1eedc
I20260709 13:01:07.097492  1741 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:07.099606  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet 873c3a32540d4c2cb6d882298cb1eedc (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:07.105154  2314 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:07.105664  1667 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:07.106984  2611 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:07.108990  2610 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:07.119689  2629 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:07.121112  1667 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:07.123190  2314 raft_consensus.cc:2955] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:07.137415  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b9c6fca959014bb6a785fac033e5fe47 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:07.145946  1491 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:07.169214  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet b9c6fca959014bb6a785fac033e5fe47 with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:01:07.181660  2666 tablet_replica.cc:333] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:07.189880  2666 raft_consensus.cc:2243] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:07.191939  2666 raft_consensus.cc:2272] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:07.196559  2666 ts_tablet_manager.cc:1916] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:07.211548  2666 ts_tablet_manager.cc:1929] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:07.211946  2666 log.cc:1199] T b9c6fca959014bb6a785fac033e5fe47 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/b9c6fca959014bb6a785fac033e5fe47
I20260709 13:01:07.213621  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet b9c6fca959014bb6a785fac033e5fe47 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:08.267541  1741 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:08.281968  1706 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4817dbc3c2e469195aa3f5e05a1534a. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:08.282734  1665 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:08.283890  1588 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:08.284962  2668 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:08.287312  2669 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:08.311426  2668 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:08.317449  1589 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:08.320344  1667 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:08.339607  1665 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:08.340138  1488 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:08.359318  1632 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2dffe36aa52848e8962262b0e01bc346. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:08.361703  1741 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:08.363430  2662 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:08.364732  1588 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:08.368211  2667 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:08.385726  2662 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:08.397423  1737 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:08.797398  2688 ts_tablet_manager.cc:933] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:08.807685  2688 tablet_copy_client.cc:323] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:08.811524  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet d4817dbc3c2e469195aa3f5e05a1534a from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:08.812383  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet d4817dbc3c2e469195aa3f5e05a1534a from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-d4817dbc3c2e469195aa3f5e05a1534a
I20260709 13:01:08.839319  1752 tablet_copy_source_session.cc:215] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:08.901153  2688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4817dbc3c2e469195aa3f5e05a1534a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:01:08.904633  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67189.6us
W20260709 13:01:08.925037  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.023s	sys 0.004s
W20260709 13:01:08.925516  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.024s	sys 0.004s
I20260709 13:01:08.930891  2689 ts_tablet_manager.cc:933] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:08.935043  2688 tablet_copy_client.cc:806] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:08.936115  2688 tablet_copy_client.cc:670] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:08.960739  2689 tablet_copy_client.cc:323] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:01:08.963132  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 2dffe36aa52848e8962262b0e01bc346 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:08.963809  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 2dffe36aa52848e8962262b0e01bc346 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-2dffe36aa52848e8962262b0e01bc346
I20260709 13:01:08.970512  2688 tablet_copy_client.cc:538] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:01:09.001639  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52336.5us
I20260709 13:01:09.012246  1740 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:09.037178  1706 consensus_peers.cc:597] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5a2ef920c894818b430473b4809d61d. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:09.044781  1677 tablet_copy_source_session.cc:215] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:09.047755  1491 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:09.049173  2689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dffe36aa52848e8962262b0e01bc346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:09.083671  1585 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.100971  2688 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:09.107052  1587 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.116183  1664 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.116585  2669 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:09.146461  2669 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:09.147092  2689 tablet_copy_client.cc:806] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:09.154242  2689 tablet_copy_client.cc:670] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:09.167359  2689 tablet_copy_client.cc:538] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:09.184774  2669 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.200429  1667 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.213063  1589 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.212232  1740 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.243871  2689 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:09.313127  2688 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:09.314491  1488 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:01:09.324807  1706 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92fb459de10a4e038d6cbee7b88dc3ba. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:09.314534  2688 tablet_bootstrap.cc:492] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:09.328675  2688 ts_tablet_manager.cc:1403] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.228s	user 0.033s	sys 0.022s
I20260709 13:01:09.331612  1665 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.332008  2688 raft_consensus.cc:359] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.333144  2688 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:09.333074  2611 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:09.334059  2688 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.350304  2688 ts_tablet_manager.cc:1434] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.021s	user 0.012s	sys 0.003s
I20260709 13:01:09.353108  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-d4817dbc3c2e469195aa3f5e05a1534a received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:09.354143  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-d4817dbc3c2e469195aa3f5e05a1534a on tablet d4817dbc3c2e469195aa3f5e05a1534a with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:09.363245  1585 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:09.382925  2611 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:09.432602  2688 ts_tablet_manager.cc:933] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:09.439541  1665 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.436513  2669 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.446135  1585 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.455963  2689 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:09.457118  2689 tablet_bootstrap.cc:492] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:09.457962  2689 ts_tablet_manager.cc:1403] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.214s	user 0.053s	sys 0.074s
I20260709 13:01:09.460811  1490 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:09.462862  2688 tablet_copy_client.cc:323] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:09.464622  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet f5a2ef920c894818b430473b4809d61d from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:09.464500  2689 raft_consensus.cc:359] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.465481  2689 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:09.466181  2689 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.465277  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet f5a2ef920c894818b430473b4809d61d from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-f5a2ef920c894818b430473b4809d61d
I20260709 13:01:09.477635  2689 ts_tablet_manager.cc:1434] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.019s	user 0.001s	sys 0.008s
I20260709 13:01:09.481426  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-2dffe36aa52848e8962262b0e01bc346 received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:09.482005  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-2dffe36aa52848e8962262b0e01bc346 on tablet 2dffe36aa52848e8962262b0e01bc346 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:09.519045  1752 tablet_copy_source_session.cc:215] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:09.532172  2688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5a2ef920c894818b430473b4809d61d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:09.582204  2688 tablet_copy_client.cc:806] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:09.583024  2688 tablet_copy_client.cc:670] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:09.591588  2688 tablet_copy_client.cc:538] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:09.605813  2688 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:09.631574  2688 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:09.632634  2688 tablet_bootstrap.cc:492] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:09.633467  2688 ts_tablet_manager.cc:1403] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.009s
I20260709 13:01:09.636090  2688 raft_consensus.cc:359] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.636740  2688 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:09.637380  2688 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:09.640666  2688 ts_tablet_manager.cc:1434] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.002s
I20260709 13:01:09.644057  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-f5a2ef920c894818b430473b4809d61d received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:09.644613  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-f5a2ef920c894818b430473b4809d61d on tablet f5a2ef920c894818b430473b4809d61d with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:09.742192  2314 raft_consensus.cc:1217] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:09.782682  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08002c6c80 after lost signal to thread 434
W20260709 13:01:09.784093  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800268a80 after lost signal to thread 1525
W20260709 13:01:09.785589  2339 debug-util.cc:398] Leaking SignalData structure 0x7b080030ba40 after lost signal to thread 1617
W20260709 13:01:09.786890  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08000a9d60 after lost signal to thread 1692
W20260709 13:01:09.788213  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08002e1460 after lost signal to thread 1766
W20260709 13:01:09.789552  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800309ec0 after lost signal to thread 2342
I20260709 13:01:09.837158  2314 raft_consensus.cc:1217] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:09.922289  2688 ts_tablet_manager.cc:933] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:09.922289  2314 raft_consensus.cc:1217] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:09.932221  2688 tablet_copy_client.cc:323] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:09.935369  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 92fb459de10a4e038d6cbee7b88dc3ba from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:09.937006  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 92fb459de10a4e038d6cbee7b88dc3ba from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-92fb459de10a4e038d6cbee7b88dc3ba
I20260709 13:01:10.006531  1752 tablet_copy_source_session.cc:215] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 13:01:10.189841  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 244942us
I20260709 13:01:10.306527  2688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92fb459de10a4e038d6cbee7b88dc3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:10.351699  2688 tablet_copy_client.cc:806] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:10.356196  2688 tablet_copy_client.cc:670] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:10.371165  2717 raft_consensus.cc:1064] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:10.374809  2669 raft_consensus.cc:1064] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:10.383349  2669 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:10.383558  2717 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:10.424850  2688 tablet_copy_client.cc:538] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:10.440213  2318 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.442538  2667 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:10.450587  1585 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.454972  1585 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.453141  2669 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 13:01:10.461064  2718 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:10.461870  1661 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.463922  2312 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.466254  2669 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:10.474896  2669 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:10.479910  1741 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.481989  2718 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:10.487743  2686 raft_consensus.cc:1064] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:10.492585  2688 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
W20260709 13:01:10.495551  2721 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 13:01:10.499447  2686 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:10.532963  2669 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.541597  2717 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.545902  1592 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.555110  1667 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.574584  2318 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.577382  1665 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.579205  1739 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.580593  2695 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:10.582396  2611 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:10.583598  1592 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:10.589167  2314 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.611984  2688 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:10.617108  2687 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:10.622758  1592 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.623979  1491 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.636286  2313 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.674861  2688 tablet_bootstrap.cc:492] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:10.675778  2688 ts_tablet_manager.cc:1403] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.183s	user 0.026s	sys 0.002s
W20260709 13:01:10.686508  2731 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:01:10.699774  2688 raft_consensus.cc:359] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:10.700850  2688 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:10.701527  2688 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:10.705920  2688 ts_tablet_manager.cc:1434] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
W20260709 13:01:10.706408  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52326.8us
I20260709 13:01:10.716311  1667 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.727290  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-92fb459de10a4e038d6cbee7b88dc3ba received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:10.727952  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-92fb459de10a4e038d6cbee7b88dc3ba on tablet 92fb459de10a4e038d6cbee7b88dc3ba with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:10.708051  2716 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.747326  1592 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.850919  2313 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:10.894424  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88340.9us
I20260709 13:01:10.950877  1667 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:10.954679  1490 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.969951  1490 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:10.976290  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:10.981233  2737 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:10.990021  2315 raft_consensus.cc:1217] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:10.992897  2313 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.001705  2737 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.012171  1739 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:11.012758  1736 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:11.024466  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08000647c0 after lost signal to thread 434
W20260709 13:01:11.025702  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800064ea0 after lost signal to thread 1525
W20260709 13:01:11.029179  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08002be460 after lost signal to thread 1617
W20260709 13:01:11.030524  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800317640 after lost signal to thread 1692
W20260709 13:01:11.031919  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800323520 after lost signal to thread 1766
W20260709 13:01:11.033795  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08003402e0 after lost signal to thread 2342
I20260709 13:01:11.035409  2315 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.037920  2611 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:11.044166  1667 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.045822  1665 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.047492  2313 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:11.046396  2717 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.049832  2715 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.062331  2713 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.075274  2695 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.049155  2314 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.082521  1665 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.054594  2716 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:11.109802  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2dffe36aa52848e8962262b0e01bc346 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.117724  2687 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.125238  1667 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.051127  1592 raft_consensus.cc:2955] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.135694  2313 raft_consensus.cc:2955] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.139122  1488 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:11.152549  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet 2dffe36aa52848e8962262b0e01bc346 with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:01:11.185478  2742 tablet_replica.cc:333] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
W20260709 13:01:11.187471  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 54375.7us
I20260709 13:01:11.187603  1488 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.214550  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d4817dbc3c2e469195aa3f5e05a1534a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.239166  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f5a2ef920c894818b430473b4809d61d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:11.247274  2314 raft_consensus.cc:2955] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.271231  1490 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:11.277486  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet f5a2ef920c894818b430473b4809d61d with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:01:11.337738  2742 raft_consensus.cc:2243] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:11.339224  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet d4817dbc3c2e469195aa3f5e05a1534a with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:01:11.342089  2742 raft_consensus.cc:2272] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:11.363899  2743 tablet_replica.cc:333] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:11.395270  2743 raft_consensus.cc:2243] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:11.402894  2743 raft_consensus.cc:2272] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:11.424605  2743 ts_tablet_manager.cc:1916] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:11.431878  2742 ts_tablet_manager.cc:1916] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:11.468420  2743 ts_tablet_manager.cc:1929] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:11.469441  2743 log.cc:1199] T f5a2ef920c894818b430473b4809d61d P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/f5a2ef920c894818b430473b4809d61d
I20260709 13:01:11.503032  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet f5a2ef920c894818b430473b4809d61d (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:11.508603  2743 tablet_replica.cc:333] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:11.518585  2743 raft_consensus.cc:2243] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:11.525112  2742 ts_tablet_manager.cc:1929] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:11.527853  2742 log.cc:1199] T 2dffe36aa52848e8962262b0e01bc346 P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/2dffe36aa52848e8962262b0e01bc346
I20260709 13:01:11.532694  2743 raft_consensus.cc:2272] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:11.548864  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet 2dffe36aa52848e8962262b0e01bc346 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:11.558451  2743 ts_tablet_manager.cc:1916] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:11.569284  2714 raft_consensus.cc:1064] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
W20260709 13:01:11.713953  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 109330us
I20260709 13:01:11.694669  2714 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:11.764760  2743 ts_tablet_manager.cc:1929] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:11.798242  2743 log.cc:1199] T d4817dbc3c2e469195aa3f5e05a1534a P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/d4817dbc3c2e469195aa3f5e05a1534a
I20260709 13:01:11.809840  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet d4817dbc3c2e469195aa3f5e05a1534a (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:11.861131  2312 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:11.865402  2747 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20260709 13:01:11.907816  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81607.6us
W20260709 13:01:11.908818  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67508.6us
I20260709 13:01:11.922590  1589 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:11.950640  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46953.7us
I20260709 13:01:11.973399  1661 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:12.019676  2714 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:12.074061  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47502us
I20260709 13:01:12.087088  2748 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:12.098644  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56969.3us
I20260709 13:01:12.105792  2313 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:12.106283  1664 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:12.108798  2669 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:12.190030  1490 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:12.188406  1589 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:12.817096  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52934.5us
I20260709 13:01:12.903301  1741 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:12.924266  2313 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:12.926867  2747 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:12.938200  1589 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:12.942305  2747 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:12.958599  2766 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:12.973487  2313 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:12.997808  1592 raft_consensus.cc:2955] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:13.007622  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 92fb459de10a4e038d6cbee7b88dc3ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:13.026990  1490 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 3 to 4, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:13.164804  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet 92fb459de10a4e038d6cbee7b88dc3ba with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:13.169606  2771 tablet_replica.cc:333] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:13.171118  2771 raft_consensus.cc:2243] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:13.174082  2771 raft_consensus.cc:2272] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:13.196584  2771 ts_tablet_manager.cc:1916] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:13.217031  2771 ts_tablet_manager.cc:1929] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:13.217634  2771 log.cc:1199] T 92fb459de10a4e038d6cbee7b88dc3ba P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/92fb459de10a4e038d6cbee7b88dc3ba
I20260709 13:01:13.233012  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet 92fb459de10a4e038d6cbee7b88dc3ba (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:13.300096   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:13.536867  1741 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:13.537915  1741 raft_consensus.cc:606] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:13.541529  1741 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "278074adc39c4830a2791129a7815557"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:13.542219  1741 raft_consensus.cc:606] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:13.549183  1739 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:13.550052  1739 raft_consensus.cc:606] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:13.610250  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:13.610987  1664 raft_consensus.cc:606] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:13.614374  1739 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:13.614976  1739 raft_consensus.cc:606] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:13.618144  1739 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:13.618770  1739 raft_consensus.cc:606] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:13.620128   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:13.622159   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:13.726096  2774 raft_consensus.cc:993] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.726704  2774 raft_consensus.cc:1081] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.731594  1592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:13.732354  1592 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.732743  1592 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.763738  1592 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.767107  2774 raft_consensus.cc:993] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:13.767800  2774 raft_consensus.cc:1081] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:13.775183  2747 raft_consensus.cc:993] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:13.778421  2695 raft_consensus.cc:993] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.779165  2695 raft_consensus.cc:1081] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.788020  1589 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "278074adc39c4830a2791129a7815557"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:13.789027  1589 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.791945  1589 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:13.796958  2312 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58316
I20260709 13:01:13.798216  2312 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.798738  2312 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.803300  2748 raft_consensus.cc:993] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.803843  2748 raft_consensus.cc:1081] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.776074  2747 raft_consensus.cc:1081] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:13.806465  1587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:13.807025  1587 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.807348  1587 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.808573  2313 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58316
I20260709 13:01:13.809218  2313 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.809592  2313 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.815558  1587 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.816569  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:13.820590  1664 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.834263  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:13.834983  1741 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:13.835490  1741 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:13.818229  1592 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:13.836470  1741 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.840322  1741 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.842778  1664 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:13.844475  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:13.834304  2313 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:13.846335  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:13.846935  1739 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:13.847254  1739 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:13.846952  2765 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:13.847949  2765 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:13.848112  1739 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.849041  2765 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.849332  1739 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.856714  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:13.845202  1587 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:13.858124  1664 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.863014  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:13.863721  2313 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:13.863770  1661 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.864952  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:13.865605  1738 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:13.865900  1738 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:13.866806  1738 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:13.868440  1738 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.876622  2312 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:13.867282  1739 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:13.886121  1741 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:13.888116  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:13.887804  1491 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:13.889016  2765 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:13.889575  2765 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:13.890801  2765 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.895814  1738 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:13.882295  1589 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.900964  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:13.904173  1589 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:13.906024  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:13.906618  1738 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:13.906870  1738 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:13.907394  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:13.907997  1667 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:13.907941  1738 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.903208  1661 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:13.909785  1664 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:13.911484  1738 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:13.924240  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:13.926002  2312 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:13.929121  1589 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.938791  1667 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:13.940130  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:13.944288  2776 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:13.944980  2776 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:13.946024  2776 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:13.950328  2758 raft_consensus.cc:993] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.951308  2746 raft_consensus.cc:1081] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:13.956957  1587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36806
I20260709 13:01:13.957934  1587 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:13.958952  1587 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:13.958199  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:13.975507  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74766.6us
I20260709 13:01:13.983816  1589 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:13.988637  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:13.988886  1587 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:14.001951  1587 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:14.003345  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:14.005316  1738 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:14.007241  2780 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:14.018337  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:14.019086  1741 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:14.019426  1741 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:14.019613  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:14.023494  1741 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:14.027523  1741 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:14.028432  1738 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:14.034507  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:14.035622  2776 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:14.036155  2776 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:14.037278  2776 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:14.039933  1738 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:14.041566  1741 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:14.044813  1664 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:14.045666  1664 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:14.067641  2783 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:14.078318  1664 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:14.086617  2780 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:14.095534  2780 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:14.083709  2746 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:01:14.105989  2783 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:14.130980  2783 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:14.164157  1664 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:14.168434  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44093.4us
I20260709 13:01:14.179801  1664 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:14.180799  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 75608.7us
I20260709 13:01:14.231910  1738 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:14.237399  2792 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.237461  1491 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:14.240378  1491 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:14.299746  1488 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:14.302042  1488 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 2 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:14.303797  1488 catalog_manager.cc:5697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:14.350574  1667 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:14.354599  1661 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:14.366281  1661 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:14.383100  2792 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.383518  2755 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.472745  1738 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:14.472745  1741 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:14.475579  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51653.2us
I20260709 13:01:14.491009  1741 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:14.493228  1587 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 13:01:14.493469  1632 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dc3659523cf47fa8918889f5129c8ba. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:14.494517  2793 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.503933  2739 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 13:01:14.527477  2785 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:14.543020  2806 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:01:14.542919  1592 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:01:14.569478  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 90996.1us
I20260709 13:01:14.559909  2805 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 13:01:14.602888  1664 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:14.602893  1661 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:14.603662  1738 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:14.602746  1739 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:14.623123  2802 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.630139  1741 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:14.628793  1491 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:14.642833   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:14.650413  1739 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:14.636737  2783 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 13:01:14.644421  2805 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260709 13:01:14.724725  2783 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:14.729269  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73610.4us
W20260709 13:01:14.757349  1706 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7f9cc5e1ffa141e295028cdd523da631. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:14.731323  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54895.7us
W20260709 13:01:14.802907  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 144039us
W20260709 13:01:14.806078  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 44785.5us
W20260709 13:01:14.862268  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51468.7us
I20260709 13:01:14.870609  1741 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:14.901748  2806 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:14.934561  1587 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:14.954931   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:14.965943  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 63255.7us
W20260709 13:01:14.967335  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 162959us
I20260709 13:01:14.978479  2783 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:14.981771  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75123.7us
I20260709 13:01:14.989478  1663 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:15.004230  2747 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:15.018401  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49834.5us
I20260709 13:01:15.032812  1591 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:15.081715  1665 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:15.084262  2783 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:15.111093  2791 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:15.236698  1662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0444300de6924432844fa6519c92b570"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:15.237684  1662 raft_consensus.cc:606] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:15.270176  1738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:15.271008  1738 raft_consensus.cc:606] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:15.304188  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44607.6us
I20260709 13:01:15.358085  1738 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:15.359972  1738 raft_consensus.cc:606] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:15.378925  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71999.7us
I20260709 13:01:15.394184  2835 ts_tablet_manager.cc:933] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:15.405448  1736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:15.406960  1663 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:15.403589  2791 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:15.426950  2790 raft_consensus.cc:993] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:15.464275  2695 raft_consensus.cc:1081] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:15.529795  2314 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58316
I20260709 13:01:15.530522  2314 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:15.530918  2314 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:15.590181  2838 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.000s	sys 0.003s
W20260709 13:01:15.647702  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 72397.8us
I20260709 13:01:15.668252  2314 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:15.705559  2314 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:15.709908  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46938us
I20260709 13:01:15.684813  2791 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:42018 (request call id 10) took 1067 ms. Trace:
I20260709 13:01:15.673890  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:15.739631  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 207494us
I20260709 13:01:15.755277  1586 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:15.845186  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 273066us
I20260709 13:01:15.724047  2791 rpcz_store.cc:276] 0709 13:01:14.598983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:14.599178 (+   195us) service_pool.cc:224] Handling call
0709 13:01:14.620886 (+ 21708us) log.cc:844] Serialized 506 byte log entry
0709 13:01:15.665865 (+1044979us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7750,"raft.queue_time_us":29503,"raft.run_cpu_time_us":2590,"raft.run_wall_time_us":15222,"thread_start_us":215331,"threads_started":2,"wal-append.queue_time_us":124852}
I20260709 13:01:15.864995  2312 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:15.865355  1736 raft_consensus.cc:606] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:15.874382  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54995.3us
I20260709 13:01:15.880036  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:15.889321  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76777.2us
W20260709 13:01:15.877458  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 75945.7us
I20260709 13:01:15.876590  2791 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20260709 13:01:15.940402  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49927.1us
I20260709 13:01:15.958480  1662 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:15.961768  2851 ts_tablet_manager.cc:933] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:15.945101  2783 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.067s	sys 0.093s
I20260709 13:01:15.968782  2835 tablet_copy_client.cc:323] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
W20260709 13:01:15.972169  2783 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.067s	sys 0.095s
W20260709 13:01:15.992262  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110095us
I20260709 13:01:16.007014  1741 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:16.007555  1741 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:16.008424  1741 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:16.027016  1586 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:16.028095  2848 raft_consensus.cc:993] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.028865  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:16.028898  2848 raft_consensus.cc:1081] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.030596  2853 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:16.031107  2853 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:16.032073  2853 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:16.037973  1587 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:16.038641  1587 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:16.039002  1587 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.018008  1592 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:16.058821  1587 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.060701  1741 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.010181  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 7dc3659523cf47fa8918889f5129c8ba from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:16.067886  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 7dc3659523cf47fa8918889f5129c8ba from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-7dc3659523cf47fa8918889f5129c8ba
I20260709 13:01:16.069855  2851 tablet_copy_client.cc:323] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:16.089145  1741 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
W20260709 13:01:16.092259  2823 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 13:01:16.098356  1587 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:16.119683  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:16.120419  1742 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:16.120712  1742 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:16.121587  1742 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.123143  1742 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:16.117132  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106098us
I20260709 13:01:16.141634  2814 raft_consensus.cc:993] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:16.142513  2814 raft_consensus.cc:1081] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:16.172044  1677 tablet_copy_source_session.cc:215] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:16.170877  1742 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:16.187780  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:16.192665  1736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0444300de6924432844fa6519c92b570"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55272
I20260709 13:01:16.193435  1736 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:16.193986  1736 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.198289  2803 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:16.198863  2803 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:16.197399  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:16.201077  1662 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.199826  2803 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.205149  1491 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:16.214354  1662 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:16.216693  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 7f9cc5e1ffa141e295028cdd523da631 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:16.218276  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 7f9cc5e1ffa141e295028cdd523da631 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-7f9cc5e1ffa141e295028cdd523da631
I20260709 13:01:16.227530  1666 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:16.230444  2835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dc3659523cf47fa8918889f5129c8ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:16.230572  1733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
W20260709 13:01:16.231065  1632 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 110eba2eb7b34ea1b70ed63aca2feda8. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:16.231339  1733 raft_consensus.cc:606] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:16.231508  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 43536us
I20260709 13:01:16.233428  1741 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:16.231762  1736 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.239022  1591 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:16.237481  2830 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:01:16.250029  1752 tablet_copy_source_session.cc:215] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:16.249897  2830 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:16.254662  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:16.262472  1736 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:16.263600  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:16.266703  1663 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:16.267118  1663 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:16.269137  1587 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.272303  2861 raft_consensus.cc:993] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.280836  1663 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.282379  1663 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.284837  2851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f9cc5e1ffa141e295028cdd523da631. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:16.273671  2848 raft_consensus.cc:1081] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.301209  1590 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:16.303072  1663 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:16.304454  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:16.313144  1590 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:16.332127  1590 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:16.331491  2848 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:16.347127  2848 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:16.349968  2848 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:16.373732  1587 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:16.377159  2835 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58302 (request call id 258) took 1318 ms. Trace:
I20260709 13:01:16.424288  2821 raft_consensus.cc:993] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.424943  2821 raft_consensus.cc:1081] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:16.438454  2866 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:16.445428  1739 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:16.383638  2835 rpcz_store.cc:276] 0709 13:01:15.059774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:15.060004 (+   230us) service_pool.cc:224] Handling call
0709 13:01:15.454522 (+394518us) ts_tablet_manager.cc:934] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
0709 13:01:16.222406 (+767884us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:01:16.319415 (+ 97009us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:16.320743 (+  1328us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:01:16.375873 (+ 55130us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:01:16.376588 (+   715us) tablet_replica.cc:206] Creating consensus instance
0709 13:01:16.376970 (+   382us) ts_tablet_manager.cc:989] Replica registered.
0709 13:01:16.377038 (+    68us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":114155,"tablet-copy.queue_time_us":209380,"thread_start_us":478630,"threads_started":2}
I20260709 13:01:16.432287  1590 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:16.489518  2835 tablet_copy_client.cc:806] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:16.605086  1489 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:16.603113  2835 tablet_copy_client.cc:670] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:16.647109  1586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:16.707358  2835 tablet_copy_client.cc:538] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:01:16.730662  2873 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 13:01:16.724545  1586 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:16.775244  1586 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:16.789428  1590 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:16.810235  2874 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.162s	user 0.002s	sys 0.005s
W20260709 13:01:16.831732  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55091.7us
I20260709 13:01:16.843910  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:16.849647  1666 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:16.862095  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 149230us
W20260709 13:01:16.895460  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49678.8us
I20260709 13:01:16.906961  1589 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:16.908936  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45253us
I20260709 13:01:16.915700  2819 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:16.938218  2876 ts_tablet_manager.cc:933] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:16.943923  1666 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:16.950366  1664 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:16.955430  2874 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:01:16.959895  1588 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:16.962420  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107058us
W20260709 13:01:16.969592  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84152.6us
I20260709 13:01:16.986644  1585 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fbe322e4447d486680550d1490266b17"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:16.987434  1585 raft_consensus.cc:606] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:16.998816  1539 auto_rebalancer.cc:629] Failed to schedule move for tablet 0444300de6924432844fa6519c92b570: Illegal state: Replica ff44a4643e01482cb6b6a79b099fa766 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:01:17.006083  1666 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:17.008368   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:17.010903   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:17.011296  2835 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:17.015491  1734 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:17.016793  1587 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:17.017164  2853 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:01:17.019270  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47771.9us
I20260709 13:01:17.026161  1586 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:17.028141  1588 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:16.961481  2875 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.268s	user 0.002s	sys 0.000s
I20260709 13:01:17.043606  1593 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:17.046452  2846 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:01:17.050282  2856 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:17.051502  1666 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:17.051990  1556 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:17.073930  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:17.074895  1664 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:17.060580  2802 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:17.076941  2802 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:17.076154  2859 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:17.077894  2859 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:17.078478  2841 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 13:01:17.079244  2802 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:17.088336  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:17.061407  2851 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58316 (request call id 296) took 1642 ms. Trace:
I20260709 13:01:17.100450  2851 rpcz_store.cc:276] 0709 13:01:15.419072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:15.450228 (+ 31156us) service_pool.cc:224] Handling call
0709 13:01:15.962257 (+512029us) ts_tablet_manager.cc:934] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
0709 13:01:16.273294 (+311037us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:01:16.791270 (+517976us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:16.791576 (+   306us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:01:16.989060 (+197484us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:01:16.989433 (+   373us) tablet_replica.cc:206] Creating consensus instance
0709 13:01:17.059651 (+ 70218us) ts_tablet_manager.cc:989] Replica registered.
0709 13:01:17.061138 (+  1487us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":57951,"tablet-copy.queue_time_us":358123,"thread_start_us":341970,"threads_started":2}
I20260709 13:01:17.111274  2851 tablet_copy_client.cc:806] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:17.112058  2851 tablet_copy_client.cc:670] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:17.096714  1586 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:17.094210  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:17.143538  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47520us
I20260709 13:01:17.145186  1742 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:17.106030  2859 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:17.159336  2859 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:17.172863  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55009.7us
I20260709 13:01:17.154465  2876 tablet_copy_client.cc:323] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:01:17.201731  1491 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 226) took 1027 ms. Trace:
I20260709 13:01:17.206404  1662 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:17.196537  1490 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:17.223327  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:17.204802  1491 rpcz_store.cc:276] 0709 13:01:16.174604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.174747 (+   143us) service_pool.cc:224] Handling call
0709 13:01:17.201660 (+1026913us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:16.224973 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:16.225609 (+   636us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:16.225649 (+    40us) write_op.cc:435] Acquired schema lock
0709 13:01:16.225667 (+    18us) tablet.cc:662] Decoding operations
0709 13:01:16.226018 (+   351us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:16.226092 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:16.226127 (+    35us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:16.226307 (+   180us) tablet.cc:701] Row locks acquired
0709 13:01:16.226324 (+    17us) write_op.cc:260] PREPARE: finished
0709 13:01:16.226462 (+   138us) write_op.cc:270] Start()
0709 13:01:16.227720 (+  1258us) write_op.cc:276] Timestamp: P: 1783602076226418 usec, L: 0
0709 13:01:16.228415 (+   695us) op_driver.cc:348] REPLICATION: starting
0709 13:01:16.384220 (+155805us) log.cc:844] Serialized 598 byte log entry
0709 13:01:17.022524 (+638304us) op_driver.cc:464] REPLICATION: finished
0709 13:01:17.124958 (+102434us) write_op.cc:301] APPLY: starting
0709 13:01:17.125074 (+   116us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:17.125209 (+   135us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:17.127525 (+  2316us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:17.127832 (+   307us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:17.128062 (+   230us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:17.128137 (+    75us) write_op.cc:312] APPLY: finished
0709 13:01:17.137844 (+  9707us) log.cc:844] Serialized 22 byte log entry
0709 13:01:17.138110 (+   266us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.138222 (+   112us) write_op.cc:454] Released schema lock
0709 13:01:17.138291 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:17.138806 (+   515us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.138855 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7850,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":102121,"apply.run_cpu_time_us":2695,"apply.run_wall_time_us":13834,"num_ops":1,"prepare.queue_time_us":9369,"prepare.run_cpu_time_us":19730,"prepare.run_wall_time_us":201814,"raft.queue_time_us":325202,"raft.run_cpu_time_us":12370,"raft.run_wall_time_us":133575,"replication_time_us":793615,"thread_start_us":152262,"threads_started":4,"wal-append.queue_time_us":66205}]]}
I20260709 13:01:17.263259  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:17.198987  2790 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
W20260709 13:01:17.307372  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61068.6us
I20260709 13:01:17.222374  2856 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20260709 13:01:17.306866  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59161.1us
I20260709 13:01:17.324944  2843 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:17.321691  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 110eba2eb7b34ea1b70ed63aca2feda8 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:17.326941  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 110eba2eb7b34ea1b70ed63aca2feda8 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-110eba2eb7b34ea1b70ed63aca2feda8
I20260709 13:01:17.340343  1677 tablet_copy_source_session.cc:215] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:17.316385  2851 tablet_copy_client.cc:538] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:17.368978  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:17.416146  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 70510.3us
W20260709 13:01:17.459370  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 114193us
I20260709 13:01:17.446130  2843 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:17.498502  1735 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:17.499081  1735 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:17.523033  1735 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:17.525695  1735 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:17.500744  2895 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20260709 13:01:17.521440  1490 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 99) took 1345 ms. Trace:
I20260709 13:01:17.517189  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 206) took 1089 ms. Trace:
W20260709 13:01:17.527027  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50447.3us
I20260709 13:01:17.531049  1489 rpcz_store.cc:276] 0709 13:01:16.427522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.427674 (+   152us) service_pool.cc:224] Handling call
0709 13:01:17.517064 (+1089390us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:16.654240 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:16.654950 (+   710us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:16.654987 (+    37us) write_op.cc:435] Acquired schema lock
0709 13:01:16.655008 (+    21us) tablet.cc:662] Decoding operations
0709 13:01:16.655383 (+   375us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:16.655442 (+    59us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:16.655476 (+    34us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:16.655675 (+   199us) tablet.cc:701] Row locks acquired
0709 13:01:16.655698 (+    23us) write_op.cc:260] PREPARE: finished
0709 13:01:16.655865 (+   167us) write_op.cc:270] Start()
0709 13:01:16.656003 (+   138us) write_op.cc:276] Timestamp: P: 1783602076655802 usec, L: 0
0709 13:01:16.656029 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:01:16.656785 (+   756us) log.cc:844] Serialized 540 byte log entry
0709 13:01:17.148145 (+491360us) op_driver.cc:464] REPLICATION: finished
0709 13:01:17.162599 (+ 14454us) write_op.cc:301] APPLY: starting
0709 13:01:17.168966 (+  6367us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:17.220715 (+ 51749us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:17.237598 (+ 16883us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:17.289254 (+ 51656us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:17.310475 (+ 21221us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:17.314847 (+  4372us) write_op.cc:312] APPLY: finished
0709 13:01:17.348091 (+ 33244us) log.cc:844] Serialized 22 byte log entry
0709 13:01:17.368031 (+ 19940us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.373811 (+  5780us) write_op.cc:454] Released schema lock
0709 13:01:17.376239 (+  2428us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:17.408242 (+ 32003us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.411212 (+  2970us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":2857,"apply.run_cpu_time_us":3608,"apply.run_wall_time_us":240663,"num_ops":1,"prepare.queue_time_us":3375,"prepare.run_cpu_time_us":2859,"prepare.run_wall_time_us":2858,"replication_time_us":470937}]]}
I20260709 13:01:17.594290  2876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 110eba2eb7b34ea1b70ed63aca2feda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:17.530987  1490 rpcz_store.cc:276] 0709 13:01:16.176113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.177607 (+  1494us) service_pool.cc:224] Handling call
0709 13:01:17.521347 (+1343740us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:17.369062 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:17.411368 (+ 42306us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:17.412878 (+  1510us) write_op.cc:435] Acquired schema lock
0709 13:01:17.414304 (+  1426us) tablet.cc:662] Decoding operations
0709 13:01:17.455164 (+ 40860us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:17.461636 (+  6472us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:17.463350 (+  1714us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:17.503914 (+ 40564us) tablet.cc:701] Row locks acquired
0709 13:01:17.503942 (+    28us) write_op.cc:260] PREPARE: finished
0709 13:01:17.504127 (+   185us) write_op.cc:270] Start()
0709 13:01:17.504260 (+   133us) write_op.cc:276] Timestamp: P: 1783602077504080 usec, L: 0
0709 13:01:17.504287 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 13:01:17.505178 (+   891us) log.cc:844] Serialized 536 byte log entry
0709 13:01:17.507089 (+  1911us) op_driver.cc:464] REPLICATION: finished
0709 13:01:17.509576 (+  2487us) write_op.cc:301] APPLY: starting
0709 13:01:17.509665 (+    89us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:17.509837 (+   172us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:17.509971 (+   134us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:17.510214 (+   243us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:17.510389 (+   175us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:17.510439 (+    50us) write_op.cc:312] APPLY: finished
0709 13:01:17.510634 (+   195us) log.cc:844] Serialized 23 byte log entry
0709 13:01:17.510807 (+   173us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.510895 (+    88us) write_op.cc:454] Released schema lock
0709 13:01:17.510933 (+    38us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:17.511240 (+   307us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:17.511273 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19062,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":2237,"apply.run_cpu_time_us":1647,"apply.run_wall_time_us":1647,"num_ops":1,"prepare.queue_time_us":134001,"prepare.run_cpu_time_us":5391,"prepare.run_wall_time_us":136572,"replication_time_us":2663,"spinlock_wait_cycles":26368,"thread_start_us":115308,"threads_started":1}]]}
W20260709 13:01:17.629710  1557 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:01:17.637687  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70054.1us
W20260709 13:01:17.658152  2899 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:01:17.678308  2835 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:17.590592  2843 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:17.684454  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:17.165187  2858 consensus_queue.cc:1175] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "316f3bf18ed94e32b238130a6bfedf1a" responder_term: 2 status { last_received { term: 1 index: 2 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:01:17.691679  2835 tablet_bootstrap.cc:492] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:17.692953  2835 ts_tablet_manager.cc:1403] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.682s	user 0.021s	sys 0.021s
I20260709 13:01:17.734273  2803 raft_consensus.cc:993] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:17.737799  1735 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:17.741557  2820 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
W20260709 13:01:17.783140  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 56367.2us
I20260709 13:01:17.786470  1666 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:17.825778  1491 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:17.864805  2902 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:01:17.926906  2897 raft_consensus.cc:1081] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:17.902667  1586 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:17.970438  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:17.987782  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50292.1us
I20260709 13:01:17.998544  2851 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:18.022162   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:18.002899  1733 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:18.062909  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62401.6us
I20260709 13:01:17.992235  2835 raft_consensus.cc:359] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:18.077311  2835 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:18.078608  2835 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:18.079916  2917 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:01:18.118772  2313 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fbe322e4447d486680550d1490266b17"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58332
I20260709 13:01:18.119634  2313 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:18.120010  2313 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:18.109445  2788 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:18.143280  2835 ts_tablet_manager.cc:1434] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.450s	user 0.002s	sys 0.013s
I20260709 13:01:18.145800  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-7dc3659523cf47fa8918889f5129c8ba received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:18.146380  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-7dc3659523cf47fa8918889f5129c8ba on tablet 7dc3659523cf47fa8918889f5129c8ba with peer e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:18.122537  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 241056us
I20260709 13:01:18.124276  2788 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:18.159260  2788 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:18.165148  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:18.165983  2912 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.161s	user 0.001s	sys 0.007s
I20260709 13:01:18.170150  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 798) took 1088 ms. Trace:
I20260709 13:01:18.171224  1741 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:01:18.170668  1733 rpcz_store.cc:276] 0709 13:01:17.081685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:17.084842 (+  3157us) service_pool.cc:224] Handling call
0709 13:01:17.368726 (+283884us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:17.815186 (+446460us) spinlock_profiling.cc:243] Waited 78.9 ms on lock 0x7b54001665a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0e18 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:18.169918 (+354732us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54003f9dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:18.170056 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":749195520}
I20260709 13:01:18.195029  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:18.201351  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79592.8us
I20260709 13:01:18.205260  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:18.228267  2313 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:18.247511  2313 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:18.249711  2849 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:18.279021  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53089.1us
I20260709 13:01:18.294028  2312 raft_consensus.cc:1217] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:18.302462  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:18.304791  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:18.305691  1589 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:18.306291  1589 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:18.294065  2901 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 13:01:18.309873  1665 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:18.324999  1736 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:18.338668  2876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58302 (request call id 278) took 1648 ms. Trace:
I20260709 13:01:18.339354  2876 rpcz_store.cc:276] 0709 13:01:16.690101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:16.690656 (+   555us) service_pool.cc:224] Handling call
0709 13:01:16.938617 (+247961us) ts_tablet_manager.cc:934] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
0709 13:01:17.538780 (+600163us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:01:17.941727 (+402947us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:17.948934 (+  7207us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:01:18.336092 (+387158us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:01:18.336572 (+   480us) tablet_replica.cc:206] Creating consensus instance
0709 13:01:18.338513 (+  1941us) ts_tablet_manager.cc:989] Replica registered.
0709 13:01:18.338589 (+    76us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":90550,"tablet-copy.queue_time_us":244757,"thread_start_us":283074,"threads_started":2}
I20260709 13:01:18.342568  2876 tablet_copy_client.cc:806] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:18.343268  2876 tablet_copy_client.cc:670] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:18.351634  1589 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:18.365401  1589 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:18.367658  2792 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:18.375008  2876 tablet_copy_client.cc:538] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:18.379815  1665 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:18.382390  1667 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:18.391500  1589 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:18.393553  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:18.396059  2819 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:18.417620  1665 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:18.423775  2819 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:18.487710  2819 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:18.524748  1490 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:01:18.531184  1490 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:18.535488  1490 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 2 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:18.558526  2792 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:18.603874  2876 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:18.623399   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:18.659984  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72988.4us
W20260709 13:01:18.666186  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78806us
W20260709 13:01:18.815943  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 111429us
I20260709 13:01:18.840941  1489 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:18.872785  2849 raft_consensus.cc:1064] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:18.884992  2849 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:18.899194  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53448us
I20260709 13:01:18.909929  1737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:18.910760  1737 raft_consensus.cc:606] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:18.914930  2851 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.057s	user 0.002s	sys 0.000s
I20260709 13:01:18.915725  2851 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:18.916781  2851 tablet_bootstrap.cc:492] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:19.005739  1589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:19.006482  1589 raft_consensus.cc:606] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:19.001668  2851 ts_tablet_manager.cc:1403] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 1.003s	user 0.033s	sys 0.019s
I20260709 13:01:18.986092  2883 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:19.027456  2883 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:19.041421  1589 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:19.049216  2312 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:19.058028  1737 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:19.061690  2879 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:19.063042  2910 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:19.066250  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59552.9us
I20260709 13:01:19.104267  2883 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:19.111276  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:19.126129  1588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:19.126909  1588 raft_consensus.cc:606] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:19.121681  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:19.159026  1665 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:19.161022  2879 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:19.085587  2758 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260709 13:01:19.139071  2851 raft_consensus.cc:359] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:19.179291  2851 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:19.180147  2851 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:19.181143  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:19.182669  2849 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:19.257323  1667 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:19.295066  1667 raft_consensus.cc:606] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:19.228231  2851 ts_tablet_manager.cc:1434] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.217s	user 0.006s	sys 0.004s
I20260709 13:01:19.310292  1593 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260709 13:01:19.401396  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 98359.8us
W20260709 13:01:19.426939  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57540.8us
W20260709 13:01:19.416987  2940 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.171s	user 0.002s	sys 0.000s
I20260709 13:01:19.433919  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-7f9cc5e1ffa141e295028cdd523da631 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:19.435539  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-7f9cc5e1ffa141e295028cdd523da631 on tablet 7f9cc5e1ffa141e295028cdd523da631 with peer e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:19.436546  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 84216.1us
I20260709 13:01:19.476488  1589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:19.478106  1589 raft_consensus.cc:606] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:19.533865  1589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fbe322e4447d486680550d1490266b17"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:19.534827  1589 raft_consensus.cc:606] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:19.535311  1589 raft_consensus.cc:614] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:19.538189   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:01:19.542011  2861 raft_consensus.cc:993] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:19.542730  2861 raft_consensus.cc:1081] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:19.593425  2318 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Advancing to term 2
I20260709 13:01:19.610977  2944 raft_consensus.cc:993] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:19.626423   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:19.630254  2896 raft_consensus.cc:1081] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:19.633981  2895 raft_consensus.cc:993] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:19.636334  2895 raft_consensus.cc:1081] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:19.646565  2317 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58332
I20260709 13:01:19.647430  2317 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:19.647847  2317 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:19.674099  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 265312us
W20260709 13:01:19.762012  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 108736us
W20260709 13:01:19.652842  2876 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.051s	user 0.003s	sys 0.000s
I20260709 13:01:19.775731  2897 raft_consensus.cc:993] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
W20260709 13:01:19.777729  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102001us
I20260709 13:01:19.818403  1593 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
W20260709 13:01:19.786963  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44092.5us
I20260709 13:01:19.792285  2788 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:19.819707  1593 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:19.824342  1593 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:19.842607  1737 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55270
I20260709 13:01:19.849640  1737 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:19.850252  1737 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:19.854969  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90592.1us
I20260709 13:01:19.824839  2894 raft_consensus.cc:1081] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:19.858558  2317 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:19.924506  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62316.4us
I20260709 13:01:19.896134  2876 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:19.943825  2317 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:19.965626  1593 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:19.970520  1590 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:19.990875  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:19.991889  1589 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:19.992326  1589 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:19.999706  1589 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.005443  1589 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.014734  2876 tablet_bootstrap.cc:492] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
W20260709 13:01:20.017902  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91446.4us
W20260709 13:01:20.036516  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102462us
I20260709 13:01:20.032892  2876 ts_tablet_manager.cc:1403] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 1.430s	user 0.044s	sys 0.010s
I20260709 13:01:20.034787  2311 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58332
I20260709 13:01:20.057443  2311 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:20.058493  2311 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.053308  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:20.061666  1734 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.054164  2313 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:20.068095  1589 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:20.074837  2876 raft_consensus.cc:359] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:20.075913  2876 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:20.076628  2876 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:20.098356  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:20.115721  2879 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:20.116549  2879 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:20.118844  2879 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.125435  1593 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:20.134055  2910 raft_consensus.cc:993] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:20.141899  2910 raft_consensus.cc:1081] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:20.144712  2317 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58302
I20260709 13:01:20.145799  2317 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:20.146409  2317 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.157167  2826 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:20.185969  2317 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.188241  2317 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:20.195281  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:20.196221  1593 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.195489  1734 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:20.199048  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:20.199782  1667 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:20.200070  1667 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:20.184482  1737 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:20.215992  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 233426us
I20260709 13:01:20.218153  1667 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:20.272611  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45575.5us
W20260709 13:01:20.276132  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 329560us
I20260709 13:01:20.277491  1737 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:20.285739  1667 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.303774  1593 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:20.302911  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:20.306517  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:20.314025  1663 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.308321  1666 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:20.322153  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:20.313524  2856 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 13:01:20.343964  2856 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:20.344430  2856 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:20.345057  2853 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:20.345734  2853 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:20.346735  2853 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.362735  2876 ts_tablet_manager.cc:1434] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.312s	user 0.010s	sys 0.000s
I20260709 13:01:20.376263  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-110eba2eb7b34ea1b70ed63aca2feda8 received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:20.376801  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-110eba2eb7b34ea1b70ed63aca2feda8 on tablet 110eba2eb7b34ea1b70ed63aca2feda8 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:20.335263  1667 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:20.395665  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:20.396862  1590 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:20.397328  1590 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:20.395499  2311 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.405431  1590 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:20.407902  1663 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:20.408106  1590 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:20.417299  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:20.418330  2311 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:20.421681  2312 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:20.406522  2962 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.105s	user 0.000s	sys 0.005s
I20260709 13:01:20.357004  2856 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:20.446653  2856 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.455255  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:20.456331  1663 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.463181  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:20.464040  1593 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:20.464469  1593 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:20.465274  1593 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.466414  1593 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:20.499084  1593 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
W20260709 13:01:20.501159  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 171289us
I20260709 13:01:20.503582  1663 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:20.504375  1666 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:20.505072  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:20.508951  1666 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:20.513767  1590 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:20.519049  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:20.530052  2318 raft_consensus.cc:1217] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:20.540494  1737 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:20.550812  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:20.549773  2950 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:20.573732  2950 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:20.573916  2952 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:20.574652  2952 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:20.535645  2819 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:20.574954  2950 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:20.575601  2952 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:20.575618  2819 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:20.578604  2819 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.594439  1593 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:20.665625   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:20.672019  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47411.4us
W20260709 13:01:20.697069  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 133448us
I20260709 13:01:20.724915  1733 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:20.798978  2980 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.084s	user 0.004s	sys 0.000s
I20260709 13:01:20.822688  2943 raft_consensus.cc:1064] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:20.828608  2943 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:20.865027  1491 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:20.889179  1734 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:20.892719  2986 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:01:20.920692  1735 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:20.921198  1586 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:20.925348  2948 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:20.956091  1489 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:20.984198  1664 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:20.998461  1734 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:21.000676  2827 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:21.004042  2819 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:21.008749  2311 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:01:21.012156  2802 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:21.018263  1587 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:21.022017  1590 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:21.029959  1664 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:21.039250  2853 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 13:01:21.053884  2827 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 13:01:21.078406  2853 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:21.078293  2968 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:21.091295  1593 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:21.104948  2967 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:21.139102  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 80591.8us
I20260709 13:01:21.143208  2918 raft_consensus.cc:1064] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
W20260709 13:01:21.159435  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 156411us
W20260709 13:01:21.183521  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:21.206678  1666 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:21.208603  2983 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:21.216693  1666 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:21.230607  2950 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:21.233003  1663 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:21.246205  2991 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:21.274930  2918 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:21.346953  1740 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:21.374261  1664 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:21.376636   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:21.424499  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78738.1us
W20260709 13:01:21.461364  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62085.8us
W20260709 13:01:21.460261  2972 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:01:21.470618  2984 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20260709 13:01:21.568097  2990 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:01:21.537662  2944 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
W20260709 13:01:21.669097  2997 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
W20260709 13:01:21.671200  2998 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
W20260709 13:01:21.723193  2983 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.502s	user 0.012s	sys 0.006s
I20260709 13:01:21.874802  1735 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:21.969260  1491 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 211) took 1214 ms. Trace:
I20260709 13:01:21.951295  1490 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 2 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 3 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:01:22.046437  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48895us
I20260709 13:01:22.056324  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 232) took 1419 ms. Trace:
I20260709 13:01:21.970101  1491 rpcz_store.cc:276] 0709 13:01:20.754213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.754397 (+   184us) service_pool.cc:224] Handling call
0709 13:01:21.968828 (+1214431us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:20.922193 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:20.922911 (+   718us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:20.922953 (+    42us) write_op.cc:435] Acquired schema lock
0709 13:01:20.922973 (+    20us) tablet.cc:662] Decoding operations
0709 13:01:20.923359 (+   386us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:20.924593 (+  1234us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:20.924635 (+    42us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:20.924915 (+   280us) tablet.cc:701] Row locks acquired
0709 13:01:20.924933 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:01:20.925094 (+   161us) write_op.cc:270] Start()
0709 13:01:20.925228 (+   134us) write_op.cc:276] Timestamp: P: 1783602080925049 usec, L: 0
0709 13:01:20.925710 (+   482us) op_driver.cc:348] REPLICATION: starting
0709 13:01:20.926713 (+  1003us) log.cc:844] Serialized 541 byte log entry
0709 13:01:21.192677 (+265964us) op_driver.cc:464] REPLICATION: finished
0709 13:01:21.395401 (+202724us) write_op.cc:301] APPLY: starting
0709 13:01:21.395510 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:21.395660 (+   150us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:21.395856 (+   196us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:21.396221 (+   365us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:21.396461 (+   240us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:21.396802 (+   341us) write_op.cc:312] APPLY: finished
0709 13:01:21.397165 (+   363us) log.cc:844] Serialized 23 byte log entry
0709 13:01:21.397394 (+   229us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.397478 (+    84us) write_op.cc:454] Released schema lock
0709 13:01:21.433215 (+ 35737us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:21.438950 (+  5735us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.438981 (+    31us) write_op.cc:454] Released schema lock
0709 13:01:21.716310 (+277329us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfda32a 00007fcdedf5ff14 00007fcdedf5fec3 00007fcdedf5fe38 00007fcdedf5fdf3 00007fcdedf5f0dc 00007fcdedf88e16 00007fcdedf70cbb 00007fcdedf5b39d 00007fcdedf5ed75
0709 13:01:21.721169 (+  4859us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.721213 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":54334,"reported_tablets":1,"spinlock_wait_cycles":10880,"child_traces":[["op",{"apply.queue_time_us":190411,"apply.run_cpu_time_us":2693,"apply.run_wall_time_us":43574,"num_ops":1,"prepare.queue_time_us":34671,"prepare.run_cpu_time_us":73292,"prepare.run_wall_time_us":119298,"raft.queue_time_us":611483,"raft.run_cpu_time_us":37202,"raft.run_wall_time_us":552939,"replication_time_us":252918,"spinlock_wait_cycles":4451712,"thread_start_us":344192,"threads_started":4,"wal-append.queue_time_us":114902}]]}
I20260709 13:01:22.030705  2312 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:22.056990  1489 rpcz_store.cc:276] 0709 13:01:20.637046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.637212 (+   166us) service_pool.cc:224] Handling call
0709 13:01:22.056063 (+1418851us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:21.041885 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:21.042623 (+   738us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:21.042660 (+    37us) write_op.cc:435] Acquired schema lock
0709 13:01:21.042673 (+    13us) tablet.cc:662] Decoding operations
0709 13:01:21.043023 (+   350us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:21.043064 (+    41us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:21.043083 (+    19us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:21.043311 (+   228us) tablet.cc:701] Row locks acquired
0709 13:01:21.043324 (+    13us) write_op.cc:260] PREPARE: finished
0709 13:01:21.043453 (+   129us) write_op.cc:270] Start()
0709 13:01:21.043556 (+   103us) write_op.cc:276] Timestamp: P: 1783602081043402 usec, L: 0
0709 13:01:21.043570 (+    14us) op_driver.cc:348] REPLICATION: starting
0709 13:01:21.044218 (+   648us) log.cc:844] Serialized 599 byte log entry
0709 13:01:21.718925 (+674707us) op_driver.cc:464] REPLICATION: finished
0709 13:01:21.719288 (+   363us) write_op.cc:301] APPLY: starting
0709 13:01:21.719400 (+   112us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:21.719531 (+   131us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:21.719649 (+   118us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:21.719894 (+   245us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:21.720074 (+   180us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:21.720127 (+    53us) write_op.cc:312] APPLY: finished
0709 13:01:21.720352 (+   225us) log.cc:844] Serialized 23 byte log entry
0709 13:01:21.720574 (+   222us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:21.720665 (+    91us) write_op.cc:454] Released schema lock
0709 13:01:21.720710 (+    45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":71290,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":1722,"apply.run_wall_time_us":1721,"num_ops":1,"prepare.queue_time_us":51463,"prepare.run_cpu_time_us":2750,"prepare.run_wall_time_us":2748,"replication_time_us":675182}]]}
W20260709 13:01:22.298992  2918 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.015s	sys 0.023s
W20260709 13:01:22.299521  2918 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.015s	sys 0.024s
W20260709 13:01:22.383092  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138520us
W20260709 13:01:22.412292  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202872us
W20260709 13:01:22.433369  3021 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.108s	user 0.002s	sys 0.000s
I20260709 13:01:22.434995  3019 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.436105  3019 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:22.441483  2830 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.441147  2974 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:22.450482  2974 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:22.442276  2830 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:22.441236  2963 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:01:22.507620  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89655.5us
I20260709 13:01:22.508445  2974 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:22.470840  2963 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:22.517442  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.536506  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:22.537786  2966 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.546633  2884 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.572618  2966 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:22.586217  3019 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:22.588433  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:22.596874  3034 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.158s	user 0.000s	sys 0.005s
I20260709 13:01:22.614380  1589 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:22.615393  1589 raft_consensus.cc:606] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:22.627259  2830 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:22.610409  2884 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:22.635102  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58138.9us
I20260709 13:01:22.643096  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:22.652319  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.678609  2963 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:22.690730  2966 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:22.678761  2884 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:22.703811  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.706153  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.706642  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.720342  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:22.721585  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67360.4us
I20260709 13:01:22.726302  2932 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:22.737217  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:22.740408  2937 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.763855  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:22.772614  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 108693us
W20260709 13:01:22.774617  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 97782.8us
I20260709 13:01:22.741813  2937 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:22.818259  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:22.823264  1587 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:22.824043  1587 raft_consensus.cc:606] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:22.824622  1587 raft_consensus.cc:614] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:22.825760  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:22.829108  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:22.830528  2930 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:22.831517  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 111800us
I20260709 13:01:22.836107  2964 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:22.846861  2785 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:22.850034  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:22.858939  2944 raft_consensus.cc:993] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:22.864341  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:22.871049  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67445.5us
W20260709 13:01:22.872697  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53437.1us
I20260709 13:01:22.903391  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:22.905682  2943 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:22.906327  2943 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:22.908789  2937 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:22.910485  2826 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:22.912554  2944 raft_consensus.cc:1081] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:22.912637  1733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:22.913662  1733 raft_consensus.cc:606] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:22.921705  1590 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:22.955144  1490 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 216) took 1803 ms. Trace:
I20260709 13:01:22.949759  1660 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:22.965824  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:22.984254  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:22.982709  1742 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:22.965020  1660 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:22.955670  1490 rpcz_store.cc:276] 0709 13:01:21.151376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:21.151703 (+   327us) service_pool.cc:224] Handling call
0709 13:01:22.955065 (+1803362us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:22.393839 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:22.432690 (+ 38851us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:22.432749 (+    59us) write_op.cc:435] Acquired schema lock
0709 13:01:22.432773 (+    24us) tablet.cc:662] Decoding operations
0709 13:01:22.433256 (+   483us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:22.433371 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:22.433396 (+    25us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:22.433862 (+   466us) tablet.cc:701] Row locks acquired
0709 13:01:22.433918 (+    56us) write_op.cc:260] PREPARE: finished
0709 13:01:22.434103 (+   185us) write_op.cc:270] Start()
0709 13:01:22.434256 (+   153us) write_op.cc:276] Timestamp: P: 1783602082434042 usec, L: 0
0709 13:01:22.434292 (+    36us) op_driver.cc:348] REPLICATION: starting
0709 13:01:22.435107 (+   815us) log.cc:844] Serialized 541 byte log entry
0709 13:01:22.696628 (+261521us) op_driver.cc:464] REPLICATION: finished
0709 13:01:22.909084 (+212456us) write_op.cc:301] APPLY: starting
0709 13:01:22.909285 (+   201us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:22.909401 (+   116us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:22.909580 (+   179us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:22.910006 (+   426us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:22.910674 (+   668us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:22.910892 (+   218us) write_op.cc:312] APPLY: finished
0709 13:01:22.911254 (+   362us) log.cc:844] Serialized 23 byte log entry
0709 13:01:22.911576 (+   322us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:22.911672 (+    96us) write_op.cc:454] Released schema lock
0709 13:01:22.911756 (+    84us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:22.912345 (+   589us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:22.912382 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":202628,"apply.run_cpu_time_us":3227,"apply.run_wall_time_us":3227,"num_ops":1,"prepare.queue_time_us":297490,"prepare.run_cpu_time_us":4270,"prepare.run_wall_time_us":73735,"replication_time_us":224087,"thread_start_us":263312,"threads_started":2}]]}
I20260709 13:01:23.004786  2943 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:23.006589  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:22.969959  2830 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:23.002251  2311 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:23.015102  1739 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55270
I20260709 13:01:23.037827  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:23.047289  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:23.054033  1488 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:23.056455  2884 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:23.058215  1593 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:23.059871  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:23.062377  1593 raft_consensus.cc:606] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Received request to transfer leadership to TS e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:23.070734  1593 raft_consensus.cc:614] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:23.074683  2849 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:23.057242  1488 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:23.084968  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:23.090106   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:01:23.093298   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:23.100307  1739 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:23.100790  1739 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.118170  1488 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:23.100535  3048 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:23.226706  1739 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:23.238488  1739 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:23.258030  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:23.259536  1586 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:23.260106  1586 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:23.265014  1736 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:23.290762  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:23.291546  1660 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.292409  1586 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:23.295917  2943 consensus_queue.cc:1175] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df6a18af06d543b2931837f5e6bab95f" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:01:23.298887  1586 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.301990  3047 raft_consensus.cc:993] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:23.302695  3047 raft_consensus.cc:1081] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:23.317798  1588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:23.318508  1588 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:23.318892  1588 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.322031  2961 raft_consensus.cc:993] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:23.322628  2961 raft_consensus.cc:1081] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:01:23.340164  2311 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58302
I20260709 13:01:23.344040  2311 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:23.344702  2311 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.351109  1660 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:23.334774  1588 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:23.361361  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:23.362196  1660 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.370041  1588 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:23.376832  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:23.384822  2311 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:23.394042  3042 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:23.401213  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:23.401883  1660 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:23.401214  3042 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:23.403635  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:23.404999  2943 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:23.403512  3042 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:23.405604  2943 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:23.402256  1734 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:23.407332  1734 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:23.408506  2943 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:23.409211  1734 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:23.410835  1734 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.423614  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:23.426246  1588 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.389026  1586 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:23.469133  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:23.472786  2311 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:23.491086  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:23.495821  1663 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:23.496676  1663 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:23.479826  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:23.525663  1663 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:23.532286  1663 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:23.538357  3029 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:23.545852  3029 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:23.540016  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65593.2us
I20260709 13:01:23.561149  3029 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:23.594308  1734 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:23.712997  1663 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:23.742798  1489 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:23.754051  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65240.8us
I20260709 13:01:23.814702  1488 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 109) took 2891 ms. Trace:
I20260709 13:01:23.815229  1488 rpcz_store.cc:276] 0709 13:01:20.912074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:20.912347 (+   273us) service_pool.cc:224] Handling call
0709 13:01:23.803254 (+2890907us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:23.336815 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:23.337665 (+   850us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:23.337702 (+    37us) write_op.cc:435] Acquired schema lock
0709 13:01:23.337721 (+    19us) tablet.cc:662] Decoding operations
0709 13:01:23.338336 (+   615us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:23.338447 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:23.338476 (+    29us) tablet.cc:685] Acquiring locks for 3 operations
0709 13:01:23.338824 (+   348us) tablet.cc:701] Row locks acquired
0709 13:01:23.338842 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:01:23.338991 (+   149us) write_op.cc:270] Start()
0709 13:01:23.339124 (+   133us) write_op.cc:276] Timestamp: P: 1783602083338945 usec, L: 0
0709 13:01:23.339799 (+   675us) op_driver.cc:348] REPLICATION: starting
0709 13:01:23.341877 (+  2078us) log.cc:844] Serialized 1302 byte log entry
0709 13:01:23.498611 (+156734us) op_driver.cc:464] REPLICATION: finished
0709 13:01:23.598113 (+ 99502us) write_op.cc:301] APPLY: starting
0709 13:01:23.598197 (+    84us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:23.598343 (+   146us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:23.598876 (+   533us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:23.719462 (+120586us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:23.719778 (+   316us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 13:01:23.719883 (+   105us) write_op.cc:312] APPLY: finished
0709 13:01:23.720534 (+   651us) log.cc:844] Serialized 35 byte log entry
0709 13:01:23.720858 (+   324us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:23.721011 (+   153us) write_op.cc:454] Released schema lock
0709 13:01:23.721080 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:23.790270 (+ 69190us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:23.790304 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26259,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":96577,"apply.run_cpu_time_us":3702,"apply.run_wall_time_us":192167,"num_ops":3,"prepare.queue_time_us":205250,"prepare.run_cpu_time_us":3588,"prepare.run_wall_time_us":33995,"replication_time_us":147736,"thread_start_us":226413,"threads_started":2}]]}
W20260709 13:01:23.879211  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58936.8us
I20260709 13:01:23.882151  1663 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:23.886989  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 212) took 1320 ms. Trace:
I20260709 13:01:23.897305  2971 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:23.887508  1489 rpcz_store.cc:276] 0709 13:01:22.568985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.569647 (+   662us) service_pool.cc:224] Handling call
0709 13:01:23.886914 (+1317267us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:23.748069 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:23.748783 (+   714us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:23.748820 (+    37us) write_op.cc:435] Acquired schema lock
0709 13:01:23.748838 (+    18us) tablet.cc:662] Decoding operations
0709 13:01:23.749214 (+   376us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:23.749277 (+    63us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:23.749308 (+    31us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:23.749567 (+   259us) tablet.cc:701] Row locks acquired
0709 13:01:23.749588 (+    21us) write_op.cc:260] PREPARE: finished
0709 13:01:23.749768 (+   180us) write_op.cc:270] Start()
0709 13:01:23.749904 (+   136us) write_op.cc:276] Timestamp: P: 1783602083749712 usec, L: 0
0709 13:01:23.749932 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 13:01:23.750994 (+  1062us) log.cc:844] Serialized 599 byte log entry
0709 13:01:23.754009 (+  3015us) op_driver.cc:464] REPLICATION: finished
0709 13:01:23.798434 (+ 44425us) write_op.cc:301] APPLY: starting
0709 13:01:23.798508 (+    74us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:23.799015 (+   507us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:23.799114 (+    99us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:23.799401 (+   287us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:23.799559 (+   158us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:23.799609 (+    50us) write_op.cc:312] APPLY: finished
0709 13:01:23.800175 (+   566us) log.cc:844] Serialized 23 byte log entry
0709 13:01:23.800502 (+   327us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:23.800638 (+   136us) write_op.cc:454] Released schema lock
0709 13:01:23.800683 (+    45us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":21998,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":36633,"apply.run_cpu_time_us":2353,"apply.run_wall_time_us":2622,"num_ops":1,"prepare.queue_time_us":121,"prepare.run_cpu_time_us":3388,"prepare.run_wall_time_us":5413,"replication_time_us":3866,"spinlock_wait_cycles":83840,"thread_start_us":86328,"threads_started":1}]]}
I20260709 13:01:23.916814  1586 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:23.931421  1660 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:01:23.934458  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 87696.6us
I20260709 13:01:23.935266  1488 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:23.950145  1491 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 233) took 1323 ms. Trace:
I20260709 13:01:23.952507  1491 rpcz_store.cc:276] 0709 13:01:22.627184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:22.627536 (+   352us) service_pool.cc:224] Handling call
0709 13:01:23.950060 (+1322524us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":6}
I20260709 13:01:23.956014  2903 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:01:23.961907  1660 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:23.971395  1488 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 2, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:23.993439  1736 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:23.995805  2312 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:24.007180  1489 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:24.008539  2952 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:24.012049  1588 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:24.014611  2911 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:24.023975  2943 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:24.036310  1737 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:24.068910  2311 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:24.076596  1588 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:24.051563  2903 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 13:01:24.090662  3059 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 13:01:24.090588  1586 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:24.149976   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:24.142830  2884 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 13:01:24.203522  3071 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:24.280539  1661 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 13:01:24.282495  3064 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.257s	user 0.003s	sys 0.000s
I20260709 13:01:24.296993  3023 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260709 13:01:24.326933  3072 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.002s
I20260709 13:01:24.415030  3013 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:24.395792  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53102.7us
W20260709 13:01:24.492905  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53768.6us
I20260709 13:01:24.543534  2311 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:24.549382  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 85964.6us
W20260709 13:01:24.576566  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54055.8us
W20260709 13:01:24.613431  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62551.6us
W20260709 13:01:24.625469  3066 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.402s	user 0.000s	sys 0.007s
I20260709 13:01:24.626045  2317 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:24.573985  2966 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:24.714625  1490 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:24.733119  1478 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7f9cc5e1ffa141e295028cdd523da631 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:24.806046  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7dc3659523cf47fa8918889f5129c8ba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:24.805461  1586 raft_consensus.cc:2955] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:24.807356  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48046.5us
I20260709 13:01:24.824928  1491 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:01:24.886683  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45942.4us
I20260709 13:01:25.000847  1490 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 4 to 5, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:24.856454  1662 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:25.043668  1733 raft_consensus.cc:2955] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:01:25.051813  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51281.2us
W20260709 13:01:25.082269  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77459.7us
W20260709 13:01:25.086392  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64163.1us
W20260709 13:01:25.099350  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 56532.9us
I20260709 13:01:25.136384  2311 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:25.158151  2961 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:25.188007  2961 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:25.187896  1728 tablet_service.cc:1558] Processing DeleteTablet for tablet 7dc3659523cf47fa8918889f5129c8ba with delete_type TABLET_DATA_TOMBSTONED (TS df6a18af06d543b2931837f5e6bab95f not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:55264
I20260709 13:01:25.192879  1657 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f9cc5e1ffa141e295028cdd523da631 with delete_type TABLET_DATA_TOMBSTONED (TS ff44a4643e01482cb6b6a79b099fa766 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:49478
I20260709 13:01:25.198549  2961 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:25.202735  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:25.209118  3058 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:25.216102  2878 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:25.216928  2878 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:25.202652   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:25.232789  1734 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 13:01:25.282550  2878 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:25.286180  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:25.290088  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:25.291771  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:25.294207  1632 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:25.293851  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49486.5us
I20260709 13:01:25.302980  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:25.315371  3101 tablet_replica.cc:333] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:01:25.315992  2966 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:25.316762  3101 raft_consensus.cc:2243] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:01:25.318889  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 150559us
I20260709 13:01:25.323103  3102 tablet_replica.cc:333] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:01:25.319478  3060 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 13:01:25.326105  3102 raft_consensus.cc:2243] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:25.327280  3102 raft_consensus.cc:2272] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:25.331444  3102 ts_tablet_manager.cc:1916] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:25.320565  3101 raft_consensus.cc:2272] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:25.339128  1705 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:25.340216  3101 ts_tablet_manager.cc:1916] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 13:01:25.346901  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 130958us
I20260709 13:01:25.362881  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:25.364008  1705 raft_consensus.cc:2686] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 13:01:25.383248  2932 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:25.387907  2317 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:25.432140  1475 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 110eba2eb7b34ea1b70ed63aca2feda8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:01:25.443281  1735 raft_consensus.cc:2955] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:25.461194  1488 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 3 to 4, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:25.465308  3102 ts_tablet_manager.cc:1929] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260709 13:01:25.465992  3102 log.cc:1199] T 7f9cc5e1ffa141e295028cdd523da631 P ff44a4643e01482cb6b6a79b099fa766: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-1-root/wals/7f9cc5e1ffa141e295028cdd523da631
I20260709 13:01:25.484584  1475 catalog_manager.cc:5028] TS ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): tablet 7f9cc5e1ffa141e295028cdd523da631 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:25.485783  3101 ts_tablet_manager.cc:1929] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:25.486409  3101 log.cc:1199] T 7dc3659523cf47fa8918889f5129c8ba P df6a18af06d543b2931837f5e6bab95f: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-2-root/wals/7dc3659523cf47fa8918889f5129c8ba
I20260709 13:01:25.488772  1476 catalog_manager.cc:5028] TS df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): tablet 7dc3659523cf47fa8918889f5129c8ba (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
W20260709 13:01:25.505784  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49896.2us
W20260709 13:01:25.515365  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65406.6us
I20260709 13:01:25.520440  1578 tablet_service.cc:1558] Processing DeleteTablet for tablet 110eba2eb7b34ea1b70ed63aca2feda8 with delete_type TABLET_DATA_TOMBSTONED (TS 316f3bf18ed94e32b238130a6bfedf1a not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36792
I20260709 13:01:25.547334  3105 tablet_replica.cc:333] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:01:25.598793  1736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:25.599480  1736 raft_consensus.cc:606] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:25.621084  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:25.621845  1664 raft_consensus.cc:606] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:25.626844  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:25.627547  1664 raft_consensus.cc:606] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:25.629446   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:01:25.633065   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:25.641922  3105 raft_consensus.cc:2243] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:01:25.643100  3105 raft_consensus.cc:2272] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:01:25.648747  3105 ts_tablet_manager.cc:1916] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 13:01:25.682648  3105 ts_tablet_manager.cc:1929] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 13:01:25.683579  3105 log.cc:1199] T 110eba2eb7b34ea1b70ed63aca2feda8 P 316f3bf18ed94e32b238130a6bfedf1a: Deleting WAL directory at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0/minicluster-data/ts-0-root/wals/110eba2eb7b34ea1b70ed63aca2feda8
I20260709 13:01:25.686463  1478 catalog_manager.cc:5028] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): tablet 110eba2eb7b34ea1b70ed63aca2feda8 (table test-workload [id=680582cffa864e89afc37eaf28677d7f]) successfully deleted
I20260709 13:01:26.002667  3047 raft_consensus.cc:993] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:26.003304  3047 raft_consensus.cc:1081] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:26.007768  1586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:26.008455  1586 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:26.008831  1586 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:26.024405  1586 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.028183  1586 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:26.028873  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:26.029390  1664 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:26.075244  3056 raft_consensus.cc:993] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:26.075855  3056 raft_consensus.cc:1081] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:26.080960  1590 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36806
I20260709 13:01:26.082086  1590 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:26.082520  1590 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.060693  1664 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:26.092795  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:26.093858  3046 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:26.094427  3046 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:26.095463  3046 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.103976  1590 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:26.109253  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 86089.1us
I20260709 13:01:26.112797  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:26.113639  1664 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:26.114058  1664 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:26.114686  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:26.115408  2317 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.114928  1664 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:26.116777  1664 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.129124  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:26.130007  1733 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:26.130375  1733 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:26.131592  1733 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.138001  1590 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:26.136260  1733 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:26.135563  2317 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:26.163398  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:26.168319  3092 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:26.172101  3092 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:26.173266  3092 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:26.181450  1733 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:26.192837  1664 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:26.203986  1490 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 3 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:26.227790  3109 raft_consensus.cc:993] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:26.228482  2966 raft_consensus.cc:1081] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:26.232774  1734 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55272
I20260709 13:01:26.233656  1734 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:26.234102  1734 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.257911  1734 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.267462  1734 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:26.268013  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:26.268595  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:26.268922  1590 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.269309  1664 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:26.269757  1664 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:26.270807  1664 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.272149  1664 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:26.286499  1590 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:26.295959  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:26.296599  1664 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:26.297329  3110 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:26.297969  3110 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:26.299083  3110 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:26.327430  1490 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:26.374143  1490 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:26.538335  1664 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:26.540064  2903 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:26.609930  1733 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:26.611871  3046 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:26.634359   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:26.662989  1660 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:26.677687  1632 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e676e54141f7470db4551f4de439c017. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:26.679834  1733 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:26.682454  2932 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:26.694504  1664 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:26.697180  3046 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:26.701408  2315 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:26.703531  2903 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:26.708917  1586 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:26.718701  2966 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:26.759121  1664 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:26.765723  3108 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:26.851162  3109 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:26.854844  1590 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:26.861296   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:26.867728  1734 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:26.892922  1584 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:26.901729  2941 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:26.941697  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:26.944264  1735 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:27.026257  1706 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 162373eeadc2444e87ffd67df6a836e5. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:27.031703  1590 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:26.940649  1488 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:27.039726  1667 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 13:01:27.053632  3110 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:27.062173  2941 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:27.087705  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:27.088692  1664 raft_consensus.cc:606] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:27.098853  2966 raft_consensus.cc:993] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:27.099516  3109 raft_consensus.cc:1081] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:27.125540  3137 ts_tablet_manager.cc:933] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:27.142381  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:27.143460  1735 raft_consensus.cc:606] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:27.161000  1733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55272
I20260709 13:01:27.161881  1733 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:27.162248  1733 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.171049  1733 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:27.179819  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:27.180540  1660 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:27.180892  1660 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:27.181815  1660 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:27.183116  1660 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.186287  1584 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:27.187712  1584 raft_consensus.cc:606] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:27.194748  3108 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:27.199872  3137 tablet_copy_client.cc:323] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:01:27.202478  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet e676e54141f7470db4551f4de439c017 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:01:27.203117  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet e676e54141f7470db4551f4de439c017 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-e676e54141f7470db4551f4de439c017
I20260709 13:01:27.214746  1677 tablet_copy_source_session.cc:215] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:27.214423  1584 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:27.217063  1733 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:27.218068  1590 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:27.218947  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:27.219643  2315 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.220944  1590 raft_consensus.cc:606] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Received request to transfer leadership to TS ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:27.222491   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:27.224625   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:27.249612  3099 raft_consensus.cc:993] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower ff44a4643e01482cb6b6a79b099fa766 to start an election
I20260709 13:01:27.252529  3092 raft_consensus.cc:1081] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Signalling peer ff44a4643e01482cb6b6a79b099fa766 to start an election
I20260709 13:01:27.258834  1664 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:49484
I20260709 13:01:27.259549  1664 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:27.259938  1664 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:27.275894  3137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e676e54141f7470db4551f4de439c017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:27.288998  2315 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:27.293181  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:27.306216  3042 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:27.306838  3042 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:27.307885  3042 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:27.244652  1667 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:27.351408  3137 tablet_copy_client.cc:806] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:27.354313  3137 tablet_copy_client.cc:670] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:27.386394  3137 tablet_copy_client.cc:538] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:27.407286  3137 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:27.415553  1660 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:27.445241  3092 raft_consensus.cc:993] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:27.446162  1490 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:27.448107  3099 raft_consensus.cc:1081] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:01:27.466516  1664 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.470520  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:27.471205  1590 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:27.471496  1590 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:27.472280  1590 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.473766  1590 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:27.488479  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59050.9us
I20260709 13:01:27.493921  1734 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55270
I20260709 13:01:27.494601  1734 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:27.495007  1734 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:27.510865  1734 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.558406  1734 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:27.532364  1664 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:27.547183  1490 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 1 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:27.586634  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:27.587476  1664 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:27.565618  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:27.567660  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:27.608006  1589 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:27.608418  1589 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:27.609339  1589 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.610754  1589 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:27.568715  3137 tablet_bootstrap.cc:700] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent opening tablet: real 0.154s	user 0.002s	sys 0.000s
I20260709 13:01:27.617780  1735 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:27.627219  1589 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:27.637836  1735 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:27.641665  3143 ts_tablet_manager.cc:933] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:27.642385  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:27.648245  3109 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:27.661824  3109 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:27.520707  1590 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:27.662985  3109 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.690653  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:27.706647  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:27.708585  3142 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:27.709522  3142 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:27.710664  3142 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.719517  3143 tablet_copy_client.cc:323] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:27.722157  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 162373eeadc2444e87ffd67df6a836e5 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:27.722930  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 162373eeadc2444e87ffd67df6a836e5 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-162373eeadc2444e87ffd67df6a836e5
I20260709 13:01:27.729539  1667 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:27.741475  2941 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:27.744349  1752 tablet_copy_source_session.cc:215] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:27.753095  3143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 162373eeadc2444e87ffd67df6a836e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:01:27.758283  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 60698.3us
I20260709 13:01:27.771345  1488 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 4 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:27.773633  1491 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 2 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:01:27.827870  1539 auto_rebalancer.cc:629] Failed to schedule move for tablet 9a860695ab4c493b83dfcd08d8bee8de: Illegal state: Replica 316f3bf18ed94e32b238130a6bfedf1a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:01:27.835834  2941 raft_consensus.cc:993] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:27.843441  2317 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:27.836489  2941 raft_consensus.cc:1081] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:01:27.848894  3110 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 13:01:27.862789  1584 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:36802
I20260709 13:01:27.863778  1584 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:01:27.864319  1584 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.875941  1584 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.879000  3143 tablet_copy_client.cc:806] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:27.880018  3143 tablet_copy_client.cc:670] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:01:27.880722  1584 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:27.882651  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:27.883332  1664 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.891954  3137 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:27.893577  3137 tablet_bootstrap.cc:492] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:27.898986  1664 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:27.900660  3137 ts_tablet_manager.cc:1403] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.494s	user 0.032s	sys 0.000s
I20260709 13:01:27.907253  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:27.907425  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:27.907971  1733 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:27.908587  1733 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:27.909623  1733 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:27.911168  1733 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:27.954514  1733 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:27.956964  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52727.2us
I20260709 13:01:27.945310  2903 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:28.031777  3137 raft_consensus.cc:359] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:28.034159  3137 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
W20260709 13:01:28.050843  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 121145us
I20260709 13:01:27.974236  1733 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:28.121227  2903 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:28.127136  2903 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:28.111189  3137 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:28.190826  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57093.9us
I20260709 13:01:28.226262   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:01:28.246330  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94616.5us
W20260709 13:01:28.350977  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99762.9us
W20260709 13:01:28.478428  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 174220us
W20260709 13:01:28.528012  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 173680us
I20260709 13:01:28.531451  1733 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:01:28.540481  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 51494us
W20260709 13:01:28.579250  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.007s	sys 0.024s
W20260709 13:01:28.579744  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.007s	sys 0.024s
W20260709 13:01:28.597270  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46218.2us
W20260709 13:01:28.584685  3177 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.050s	user 0.001s	sys 0.001s
I20260709 13:01:28.621963  3182 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:28.627125  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44444.9us
I20260709 13:01:28.646765  1589 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:01:28.583894  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 640834us
I20260709 13:01:28.668010  3137 ts_tablet_manager.cc:1434] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.767s	user 0.016s	sys 0.008s
W20260709 13:01:28.668697  3137 ts_tablet_manager.cc:1467] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Tablet startup took 767ms
W20260709 13:01:28.669157  3137 ts_tablet_manager.cc:1469] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Trace:
0709 13:01:27.111936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.112158 (+   222us) service_pool.cc:224] Handling call
0709 13:01:27.125991 (+ 13833us) ts_tablet_manager.cc:934] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
0709 13:01:27.274097 (+148106us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:01:27.303819 (+ 29722us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:27.303976 (+   157us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:01:27.346820 (+ 42844us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:01:27.347196 (+   376us) tablet_replica.cc:206] Creating consensus instance
0709 13:01:27.347722 (+   526us) ts_tablet_manager.cc:989] Replica registered.
0709 13:01:27.347795 (+    73us) inbound_call.cc:177] Queueing success response
0709 13:01:27.406868 (+ 59073us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 13:01:27.901178 (+494310us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 13:01:27.953168 (+ 51990us) tablet_replica.cc:261] Starting instrumentation
0709 13:01:27.954096 (+   928us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":328,"mutex_wait_us":25443,"raft.queue_time_us":25398,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":28621,"tablet-copy.queue_time_us":5983,"tablet-download-pool.queue_time_us":139,"tablet-download-pool.run_cpu_time_us":2244,"tablet-download-pool.run_wall_time_us":5270,"thread_start_us":98552,"threads_started":2}
I20260709 13:01:28.713743  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-e676e54141f7470db4551f4de439c017 received from {username='slave'} at 127.0.0.1:56492
I20260709 13:01:28.714331  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-e676e54141f7470db4551f4de439c017 on tablet e676e54141f7470db4551f4de439c017 with peer e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:28.787635  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 156125us
W20260709 13:01:28.790531  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58732.7us
I20260709 13:01:28.804610  1664 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:28.731950  3172 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 13:01:28.826627  1590 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:28.801194  3143 tablet_copy_client.cc:538] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:28.853794  1593 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:28.841295  3141 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
W20260709 13:01:28.873982  1706 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfe67343bf1842888d6176ed2ffe1419. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:01:28.886876  1741 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:28.897047  3171 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:28.901957   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:28.902513  3143 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:28.895344  2941 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 13:01:28.914705  1661 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:28.909994  2317 raft_consensus.cc:1217] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 13:01:28.922056  3159 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:28.922760  1667 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:28.970957  3108 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:28.971262  3198 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:28.979708  1491 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 2 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:29.114948  1741 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:29.115866  1741 raft_consensus.cc:606] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:29.156340  1590 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:29.157294  1590 raft_consensus.cc:606] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Received request to transfer leadership to TS ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:29.157976  3042 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:29.450080  3201 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:29.461834  3201 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:29.492585  1667 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:01:29.503391  3215 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:01:29.516680  3201 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:29.557677  2966 raft_consensus.cc:1064] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:29.588135  3042 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:42018 (request call id 12) took 1623 ms. Trace:
I20260709 13:01:29.588901  3042 rpcz_store.cc:276] 0709 13:01:27.964568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:27.964846 (+   278us) service_pool.cc:224] Handling call
0709 13:01:28.130448 (+165602us) log.cc:844] Serialized 506 byte log entry
0709 13:01:28.736877 (+606429us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b1800314f80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:01:28.862687 (+125810us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b540013a220. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf5db26 00007fcdedf5749a 00007fcdedf56714 00007fcdedf36eaa 00007fcdedf3dd53 00007fcdedf3dcd9 00007fcdedf3dc69 00007fcdedf3dc31 00007fcdedf3cdfd 000000000038f354
0709 13:01:29.587991 (+725304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":266473,"raft.queue_time_us":289760,"raft.run_cpu_time_us":4010,"raft.run_wall_time_us":473200,"spinlock_wait_cycles":37934208,"thread_start_us":68262,"threads_started":1,"wal-append.queue_time_us":316991}
I20260709 13:01:29.561139  2966 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:29.627264  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:29.628468  3143 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:29.630350  3143 tablet_bootstrap.cc:492] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:29.631453  3143 ts_tablet_manager.cc:1403] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.729s	user 0.018s	sys 0.024s
I20260709 13:01:29.700083  3143 raft_consensus.cc:359] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:29.701083  3143 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:29.717986  3143 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:29.721014  3143 ts_tablet_manager.cc:1434] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.055s	user 0.005s	sys 0.000s
I20260709 13:01:29.735503  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:29.740995  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:29.743547  3181 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:29.758368  1738 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:29.763031  1584 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:29.770730  2317 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:29.773761  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-162373eeadc2444e87ffd67df6a836e5 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:29.774307  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-162373eeadc2444e87ffd67df6a836e5 on tablet 162373eeadc2444e87ffd67df6a836e5 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:29.774695  3155 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:29.783115  3172 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:29.797721  3230 ts_tablet_manager.cc:933] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:30.259645  3223 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:30.260588  3223 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:30.268463  3220 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:30.439704  3224 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:30.440847  3224 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:30.444294  3224 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:30.577014  3223 net_util.cc:396] Time spent resolve address for 127.0.107.195: real 0.311s	user 0.001s	sys 0.000s
I20260709 13:01:30.951897  3223 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:30.953786  3223 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:30.954696  3223 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:30.957837  3223 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:30.922138  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1653) took 1167 ms. Trace:
I20260709 13:01:30.684851  3220 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.017234  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1655) took 1261 ms. Trace:
I20260709 13:01:31.018034  3220 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.017964  1585 rpcz_store.cc:276] 0709 13:01:29.755262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.015805 (+1260543us) service_pool.cc:224] Handling call
0709 13:01:31.016325 (+   520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.016578 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.016616 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.016674 (+    58us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:31.016719 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.016790 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.016948 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:31.019644  3220 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.020246  3220 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.021061  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1136) took 1454 ms. Trace:
I20260709 13:01:31.021447  1739 rpcz_store.cc:276] 0709 13:01:29.566770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.569933 (+  3163us) service_pool.cc:224] Handling call
0709 13:01:31.020297 (+1450364us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:31.020530 (+   233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.020715 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.020741 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.020782 (+    41us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:31.020812 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.020869 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.020975 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3293440}
I20260709 13:01:31.022318  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1656) took 1266 ms. Trace:
I20260709 13:01:30.966187  3224 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.022858  1586 rpcz_store.cc:276] 0709 13:01:29.755521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.021228 (+1265707us) service_pool.cc:224] Handling call
0709 13:01:31.021669 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.021888 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.021923 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.021981 (+    58us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:31.022026 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.022099 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.022246 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:31.023555  3224 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.026086  1590 rpcz_store.cc:276] 0709 13:01:29.754685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:30.920912 (+1166227us) service_pool.cc:224] Handling call
0709 13:01:30.921423 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:30.921691 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:30.921730 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:30.921786 (+    56us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:30.921817 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.921888 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.922048 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:31.028646  3224 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:30.978881  3157 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:30.847209  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1662) took 1090 ms. Trace:
I20260709 13:01:31.033599  3220 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:31.032617  3230 thread.cc:641] tablet-download-pool [worker] (thread pool) Time spent creating pthread: real 1.231s	user 0.019s	sys 0.014s
I20260709 13:01:31.035475  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1657) took 1279 ms. Trace:
W20260709 13:01:31.038242  3230 thread.cc:608] tablet-download-pool [worker] (thread pool) Time spent starting thread: real 1.237s	user 0.019s	sys 0.014s
I20260709 13:01:31.033736  1587 rpcz_store.cc:276] 0709 13:01:29.757244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:30.846197 (+1088953us) service_pool.cc:224] Handling call
0709 13:01:30.846630 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:30.846835 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:30.846866 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:30.846909 (+    43us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:30.846939 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:30.846993 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:30.847136 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:31.038960  3230 tablet_copy_client.cc:323] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:01:31.038992  1589 rpcz_store.cc:276] 0709 13:01:29.755778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.033334 (+1277556us) service_pool.cc:224] Handling call
0709 13:01:31.034918 (+  1584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.035135 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.035163 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.035210 (+    47us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:31.035241 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.035296 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.035409 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:30.974258  3168 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:31.041460  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:31.036933  3220 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.046533  3220 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.049194  3220 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.041247  3168 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.051216  3168 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.033453  3157 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.053287  3157 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.053813  3157 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.054358  3157 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:31.056497  3157 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.056950  3157 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.036145  3223 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.036454  3224 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:01:31.060575  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.550s	user 0.049s	sys 0.117s
W20260709 13:01:31.064435  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.553s	user 0.049s	sys 0.117s
W20260709 13:01:31.064850  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.55397e+06us
I20260709 13:01:31.065462  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1658) took 1309 ms. Trace:
I20260709 13:01:31.065907  1591 rpcz_store.cc:276] 0709 13:01:29.756075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.064443 (+1308368us) service_pool.cc:224] Handling call
0709 13:01:31.064822 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.065000 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.065023 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.065079 (+    56us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:31.065110 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.065168 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.065282 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:31.066569  3222 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.067155  3222 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.060072  3194 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:01:31.058842  3224 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.069402  3222 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.075536  3224 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.075644  3222 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.231238  3222 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:31.057471  3157 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.084589  3233 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.073460  3220 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.086644  3141 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.090009  3160 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.266430  2941 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.095220  3092 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.058493  3223 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.133014  3155 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.063027  3232 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.118799  3179 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.214004  3238 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.063365  3168 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.117455  3169 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.214484  3154 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.214959  3046 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.137271  3159 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.217618  3194 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.221724  3186 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.223332  3173 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.224319  3110 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.225589  3175 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.174795  3167 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.160120  3108 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.233232  1590 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:31.240053  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1654) took 1485 ms. Trace:
I20260709 13:01:31.244505  3239 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.227106  3237 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.260728  3242 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.264133  2966 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.247777  3233 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.267102  2941 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.266741  3160 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:31.269603  3092 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.257155  3220 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.290498  1588 rpcz_store.cc:276] 0709 13:01:29.754990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.086516 (+1331526us) service_pool.cc:224] Handling call
0709 13:01:31.086884 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.087107 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:31.105822 (+ 18715us) log.cc:844] Serialized 14 byte log entry
0709 13:01:31.239636 (+133814us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.239711 (+    75us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:31.239773 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:31.239838 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:31.239981 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":124677,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":983,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":44680,"threads_started":2,"wal-append.queue_time_us":25570}
I20260709 13:01:31.279601  3168 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.277765  3179 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.284019  3194 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.268386  3141 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.284224  3046 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.305873  3157 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.288853  3110 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.306674  2966 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.275506  3155 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:31.286944  3186 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.363979  3234 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.288653  3173 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.297608  3108 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.275206  3232 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.364140  3157 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.377136  3164 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.281212  3154 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.318220  3175 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.306855  3239 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.380393  3245 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.377238  3249 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.277697  3238 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.380985  3240 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.372094  3145 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.388525  3189 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.279163  3169 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.294061  3159 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.294308  3167 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:31.393414  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83246.2us
W20260709 13:01:31.394513  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 77313.3us
I20260709 13:01:31.398609  3248 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.398789  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.399300  3248 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.407620  3167 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.407841  3169 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.318470  3242 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.386941  3157 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.403126  3223 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.415887  3247 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.382355  3234 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.308840  3237 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:31.420941  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.411006  3167 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.425423  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.428649  3160 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.388696  3145 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.408686  3244 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.412318  3169 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.418308  2941 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.420689  3220 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.421838  3194 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.391636  3164 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.415930  3246 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.389179  3189 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.395443  3249 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.388589  3240 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.438980  3092 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.440459  2941 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.440349  3155 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.434517  3167 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.455917  3046 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.395113  3245 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.539999  3179 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.540553  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.458244  3168 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.454653  3223 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.544131  2941 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.459295  3110 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.439924  3169 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.459477  2966 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.551941  3186 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.549584  3179 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.552390  3238 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.433506  3247 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.528785  3108 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.527877  3232 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.559276  3145 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.538691  3194 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.548017  3157 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.444743  3244 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.540887  3243 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.562842  3110 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:31.579906  1614 debug-util.cc:398] Leaking SignalData structure 0x7b0800159020 after lost signal to thread 434
I20260709 13:01:31.580344  3173 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.580605  3237 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.527415  3251 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.570405  3252 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:31.570344  3175 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.561296  3223 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.490391  3246 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:31.618716  3154 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.570233  3179 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.697867  3250 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.070416  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.028813  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.123989  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.027602  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.133205  3250 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.028349  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:31.941890  3141 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.014691  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.135383  2314 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.015131  3171 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.028370  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.136425  3171 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.021338  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:31.665891  3145 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:32.137724  3250 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.138402  2318 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.138067  3145 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.139029  3171 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.579942  3254 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.139945  3254 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.140837  3145 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.138113  2312 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:31.983390  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.789204  3168 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.031325  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.787204  3173 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.153793  3173 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.154285  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.155918  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.157073  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.157670  3173 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.158294  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.136682  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.031126  3222 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.159497  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.035820  3233 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.710000  3155 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.026997  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.160627  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.160861  3155 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.161274  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.045692  3156 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.143850  3223 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.048753  3255 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:31.666363  3159 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.048457  3172 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.067407  3174 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.011492  3198 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.049422  2966 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.959223  3262 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.015594  3042 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.693856  3248 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.692463  3251 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.169587  3156 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.693377  3242 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.782452  3169 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.705916  3167 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.042825  3162 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.204006  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:31.779832  3249 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.051638  3260 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.698722  3252 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.087457  3165 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.694738  3234 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.626554  3164 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.614696  3110 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.063948  3192 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.201678  3042 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.150841  3246 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.093735  3187 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:31.789143  3244 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.094409  3193 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:31.790802  3245 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.788455  2941 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.942986  3240 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:31.780728  3189 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:31.667423  3243 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.576493  3194 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.879441  3247 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.766693  3239 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:31.787922  3256 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.161702  3168 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.216198  3193 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.164454  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.202577  3174 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.344727  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.034970  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.162873  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.066007  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.193710  3172 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:31.745539  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:31.984061  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.354905  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:31.999325  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.358107  1593 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:32.358742  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:32.031044  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1661) took 2274 ms. Trace:
W20260709 13:01:32.039417  1614 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8e40 after lost signal to thread 1525
I20260709 13:01:32.361955  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.362659  3243 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.364017  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:32.364795  1614 debug-util.cc:398] Leaking SignalData structure 0x7b08002bf320 after lost signal to thread 1617
W20260709 13:01:32.045692  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.008s	sys 0.016s
I20260709 13:01:31.620950  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.365427  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:32.366096  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.008s	sys 0.016s
I20260709 13:01:32.367869  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:31.701989  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.154312  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.372625  3164 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:31.960489  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.003s	sys 0.005s
I20260709 13:01:32.199476  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:32.374495  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.972s	user 0.003s	sys 0.010s
I20260709 13:01:32.192773  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.216624  3187 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.374948  1592 rpcz_store.cc:276] 0709 13:01:29.756945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.303337 (+1546392us) service_pool.cc:224] Handling call
0709 13:01:31.303608 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:31.394789 (+ 91181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.394831 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:31.394886 (+    55us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:31.394927 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:32.030812 (+635885us) spinlock_profiling.cc:243] Waited 90.9 ms on lock 0x7b54000fa354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:32.030857 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:32.030982 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208970112}
I20260709 13:01:32.200466  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:31.784737  3108 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.381564  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.382452  3108 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.381577  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.384438  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.385027  3108 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.163291  3155 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.206460  3156 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.204216  3162 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.318881  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:32.172895  3171 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.386147  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.387329  3171 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.314999  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.303297  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.272979  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.316980  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.308408  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.317970  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.290875  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:31.959978  3186 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.201273  3188 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.145427  3179 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.192407  3197 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.403411  3187 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.193145  3237 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.229132  3252 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.408833  3237 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.410389  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.142086  3254 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.209842  3260 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.188585  3255 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.407420  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.201889  3262 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.227473  3267 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.095860  3258 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.122838  3259 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.217947  3265 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.429858  3258 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:32.215319  3192 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.435191  3259 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.210093  3165 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.354262  3189 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:32.351089  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.225651  3256 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:31.977329  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.003s	sys 0.010s
I20260709 13:01:32.353091  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.371236  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.374166  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.197693  2966 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.387982  3042 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.394066  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.373024  2941 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.395460  3171 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:32.430758  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 112244us
I20260709 13:01:32.396708  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.396595  3186 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.394186  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.391156  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.374689  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.398921  1740 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.399338  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.457064  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.403180  3251 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.197471  3198 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.405409  3197 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.397877  3188 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.374202  3164 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.391799  3243 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.426064  3155 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.426357  3110 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.424207  1734 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:32.427954  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.481642  1664 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.427382  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.422021  3267 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.415418  3162 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.432689  1666 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:32.431636  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.436769  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.442317  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:32.447539  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.006s	sys 0.010s
I20260709 13:01:32.437623  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.435998  3265 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.433164  3187 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.442176  3189 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.457830  2941 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.458369  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.459029  3255 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.459213  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.460002  3223 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.487764  1668 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.443900  3172 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.403025  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.464172  1667 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:32.451261  3194 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.460052  2966 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.460835  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.482120  1736 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:32.464725  3156 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.470798  3174 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.491218  1665 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:32.495910  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.497615  3159 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.492381  3243 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.508267  1735 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:32.507988  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.492127  1737 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.490621  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.508028  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.510299  3243 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.500078  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.501510  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.507987  3159 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.513583  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 483) took 1226 ms. Trace:
I20260709 13:01:32.511150  3155 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.514040  2313 rpcz_store.cc:276] 0709 13:01:31.286712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.978382 (+691670us) service_pool.cc:224] Handling call
0709 13:01:31.978911 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:32.447843 (+468932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:32.447933 (+    90us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:32.448024 (+    91us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:32.448076 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:32.513154 (+ 65078us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:32.513202 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:32.513355 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073134848}
I20260709 13:01:32.498776  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.516315  2310 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.510509  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.485770  3186 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.517596  1661 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.512253  3168 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.516423  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.522166  1738 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.526381  3187 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.492723  3223 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.534866  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.535238  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.546643  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.551213  1589 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.553941  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:32.554879  1664 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:32.511576  3188 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.558650  3187 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.559846  1735 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.555991  3165 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.567876  2941 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.561694  3198 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.569905  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.518803  3260 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.575855  3168 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.578938  3265 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.579082  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.578851  3168 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:32.580045  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:32.581032  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.582324  3258 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.582527  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1680) took 1145 ms. Trace:
I20260709 13:01:32.583033  1739 rpcz_store.cc:276] 0709 13:01:31.437211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.437346 (+   135us) service_pool.cc:224] Handling call
0709 13:01:31.437933 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:31.441681 (+  3748us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:31.441702 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:31.444292 (+  2590us) log.cc:844] Serialized 564 byte log entry
0709 13:01:32.507818 (+1063526us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:32.507927 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:32.582081 (+ 74154us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b54003516d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:32.582129 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:32.582176 (+    47us) raft_consensus.cc:1685] finished
0709 13:01:32.582186 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:32.582459 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7828864,"thread_start_us":1041469,"threads_started":1,"wal-append.queue_time_us":521825}
I20260709 13:01:32.586795  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.588636  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.595728  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.596203  3262 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.597743  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.515373  3159 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.543531  3223 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.604552  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.606261  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.606329  3189 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.607097  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.607635  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.608001  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.608273  1664 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.609723  3155 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.610683  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.597107  3256 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.606616  3237 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.609095  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.611405  2311 raft_consensus.cc:1217] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
W20260709 13:01:32.631779  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72584.8us
I20260709 13:01:32.634675  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet dfe67343bf1842888d6176ed2ffe1419 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:01:32.636039  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet dfe67343bf1842888d6176ed2ffe1419 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-dfe67343bf1842888d6176ed2ffe1419
I20260709 13:01:32.637928  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.609781  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.611454  1661 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.627542  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.609349  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.508313  3156 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.633126  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:32.642549  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.644888  1666 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:32.648660  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.648953  3159 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.649915  3159 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.650367  3159 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.650820  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.651253  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.652161  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.652295  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.597370  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.654316  1659 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:32.654891  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.655807  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:32.648676  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 130187us
I20260709 13:01:32.658535  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:32.659593  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.574949  3267 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.653368  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:32.603699  1614 debug-util.cc:398] Leaking SignalData structure 0x7b08000c2380 after lost signal to thread 1692
I20260709 13:01:32.653362  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.599550  3193 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.666285  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.603442  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.668794  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.674438  3156 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.668354  3141 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:01:32.704104  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 43645.7us
I20260709 13:01:32.677744  1666 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:32.658527  3164 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.684773  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.707744  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.708664  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.687693  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.688123  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.688686  3168 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.689702  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.690923  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.692137  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.692468  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.692858  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.693353  1488 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 234) took 3571 ms. Trace:
I20260709 13:01:32.694316  3159 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.694723  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.695602  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.696195  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.713900  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.698540  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.668354  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.716271  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.601807  3197 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:32.704160  1667 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:32.716111  1662 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:32.718035  2312 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.681669  3192 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.702140  1490 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 122) took 2976 ms. Trace:
W20260709 13:01:32.700524  1614 debug-util.cc:398] Leaking SignalData structure 0x7b0800169220 after lost signal to thread 1693
I20260709 13:01:32.710352  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.696477  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.709103  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.677352  1752 tablet_copy_source_session.cc:215] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:01:32.704315  3141 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.718676  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.712273  1488 rpcz_store.cc:276] 0709 13:01:29.121782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.134742 (+ 12960us) service_pool.cc:224] Handling call
0709 13:01:32.693115 (+3558373us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":3}
I20260709 13:01:32.724669  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:32.718524  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.721319  1666 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:32.721762  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.722118  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.722136  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.722946  1661 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:32.726629  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.720233  1490 rpcz_store.cc:276] 0709 13:01:29.724450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.725603 (+  1153us) service_pool.cc:224] Handling call
0709 13:01:32.700969 (+2975366us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":204682,"reported_tablets":1}
W20260709 13:01:32.728734  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95807us
I20260709 13:01:32.710959  1660 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:32.742262  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.742959  3259 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:32.744047  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.745172  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.745910  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.746101  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.746439  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.747182  1739 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:32.747309  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.747507  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.748052  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.749397  3246 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.734133  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.749871  3246 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.750290  3246 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.752614  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:32.753756  3259 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:32.754166  3259 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:32.754549  3259 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:32.755193  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:32.726348  3141 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.769439  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:32.783004  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98844.7us
I20260709 13:01:32.784332  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:32.785147  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:32.785656  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53805.1us
I20260709 13:01:32.785985  1662 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:32.787081  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.788461  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:32.789690  3260 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.790112  3260 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.790548  3260 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.790951  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 251) took 3129 ms. Trace:
I20260709 13:01:32.794495  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:32.791325  1489 rpcz_store.cc:276] 0709 13:01:29.660992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:31.142951 (+1481959us) service_pool.cc:224] Handling call
0709 13:01:32.790676 (+1647725us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":110909,"reported_tablets":2}
I20260709 13:01:32.802858  1664 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:32.807176  3173 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:32.805310  3192 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:32.807714  3173 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.808310  3173 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:32.808547  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.795327  3267 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.818897  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.821355  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.823510  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:32.801184  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:32.826193  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.829285  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.828246  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.764467  1665 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:32.821446  3267 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.832621  1665 raft_consensus.cc:606] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:32.849821  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.854965  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.861418  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.864989  3189 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.865418  3189 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:32.865828  3189 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:32.860500  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 81060.6us
I20260709 13:01:32.871213  3159 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:32.875182  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:32.876396  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.852864  1491 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: config changed from index -1 to 2, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 2 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:01:32.860900  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:32.853891  3267 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.887606  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:32.857635  3256 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.894428  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:32.895151  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 174007us
I20260709 13:01:32.895169  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:32.899739  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.899963  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:32.911742  1614 debug-util.cc:398] Leaking SignalData structure 0x7b0800275020 after lost signal to thread 1766
W20260709 13:01:32.912246  1614 debug-util.cc:398] Leaking SignalData structure 0x7b0800273fc0 after lost signal to thread 1767
I20260709 13:01:32.893483  3256 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.919224  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.923249  3271 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:32.923888  3271 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.924379  3271 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.940529  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.942189  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.957059  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.957446  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.957154  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.959161  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.919960  3256 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:32.962706  3275 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:32.963369  3275 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:32.965984  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:32.966975  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 84282.8us
W20260709 13:01:32.967869  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47613us
I20260709 13:01:32.970721  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.973021  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.975404  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.980500  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.982280  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.972843  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.987470  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.987515  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:32.929194  3223 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.005319  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:32.996888  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:32.997524  3268 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.000025  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:33.001600  3275 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.008320  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:33.009467  1735 raft_consensus.cc:606] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:32.954614  3259 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.018258  1587 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.033515  1591 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:33.043480  3259 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.045373  3223 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:33.053840  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:33.055115  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65233.2us
I20260709 13:01:33.056231  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:33.060809  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.032506  3042 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.078608  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:33.088760  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52061us
I20260709 13:01:33.090698  3173 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.076678  3159 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:33.086748  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 209841us
I20260709 13:01:33.101845  3155 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.102572  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.103590  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:33.105225  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:33.116227  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:33.076416  3042 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:33.124815  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.083110  3197 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:33.132658  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 163808us
W20260709 13:01:33.100306  1614 debug-util.cc:398] Leaking SignalData structure 0x7b080035bb20 after lost signal to thread 2342
I20260709 13:01:33.135347  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:33.138213  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:33.143309  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:33.132150  3042 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:33.148048  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 151287us
I20260709 13:01:33.118664  3159 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.144668  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:33.136554   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:33.150307  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.132019  3197 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.157780  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1674) took 1597 ms. Trace:
I20260709 13:01:33.093446  3173 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:33.160291  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57860.8us
I20260709 13:01:33.162411  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.158293  1587 rpcz_store.cc:276] 0709 13:01:31.554353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:32.448094 (+893741us) service_pool.cc:224] Handling call
0709 13:01:32.452500 (+  4406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:33.018898 (+566398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:33.151036 (+132138us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54003b15d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:33.151231 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1300631296}
I20260709 13:01:33.143323  3155 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:33.143463  3274 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.166931  2313 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:33.170881  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:33.171183  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46499.8us
I20260709 13:01:33.171269  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.172549  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.173126  3252 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:33.173655  3252 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.174119  3252 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:33.174942   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:01:33.179286  3272 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.180449  3223 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.182499  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:33.150630  3162 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.157471  3197 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.151865  3159 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.185842  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.154888  3262 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:33.182817  3237 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.186491  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.191249  3155 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.198782  3165 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:33.202824  3265 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:33.205155  3265 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:33.227033  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.185755  3162 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:01:33.229976  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94904.9us
I20260709 13:01:33.217018  3198 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.193181  3237 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.222782  3265 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:33.238338  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:33.244921  3276 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:01:33.246028  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75345.6us
I20260709 13:01:33.246778  3276 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.247013  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.247978  3192 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:33.248167  3279 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 13:01:33.248890  3156 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:33.248936  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.237110  1592 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:33.247219  3276 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:33.255240  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83361.2us
I20260709 13:01:33.256359  3247 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.268908  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.231050  3262 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:33.261873  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 128366us
I20260709 13:01:33.266338  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:33.233229  3223 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.270417  3187 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.270758  3223 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.270853  3187 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:33.271278  3223 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.271948  3187 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.236479  3162 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.275544  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:33.277029  3275 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.277276  3187 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.278383  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.240593  3237 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.282533  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:33.281112  3168 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.285702  3282 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.286973  3274 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.290115  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:33.290431  3265 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.291752  3168 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.293097  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:33.294595  3192 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.302205  3192 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:33.302696  3192 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:33.298273  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61311.8us
I20260709 13:01:33.171850  3141 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.273911  3276 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.300006  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.299460  3198 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.299597  3168 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.310846  3168 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.305539  3281 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.306403  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:33.279747  3186 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:33.281389  3246 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.530s	user 0.005s	sys 0.010s
W20260709 13:01:33.331264  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62478.8us
I20260709 13:01:33.332127  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.328999  3230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfe67343bf1842888d6176ed2ffe1419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:01:33.335832  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.309545  3198 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.282099  3271 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:33.314450  3281 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.322655  3282 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.359998  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:33.362319  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:33.374212  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67939.5us
I20260709 13:01:33.343696  3198 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.355811  3164 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.364905  3173 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.338109  3267 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.338392  3189 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:33.387480  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55446.9us
I20260709 13:01:33.323398  3246 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.363706  3281 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.389369  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:33.380481  3260 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.589s	user 0.005s	sys 0.010s
I20260709 13:01:33.379952  3265 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:33.402666  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59248.5us
I20260709 13:01:33.404028  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:33.391646  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76392.7us
I20260709 13:01:33.395000  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:33.390064  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.396788  3286 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:33.406387  3286 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.432024  3285 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:33.432689  3285 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.405790  3260 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.433125  3285 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:33.438048  2318 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.439373  3042 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.439980  2317 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.406826  3286 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:33.454902  3222 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.455488  3222 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:33.398368  3262 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:33.455973  3222 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:33.467811  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 72915.9us
I20260709 13:01:33.468816  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.436668  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:33.489539  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.475812  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.489548  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:33.451812  3164 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:33.495797  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53970.1us
I20260709 13:01:33.506367  3258 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.507026  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.508069  3258 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:33.474972  3042 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:33.499531  3186 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.498551  3141 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:33.512194  3262 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:33.642132  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:33.649289  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 173619us
I20260709 13:01:33.650426  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:33.667603  3256 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.682s	user 0.020s	sys 0.029s
I20260709 13:01:33.668118  3256 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.668391  3222 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:33.677704  3168 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.679399  3246 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.679697  3237 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:33.696137  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45806.6us
I20260709 13:01:33.660972  3271 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.702616  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:33.703514  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:33.700266  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 191769us
I20260709 13:01:33.632884  3162 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.711560  3262 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.712160  3262 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.712639  3262 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.713238  3291 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.713877  3291 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:33.716523  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 340342us
I20260709 13:01:33.683529  3280 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.697980  3267 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.718691  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.721834  3293 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.722429  3293 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.722838  3293 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.704497  3189 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:33.681470  3159 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.730490  2312 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:33.732002  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:33.734674  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.735939  3165 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.731122  3042 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.738271  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.742626  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.747453  3282 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.758783  3282 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.759204  3276 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.746309  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:33.733464  3168 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.762035  3222 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.769217  3042 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.775874  3042 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.747016  3280 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.767030  3159 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.738574  3288 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.773716  3155 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:33.778601  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46688.3us
W20260709 13:01:33.780256  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43659.9us
I20260709 13:01:33.780661  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.781484  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.785705  3276 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.787451  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.788936  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.791517  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:33.792989  3197 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.566s	user 0.005s	sys 0.005s
I20260709 13:01:33.793874  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.799515  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.800555  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:33.800992  3291 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.744079  3256 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.759249  3282 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.805907  3288 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:33.817013  3187 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.539s	user 0.015s	sys 0.007s
I20260709 13:01:33.822633  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:33.797288  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:33.792724  3192 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.794304  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.779747  3294 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.826279  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 464) took 1026 ms. Trace:
W20260709 13:01:33.826527  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45949.9us
I20260709 13:01:33.827764  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.815726  3280 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.794494  3162 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.817042  3260 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:33.828411  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63893us
I20260709 13:01:33.826712  2317 rpcz_store.cc:276] 0709 13:01:32.799914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:32.800054 (+   140us) service_pool.cc:224] Handling call
0709 13:01:32.802926 (+  2872us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:33.440392 (+637466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:33.826031 (+385639us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b54002e8454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:33.826218 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1464685312}
I20260709 13:01:33.805706  3189 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 13:01:33.793530  3292 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.833573  3222 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.834029  3222 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:33.834398  3222 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.836479  3284 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:33.838385  3291 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:33.833693  3288 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.825696  3285 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.744307  3223 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:33.818388  3156 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.566s	user 0.016s	sys 0.008s
I20260709 13:01:33.849581  3293 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.854243  3237 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.844594  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.833351  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.793471  3197 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.873898  3192 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.869783  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:33.879778  3198 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.880852  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:33.881819  1742 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:33.886502  3293 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.886920  3042 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:01:33.887247  3252 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.001s
I20260709 13:01:33.887331  3042 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:33.887725  3252 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.888386  3042 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.889022  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.890866  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.891405  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.892758  3276 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.892916  3162 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:33.893579  3162 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:33.894215  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.894645  3162 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.897007  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:33.895172  3268 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:01:33.899047  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:33.896327  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.902860  3288 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.903349  3288 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.903721  3288 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.904387  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.911590  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:33.911636  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 120217us
I20260709 13:01:33.913028  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.919955  3198 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.924413  3262 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.932116  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.933893  1665 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.939728  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:33.940667  1589 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:33.942436  3258 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:33.943692  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 110446us
I20260709 13:01:33.948982  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.949935  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:33.952585  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:33.958312  3197 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.960630  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:33.961793  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:33.962608  1590 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.962882  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:33.961912  3042 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:33.925426  3230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:58316 (request call id 476) took 4145 ms. Trace:
I20260709 13:01:33.929937  3268 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.965979  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:33.929301  3286 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.970506  3193 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:33.972924  3281 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.564s	user 0.017s	sys 0.010s
I20260709 13:01:33.973775  3286 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.973897  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:33.935253  3269 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:33.970327  3237 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.975167  3237 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:33.956482  3285 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:33.973372  3281 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.977639  3281 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:33.975646  3237 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:33.968386  3230 rpcz_store.cc:276] 0709 13:01:29.780299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:29.790302 (+ 10003us) service_pool.cc:224] Handling call
0709 13:01:29.798232 (+  7930us) ts_tablet_manager.cc:934] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
0709 13:01:33.324134 (+3525902us) tablet_copy_client.cc:368] Tablet copy session begun
0709 13:01:33.841023 (+516889us) tablet_metadata.cc:665] Metadata flushed
0709 13:01:33.841182 (+   159us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 13:01:33.924396 (+ 83214us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 13:01:33.924779 (+   383us) tablet_replica.cc:206] Creating consensus instance
0709 13:01:33.925179 (+   400us) ts_tablet_manager.cc:989] Replica registered.
0709 13:01:33.925304 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":281,"tablet-copy.queue_time_us":6259,"thread_start_us":1248232,"threads_started":2}
I20260709 13:01:33.981454  3230 tablet_copy_client.cc:806] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:01:33.981949  3294 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:33.982277  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:33.983260  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:33.983920  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:33.982393  3294 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:33.968778  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:33.986724  3280 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:33.985284  3281 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:33.937239  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:33.993040  3237 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:33.995421  3222 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.026818  1590 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:34.027233  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.028479  1589 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:33.991067  1665 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:34.028409  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.023777  3281 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:34.030782  3281 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.031091  3268 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:34.031661  3268 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:34.032660  3268 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:33.969244  3252 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.023844  3280 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:34.034785  3280 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.036064  3223 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.036639  3280 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.036849  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.003854  3230 tablet_copy_client.cc:670] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
W20260709 13:01:34.040200  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:34.041698  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73054.2us
I20260709 13:01:34.042368  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.044500  3290 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.046588  3222 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:34.047741  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.048228  3237 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.038245  3164 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.023643  3294 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.051112  3276 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:34.052012  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.052974  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.052742  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.051846  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.055444  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.058189  1589 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.056399  3292 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.027842  3288 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.065822  3288 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.066903  3271 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:34.067446  3271 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.067832  3271 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.070430  3276 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:34.070888  3276 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:34.071832  3276 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.073902  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.074749  3141 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.075500  3281 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.080582  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.084489  3281 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:34.086423  1587 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.088656  3259 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:34.090214  3259 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:34.090694  3259 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:34.091903  3259 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.089257  3230 tablet_copy_client.cc:538] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:01:34.093636  3259 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.039019  3286 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:34.102669  1589 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:34.104774  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49434.8us
I20260709 13:01:34.105924  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.106971  3193 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.112634  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.113076  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:34.115510  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49956.4us
I20260709 13:01:34.102682  3286 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:34.119560  3294 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.120168  3286 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.120774  1593 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.123261  3156 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:34.124301  3156 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:34.131736  3262 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.133812  3141 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:34.134353  3141 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.134740  3141 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.125294  3156 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:01:34.137460  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:34.137784  3156 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.082098  3276 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.139345  3281 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:34.146569  3292 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.149889  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.152208  3294 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.154276  3282 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.158411  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.145437  3281 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.162178  3187 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.162473  3281 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.163678  3282 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.164013  3187 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.165465  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:34.172041  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62371.6us
I20260709 13:01:34.172573  3292 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.173699  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.175988  3294 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.176918  3230 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:01:34.177044  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.177266   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:34.179042  1659 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:34.179566  3271 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.181041  1589 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.166062  1591 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.168977  1585 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.182518  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:34.182641  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:34.166338  1593 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.188164  3187 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.195639  2317 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.195992  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.198916  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.198998  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.199465  3284 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 13:01:34.204432  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.205073  3284 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.205344  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.205533  3284 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.214624  3284 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.215184  3141 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.216861  3284 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.217907  3247 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.218814  3247 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.166680  3156 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.208735  3286 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.227780  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.229671  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:34.225831  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 263059us
I20260709 13:01:34.187394  3292 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.189950  3162 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:34.231660  1591 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.236579  3164 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.242033  1592 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.244485  3141 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.244874  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.245595  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.246680  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.248346  3290 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.248641  3281 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.249464  3290 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:34.250583  3290 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:34.250950  3281 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.250957  3290 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.252207  3141 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.252264  3290 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.253875  3290 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.255676  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.256853  3164 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.257742  3156 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.258339  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.259213  3164 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.263441  3286 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:34.264571  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:34.266028  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.267002  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.267987  3164 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.268950  3274 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.267357  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.270699  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.271792  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.267520  3162 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:34.270613  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.269896  3164 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.277972  3274 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:34.278426  3247 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.278996  3247 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:34.277325  1488 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.279389  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.280087  1739 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:34.276307  3280 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.276649  3292 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.283002  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.280359  1488 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:34.279335  3247 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:34.287050  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.288368  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.287909  3247 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.289822  3247 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.290652  3141 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.291453  3141 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:34.291836  3141 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.292197  3141 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.295131  1739 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:34.300256  3292 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:34.301582  3292 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.301950  3292 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.303021  3292 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.304836  3292 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.308781  1742 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.313704  3162 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.316607  1742 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.320434  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.323915  1741 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.325747  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.319433  2312 raft_consensus.cc:2955] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:01:34.328490  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.328603  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.328799  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.329147  1668 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:34.324568  2311 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:34.329878  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.330802  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.333247  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.285928  3271 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.337036  3281 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.337071  3164 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:34.333370  1592 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.337903  3164 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.344568  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.345309  1661 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.345716  1591 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.346482  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.347939  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.348407  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.349022  1668 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:34.327654  2310 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:34.336159  1663 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.336304  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.336517  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:34.355243  1591 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.354331  3286 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:34.350625  3280 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:34.344568  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.358332  3286 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.358589  3280 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.348047  3187 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:01:34.359378  3286 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.360859  3280 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:34.362075  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.363415  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.364118  1660 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.364311  3187 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.364732  3187 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.365692  3187 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:34.366740  1557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.616s (SENT) [suppressed 6 similar messages]
I20260709 13:01:34.367291  3187 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.377698  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.379133  1663 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:34.375972  3258 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:34.385361  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.353138  3164 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:34.391686  1663 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:34.393018  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.395493  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.379851  3165 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 13:01:34.399655  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.402673  1663 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:34.423430  1592 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:34.427349  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:34.432492  3165 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:34.433795  3295 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.083s	user 0.001s	sys 0.001s
I20260709 13:01:34.435729  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.436048  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.437252  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.437681  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.437875  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.439020  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.439837  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.449316  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.465946  1668 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:34.469169  1661 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:01:34.469740  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 103048us
I20260709 13:01:34.473593  1742 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.475090  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.475265  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:34.478746  1733 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:34.480713  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.481005  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.472198  1660 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:34.485096  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.396644  3141 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.491325  1741 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.491097  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.493755  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.484841  3290 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:34.498307  1738 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.495285  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.478981  3193 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.512431  3247 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:34.497691  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.479270  3156 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.511847  3193 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.583356  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.584690  1733 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.586349  1740 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:34.588120  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.589399  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.590432  3141 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:34.590571  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:34.582095  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 201342us
I20260709 13:01:34.596120  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:34.594425  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.596228  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.599907  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.601073  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.603567  2310 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.605887  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.607039  1741 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.607256  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:34.607375  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.607791  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.605121  1735 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.613134  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:34.581736  3290 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.580825  3156 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.619405  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.613525  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:34.617923  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.617879  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.618307  1663 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.618758  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:34.621469  1734 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.623898  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:34.624750  1738 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:34.630867  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.641383  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.649256  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:34.664963  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:34.653964  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.653187  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.622712  1660 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:34.664121  3162 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
W20260709 13:01:34.674424  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 82150.1us
I20260709 13:01:34.675215  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.678167  1591 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:34.678773  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.654719  3262 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.646250  3282 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.682541  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.670332  3294 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.689831  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.674525  3192 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:34.703771  3192 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.683972  3287 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 13:01:34.692555  3299 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.684135  3286 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.694337  3294 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:34.674695  3193 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.685875  3280 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:34.708043  3276 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:34.708630  3193 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.700199  3290 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:34.711310  3284 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.711735  3284 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:34.688980  3222 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 13:01:34.718679  3222 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.719229  3222 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.708771  3286 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:34.721781  3286 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.723057  3286 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.729986  3287 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:34.730437  3287 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:34.731537  3287 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.720477  3222 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.736900  3222 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.705302  1737 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:34.748222  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 106951us
I20260709 13:01:34.749719  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:34.750411  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.754407  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.754784  1668 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.765568  1667 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:34.775179  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.775550  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.776285  1665 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:34.758230  3192 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.762563  3262 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.782791  1663 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:34.762801  3299 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:01:34.784996  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 173736us
I20260709 13:01:34.785037  3299 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:34.789745  3299 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:34.792920  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 171062us
I20260709 13:01:34.802809  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:34.809199  1665 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
W20260709 13:01:34.814661  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 126400us
W20260709 13:01:34.819074  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 129353us
I20260709 13:01:34.826076  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.803685  3156 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:34.834065  1740 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.838616  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.819926  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:34.841398  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.842195  1742 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:34.846997  1662 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:34.858143  1734 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:34.864861  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.883694  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:34.832309  3156 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:34.820302  3164 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.901769  1665 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:34.894778  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:34.909857  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:34.926537  1737 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:34.931308  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 111449us
W20260709 13:01:34.945451  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 106881us
I20260709 13:01:34.948238  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:34.950755  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:34.953025  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:34.896298  3247 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:34.913555  3192 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:01:34.926643  3292 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.621s	user 0.015s	sys 0.007s
I20260709 13:01:34.922705  3280 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.911134  3290 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
W20260709 13:01:35.007086  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54234.1us
I20260709 13:01:35.007959  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:34.975946  3321 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:34.973140  3247 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:35.027689  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.022774  3319 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:35.032779  3319 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.032198  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:35.035238  3326 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:35.040727  3287 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:35.042070  3317 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 13:01:35.000025  3262 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 13:01:35.024024  3192 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:35.078181  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.079337  3247 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:35.100023  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 205285us
I20260709 13:01:35.082423  3333 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:35.009649  3308 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.097s
I20260709 13:01:34.998404  3331 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.118s
I20260709 13:01:35.032768  3327 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260709 13:01:35.034066  3319 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:35.106619  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62346.7us
W20260709 13:01:35.115043  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.119133  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.100391  3281 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.106736  3339 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:35.134908  3339 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:35.139326  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 154529us
I20260709 13:01:35.140863  1733 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:35.135501  3339 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.142405  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:35.143579  1491 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 226) took 2846 ms. Trace:
W20260709 13:01:35.146981  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.144361  1491 rpcz_store.cc:276] 0709 13:01:32.297200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:32.297582 (+   382us) service_pool.cc:224] Handling call
0709 13:01:35.143515 (+2845933us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:33.391219 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:33.625183 (+233964us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:33.633618 (+  8435us) write_op.cc:435] Acquired schema lock
0709 13:01:33.649004 (+ 15386us) tablet.cc:662] Decoding operations
0709 13:01:33.720323 (+ 71319us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:33.720442 (+   119us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:33.720481 (+    39us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:33.720857 (+   376us) tablet.cc:701] Row locks acquired
0709 13:01:33.720907 (+    50us) write_op.cc:260] PREPARE: finished
0709 13:01:33.721089 (+   182us) write_op.cc:270] Start()
0709 13:01:33.721254 (+   165us) write_op.cc:276] Timestamp: P: 1783602093721027 usec, L: 0
0709 13:01:33.721300 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 13:01:33.722140 (+   840us) log.cc:844] Serialized 599 byte log entry
0709 13:01:34.362890 (+640750us) op_driver.cc:464] REPLICATION: finished
0709 13:01:34.598806 (+235916us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b180030a720. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:01:34.623965 (+ 25159us) write_op.cc:301] APPLY: starting
0709 13:01:34.649032 (+ 25067us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:34.664452 (+ 15420us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:34.672561 (+  8109us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:34.725888 (+ 53327us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:34.726156 (+   268us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:34.726324 (+   168us) write_op.cc:312] APPLY: finished
0709 13:01:34.726852 (+   528us) log.cc:844] Serialized 23 byte log entry
0709 13:01:34.727059 (+   207us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:34.727143 (+    84us) write_op.cc:454] Released schema lock
0709 13:01:34.727217 (+    74us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:34.727811 (+   594us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:34.727842 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":403092,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":51501,"apply.run_cpu_time_us":3001,"apply.run_wall_time_us":113851,"mutex_wait_us":34490,"num_ops":1,"prepare.queue_time_us":184379,"prepare.run_cpu_time_us":26205,"prepare.run_wall_time_us":466283,"raft.queue_time_us":376619,"raft.run_cpu_time_us":43075,"raft.run_wall_time_us":273223,"replication_time_us":641421,"spinlock_wait_cycles":75235584,"thread_start_us":460601,"threads_started":4,"wal-append.queue_time_us":183020}]]}
W20260709 13:01:35.163988  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50867.8us
I20260709 13:01:35.153064  3340 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:01:35.166880  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.170197  1591 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:35.171201  1589 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:35.161394  3276 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.187760  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:35.165104  3331 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:35.193842  3345 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:35.207149  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:35.222949  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81849.4us
I20260709 13:01:35.229336  3342 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:35.232620  3344 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:35.296419  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.167244  3326 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:01:35.346071  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 175823us
I20260709 13:01:35.358722  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:35.367931  1660 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.541s	user 0.012s	sys 0.000s
I20260709 13:01:35.368474  1660 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:35.180619  3343 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.192s
I20260709 13:01:35.370529  3343 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.370939  3343 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.371961  3343 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.191236  3187 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.162s
I20260709 13:01:35.373979  3326 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.212893  3276 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:35.381155  3222 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.641s	user 0.017s	sys 0.000s
I20260709 13:01:35.354058  3333 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.382395  3276 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:35.383270  3333 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.357010  3259 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 13:01:35.390319  3339 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.391898  3322 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.392457  3322 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.393564  3322 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.395567  3322 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.352788  3345 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:35.398886  3345 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.400018  3345 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.338840  3340 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 13:01:35.404476  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48686us
I20260709 13:01:35.406149  3338 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.406529  3338 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.407366  3338 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.408828  3338 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.410130  3337 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:35.425843  3299 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.636s	user 0.013s	sys 0.001s
I20260709 13:01:35.426334  3299 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.426553  3308 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:35.427668  3308 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.428766  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:35.410209  3282 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 13:01:35.432606  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:35.443325  2318 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:35.458464  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 251404us
I20260709 13:01:35.339922  3192 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.481350  1663 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:35.450488  3156 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:01:35.487010  3230 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 13:01:35.490850  3343 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:35.491967  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.491544  3339 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:35.486748  3262 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:01:35.486038  3344 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:01:35.522873  3351 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.515290  3299 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:35.514101  3337 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.516217  3342 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:35.527781  3337 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.536597  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:35.537781  1593 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:35.525619  3308 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.537920  3337 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.525064  3156 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.523396  3351 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.546326  3351 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:35.540670  3230 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:01:35.541308  3262 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.539346  3337 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.557499  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:35.560307  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 154663us
I20260709 13:01:35.569952  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.570834  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:35.573742  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 455657us
I20260709 13:01:35.574564  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:35.579123  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.579816  1742 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:35.552096  3230 tablet_bootstrap.cc:492] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:01:35.598127  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:35.599038  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:35.550109  3314 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.554028  3350 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:35.602600  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.618496  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.554723  3339 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.619967  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.543790  3342 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.562294  3262 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.549141  3351 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:35.615878  3319 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.013s	sys 0.010s
I20260709 13:01:35.563762  3299 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:35.598330  3294 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.636459  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:35.626194  3314 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.638684  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:35.594542  3141 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:01:35.634485  3319 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.013s	sys 0.010s
I20260709 13:01:35.642030  3155 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:35.642293  3346 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.549933  3352 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260709 13:01:35.648128  3359 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.635727  3156 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.653241  1591 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:35.634938  3230 ts_tablet_manager.cc:1403] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 1.458s	user 0.042s	sys 0.001s
I20260709 13:01:35.626047  3339 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.664929  3359 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.665814  3339 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.666842  3359 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:35.667066  3339 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:35.641111  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47965us
I20260709 13:01:35.578918  3192 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.646080  3141 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.670105  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:35.670395  3141 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:35.643095  3346 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.658820  3319 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.673841  3355 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.673951  3349 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.645002  3299 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.676983  3222 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:35.677727  3349 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.678128  3222 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.679242  3299 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:35.680071  3222 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.681270  3342 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:35.681552  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:35.677083  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:35.682255  1742 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:35.687765  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:35.673836  3356 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:01:35.703825  3354 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:35.704414  3357 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.705698  3294 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:35.707522  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:35.683380  3222 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:35.701854  3162 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:35.680121  3349 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.718350  3222 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.720320  3162 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.682476  3262 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.722700  3346 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:35.729040  3322 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:35.731024  3316 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:35.715378  3357 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.746114  3222 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.736737  3162 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.741611  3353 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 13:01:35.762837  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:35.765797  3286 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:35.766744  3359 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.767174  3359 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.767570  3230 raft_consensus.cc:359] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:35.768543  3230 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:01:35.777488  2311 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:35.786563  3314 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.755141  3316 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.791070  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:35.791896  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:35.792268  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:35.769488  3230 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:35.793042  1733 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:35.793545  3316 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.794775  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:35.797484  3314 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:35.798116  3314 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:35.798578  3314 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.796188  3230 ts_tablet_manager.cc:1434] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.132s	user 0.009s	sys 0.000s
I20260709 13:01:35.822487  3354 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:35.823043  3354 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.768090  3359 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.825327  3354 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.831095  3359 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.831513  3354 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:35.759963  3262 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.758486  3322 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:35.846879  3310 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:35.847851  3308 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.848860  3310 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.848862  3308 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.816972  3349 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:35.854084  3349 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.861176  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:35.862293  3357 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.862295  3308 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:35.865473  3341 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.866036  3341 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:35.867023  3341 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.867213  3322 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.834932  3316 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:35.869307  3341 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.885356  3307 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:35.891232  3307 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:35.893404  3307 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.895522  3307 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:35.897439  3349 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:35.900352  3321 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:35.920589  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 157853us
I20260709 13:01:36.507053  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:36.088929  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.695983  3307 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:36.696729  3307 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:36.116330  1661 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.023s	sys 0.018s
I20260709 13:01:36.343357  3383 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:36.736559  1661 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.025s	sys 0.018s
I20260709 13:01:36.736817  3383 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.737205  3385 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:36.737902  3385 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.738991  3353 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:35.927867  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.122551  1742 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:36.129237  1734 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:36.327122  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:36.125643  1733 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:36.806298  3341 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:36.811914  3341 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.123798  1740 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:36.132009  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:36.818774  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 164675us
I20260709 13:01:36.205042  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:36.820015  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:36.820533  2310 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:36.820815  3268 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:36.099692  3162 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:36.250396  3192 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.029s	sys 0.036s
I20260709 13:01:36.073791  3193 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:36.827936  3193 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.829829  3375 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.831033  3375 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:36.332202  3247 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:01:36.838047  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:36.838929  1733 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:36.016088  3310 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.103101  3314 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.846410  3314 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:35.885543  3316 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:35.925931  3321 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:36.140765  3327 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.005s	sys 0.017s
I20260709 13:01:36.163851  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:35.926594  3280 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:36.182379  3281 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:36.237298  3284 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:35.926746  3331 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:01:36.101379  3338 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.691s	user 0.016s	sys 0.000s
I20260709 13:01:36.079825  3347 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:36.087023  3349 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.441701  3350 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.394446  3348 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:01:36.338341  3363 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.100s	user 0.001s	sys 0.001s
I20260709 13:01:36.470197  1489 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 2 to 3, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:36.473584  1488 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 124) took 2228 ms. Trace:
I20260709 13:01:36.532593  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:36.550155  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.012s	sys 0.050s
W20260709 13:01:36.561964  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:36.609932  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.006s	sys 0.024s
W20260709 13:01:36.646059  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:36.699477  3307 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:36.707010  3372 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:36.707571  3187 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:36.708043  3365 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:36.710048  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:36.705072  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:36.725944  3345 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.725809  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:36.727478  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:36.728600  3354 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:36.734637  3384 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.735491  3294 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:36.739720  3353 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.740939  3290 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.741760  3271 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:36.742399  3222 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.743418  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 174) took 1177 ms. Trace:
I20260709 13:01:36.743954  3287 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:36.742004  3383 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.745328  3385 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.737396  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 176) took 1320 ms. Trace:
W20260709 13:01:36.757802  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 512124us
I20260709 13:01:36.760586  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 173) took 1196 ms. Trace:
I20260709 13:01:36.795087  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:36.798302  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-dfe67343bf1842888d6176ed2ffe1419 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:01:36.798636  1741 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:36.798834  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:36.799896  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:36.805048  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:36.810393  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 483442us
I20260709 13:01:36.814826  3341 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:36.815747  3334 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.816246  3274 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:36.816551  1734 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:36.824275  2311 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:01:36.825937  3192 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.029s	sys 0.039s
I20260709 13:01:36.826349  3162 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.834738  3375 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:36.160992  3259 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.853421  3321 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.922062  3365 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.923167  3321 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:36.924005  3281 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:36.924716  2317 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEARNER]: Advancing to term 3
I20260709 13:01:36.924998  3281 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.926076  3187 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.926827  3281 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:36.927670  3287 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.928273  3187 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.928804  3348 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:36.929848  3287 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.929751  3348 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.930738  3259 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.931429  3348 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:36.932087  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:36.932425  2312 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:36.932783  1591 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:36.934391  3334 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.935284  2310 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:36.935629  1666 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:36.936534  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-dfe67343bf1842888d6176ed2ffe1419 on tablet dfe67343bf1842888d6176ed2ffe1419 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:36.938189  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 478) took 1078 ms. Trace:
I20260709 13:01:36.938540  1736 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:36.939064  1735 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:36.938916  2311 rpcz_store.cc:276] 0709 13:01:35.860354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.860831 (+   477us) service_pool.cc:224] Handling call
0709 13:01:36.937926 (+1077095us) spinlock_profiling.cc:243] Waited 962 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:36.938100 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2212636032}
I20260709 13:01:36.940196  1659 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:36.940676  3274 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:36.941138  3274 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:36.941679  3310 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:36.942375  3327 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.837s	user 0.006s	sys 0.020s
I20260709 13:01:36.942297  3310 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.942870  1661 rpcz_store.cc:276] 0709 13:01:35.416906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.417626 (+   720us) service_pool.cc:224] Handling call
0709 13:01:35.430212 (+ 12586us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:35.430440 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:35.595776 (+165336us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.736958 (+1141182us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:36.737019 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:36.737066 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.737156 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.737310 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1678,"prepare.queue_time_us":297790,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":1586,"raft.queue_time_us":44677,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":11616,"thread_start_us":1304068,"threads_started":2}
I20260709 13:01:36.943957  3222 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:36.945832  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.349s	user 0.012s	sys 0.053s
I20260709 13:01:36.947840  3339 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:36.948551  3339 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.949894  1589 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:36.951789  3320 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:36.951943  3339 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:36.952317  3320 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:36.952724  3339 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:36.893827  3141 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:36.956367  3141 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.957374  3290 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.958249  3259 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.849311  3316 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:36.962392  3384 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.963225  1742 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:36.964442  3347 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.964751  1739 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:36.966301  3294 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:36.966724  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 434245us
I20260709 13:01:36.967613  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:36.968551  3354 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:36.969012  3385 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:36.969103  3354 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:36.970302  3331 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:36.970340  3354 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.971385  3331 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.972236  3339 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.976279  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:36.978487  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:36.979322  1740 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:36.979633  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.348s	user 0.008s	sys 0.025s
I20260709 13:01:36.980098  1661 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:36.981778  3372 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:36.985944  1663 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:36.961689  3271 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.988945  3280 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.990887  2318 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:36.993336  3320 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.000765  3307 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.001543  3307 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.003113  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:37.003938  3193 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.004438  3162 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:37.005427  3314 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:37.005532  3162 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.006186  3162 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:37.006086  3314 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.004619  3193 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.008095  3193 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.982062  3345 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.010694  3316 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:36.936769  1737 rpcz_store.cc:276] 0709 13:01:35.564575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.564726 (+   151us) service_pool.cc:224] Handling call
0709 13:01:35.565173 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:35.575189 (+ 10016us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:35.575228 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:35.575786 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 13:01:35.955772 (+379986us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:36.264472 (+308700us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.760167 (+495695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.760262 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:36.760306 (+    44us) raft_consensus.cc:1685] finished
0709 13:01:36.760320 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.760508 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":492888,"prepare.queue_time_us":287817,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":536693,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":4463,"thread_start_us":553526,"threads_started":2,"wal-append.queue_time_us":539432}
I20260709 13:01:37.019881  3386 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.020903  3386 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.022225  3320 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:37.026575  3187 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.027240  3187 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.028616  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:36.925302  3365 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.035544  3365 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:37.055759  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:37.058379  3259 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:37.059238  3259 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.059700  3259 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:37.011437  3316 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:36.961954  3349 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.067193  3280 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:37.068953  3349 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.070580  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.072423  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:37.073979  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 329126us
I20260709 13:01:37.074545  1587 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:37.075518  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.45563e+06us
I20260709 13:01:37.076613  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:37.078397  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:37.146432  2317 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:37.149727  1742 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:37.151124  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:37.083711  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.105614  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.154470  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.155433  1592 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:01:37.156584  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:01:37.158334  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.158929  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.145304  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.160913  1589 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:36.961851  1488 rpcz_store.cc:276] 0709 13:01:34.244434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:34.248266 (+  3832us) service_pool.cc:224] Handling call
0709 13:01:36.473208 (+2224942us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:34.313583 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:34.314200 (+   617us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:34.314234 (+    34us) write_op.cc:435] Acquired schema lock
0709 13:01:34.314254 (+    20us) tablet.cc:662] Decoding operations
0709 13:01:34.314681 (+   427us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:34.314741 (+    60us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:34.314778 (+    37us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:01:34.315087 (+   309us) tablet.cc:701] Row locks acquired
0709 13:01:34.315104 (+    17us) write_op.cc:260] PREPARE: finished
0709 13:01:34.323730 (+  8626us) write_op.cc:270] Start()
0709 13:01:34.333693 (+  9963us) write_op.cc:276] Timestamp: P: 1783602094315217 usec, L: 0
0709 13:01:34.333727 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 13:01:34.335394 (+  1667us) log.cc:844] Serialized 919 byte log entry
0709 13:01:34.487443 (+152049us) op_driver.cc:464] REPLICATION: finished
0709 13:01:34.762144 (+274701us) write_op.cc:301] APPLY: starting
0709 13:01:34.772017 (+  9873us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:34.781008 (+  8991us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:34.791018 (+ 10010us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:34.893608 (+102590us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:34.923353 (+ 29745us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 13:01:34.928323 (+  4970us) write_op.cc:312] APPLY: finished
0709 13:01:34.976539 (+ 48216us) log.cc:844] Serialized 29 byte log entry
0709 13:01:34.993991 (+ 17452us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:35.000669 (+  6678us) write_op.cc:454] Released schema lock
0709 13:01:35.005608 (+  4939us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:35.031809 (+ 26201us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:35.043862 (+ 12053us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":710906,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":267767,"apply.run_cpu_time_us":4474,"apply.run_wall_time_us":257648,"num_ops":2,"prepare.queue_time_us":327,"prepare.run_cpu_time_us":3275,"prepare.run_wall_time_us":22155,"replication_time_us":136868,"spinlock_wait_cycles":4352,"thread_start_us":167514,"threads_started":1}]]}
I20260709 13:01:37.162220  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:37.171862  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:37.172683  1734 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:37.172516  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:37.146473  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.185110  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.174486  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:37.186005  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64757.8us
I20260709 13:01:37.186255  1590 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:37.187965  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:37.188818  1742 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:37.177755  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 101279us
W20260709 13:01:37.191531  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 334124us
I20260709 13:01:37.192109  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:37.192651  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:37.036957  3365 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:37.194725  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 116600us
I20260709 13:01:37.176219  1739 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:37.193781   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:37.206327  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:37.068953  3193 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.208256  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.202907  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:37.210234  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:37.178416  1591 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:37.241125  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:37.244244  1737 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:37.146683  3280 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:36.979354  1738 rpcz_store.cc:276] 0709 13:01:35.565888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:35.566054 (+   166us) service_pool.cc:224] Handling call
0709 13:01:35.597457 (+ 31403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:35.597769 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:36.132021 (+534252us) log.cc:844] Serialized 14 byte log entry
0709 13:01:36.742702 (+610681us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18002f9e00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:01:36.742848 (+   146us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:36.743042 (+   194us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:36.743123 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:36.743209 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:36.743347 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5415,"prepare.queue_time_us":609289,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":509947,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"spinlock_wait_cycles":5023104,"thread_start_us":600753,"threads_started":2,"wal-append.queue_time_us":147693}
I20260709 13:01:37.157907  3337 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.207811  3356 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:01:37.255774  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.174208  3371 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.200572  3349 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:37.216769  3284 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.198664  3336 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:37.158320  3327 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.177886  3390 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.216782  3341 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.246287  3262 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:37.214634  3294 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.200580  3322 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 13:01:37.293870  3286 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.295944  3394 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:37.253592  3314 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.260010  3388 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:37.243429  3287 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 13:01:37.275678  3349 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.337021  1663 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:37.326643  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:37.377825  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.378921  1663 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:37.290108  3353 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.270615  3337 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:37.411950  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.287349  3386 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.383852  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:37.385617  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 193133us
I20260709 13:01:37.412729  1663 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
W20260709 13:01:37.413659  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 203511us
I20260709 13:01:37.263985  3280 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.298051  3187 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.305668  3271 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.432969  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:37.303064  3334 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.314234  3192 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:37.443338  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:37.346161  3349 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:37.281666  3371 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:37.447794  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 247073us
I20260709 13:01:37.340209  3347 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.448983  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:37.411405  3357 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:37.452329  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.453311  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.383227  3314 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.430670  3339 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.339942  3384 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.434330  3310 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.410569  3372 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.461009  1663 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:37.463955  1588 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:37.426599  3290 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.464548  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:37.451367  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:37.447968  1689 debug-util.cc:398] Leaking SignalData structure 0x7b0800114ea0 after lost signal to thread 434
I20260709 13:01:37.445654  3356 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:37.464263  3357 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:37.473285  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.312964  3327 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.473805  3350 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.454300  3193 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:37.538533  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:37.478892  3287 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.488943  2318 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:37.315304  3286 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.333297  3394 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:37.451738  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 233756us
I20260709 13:01:37.461695  3141 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.298072  3390 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.479733  1740 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:37.503819  3357 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:37.484036  3310 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:37.539674  2317 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:37.454690  3271 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.383376  3388 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.485531  3384 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:37.490696  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:37.579259  1661 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:37.497407  3372 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:37.487602  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:37.557441  1666 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.621s	user 0.009s	sys 0.005s
I20260709 13:01:37.475068  3314 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.561436  3404 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:37.565032  3391 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:01:37.570050  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:37.539305  3193 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.571888  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:37.572279  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:37.593003  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 352054us
I20260709 13:01:37.572881  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:37.561877  1689 debug-util.cc:398] Leaking SignalData structure 0x7b0800057e80 after lost signal to thread 1525
I20260709 13:01:37.600589  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.601692  1662 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:37.561972  3310 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:37.557974  3397 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 13:01:37.546555  3350 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:37.592005  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.558075  3386 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.541731  3398 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 13:01:37.631273  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.578444  3354 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.611162  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:37.550511  3287 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.581241  1666 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:37.584522  3384 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.592463  3402 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 13:01:37.598595  3365 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.598966  3334 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.612881  3310 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:37.600117  3193 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:37.680166  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.640889  1668 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:37.648427  1661 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:37.633325  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.581995  3271 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.598387  3372 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.606278  3375 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:37.634152  3274 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.692s	user 0.003s	sys 0.021s
W20260709 13:01:37.635455  1689 debug-util.cc:398] Leaking SignalData structure 0x7b0800347840 after lost signal to thread 1617
I20260709 13:01:37.604384  3396 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.685688  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.686614  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.699407  1666 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:37.687357  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.687697  1662 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:37.596801  3404 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.634497  3192 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:37.639986  3390 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.695806  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.703790  1661 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:37.681043  3365 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.642318  3268 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 13:01:37.706146  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.716006  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.694044  3322 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:01:37.708328  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 109858us
I20260709 13:01:37.685149  3375 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.711594  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.713733  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:37.696429  3280 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:37.714902  1589 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.715152  1590 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.641059  3386 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.717252  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.724689  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:37.719730  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132432us
I20260709 13:01:37.713443  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:37.721562  1632 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
W20260709 13:01:37.721832  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 111061us
I20260709 13:01:37.690737  3331 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:01:37.710335  3371 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.692054  3274 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.683174  3334 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.676033  3354 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.733421  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.838212  1587 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:37.807780  1589 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:37.840222  1632 leader_election.cc:403] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:01:37.840883  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 130668us
I20260709 13:01:37.842334  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.725754  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:37.769865  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43891us
I20260709 13:01:37.846352  1667 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:37.846837  1668 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:37.791287  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:37.610037  3356 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.710584  3337 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:37.711233  3192 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:37.855167  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:37.856475  3403 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 13:01:37.856738  3359 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:37.857156  3162 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.850s	user 0.008s	sys 0.013s
I20260709 13:01:37.857524  3402 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.858222  3280 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.912285  3162 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.915076  2314 raft_consensus.cc:2368] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:37.894593  3271 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:37.860692  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.702155  3396 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.858150  3384 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.864915  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:37.818600  1584 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:37.776978  3290 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.093s
I20260709 13:01:37.721447  3322 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:37.720510  3365 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.720659  3390 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:37.873952  3320 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.851s	user 0.009s	sys 0.006s
I20260709 13:01:37.875505  1587 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 13:01:37.788978  3259 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.729s	user 0.011s	sys 0.005s
I20260709 13:01:37.722816  3375 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:37.890194  1592 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:37.793171  3395 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.819787  3286 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.815338  3327 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:37.750245  3399 raft_consensus.cc:2764] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:01:37.749961  3411 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.777765  3407 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:37.862838  3314 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.875062  3274 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:37.902890  3388 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.887115  3371 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:37.859359  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:37.918196  3280 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:37.858103  3400 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.135s
I20260709 13:01:37.875005  3193 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.912587  3402 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.397459  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:38.044319  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.070333  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:38.209286  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.001s	sys 0.015s
W20260709 13:01:38.406054  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.001s	sys 0.018s
I20260709 13:01:38.405946  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:38.405277  1668 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:38.397459  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:38.277827  1736 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:01:38.397763  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 520899us
W20260709 13:01:37.923593  3281 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.995s	user 0.008s	sys 0.009s
W20260709 13:01:38.358064  3357 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.778s	user 0.004s	sys 0.008s
W20260709 13:01:38.404460  2282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.010s	sys 0.009s
I20260709 13:01:37.902890  3287 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:38.407855  1587 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
W20260709 13:01:38.455648  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46312.9us
I20260709 13:01:38.453150  1663 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:37.927496  3372 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:38.406906  1593 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
W20260709 13:01:38.428046  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.469350  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:38.470522  1590 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:01:38.368158  1659 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.427s	user 0.012s	sys 0.006s
W20260709 13:01:38.360664  3284 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.077s	user 0.014s	sys 0.000s
W20260709 13:01:38.469403  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.051s	sys 0.076s
W20260709 13:01:38.529611  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.051s	sys 0.082s
W20260709 13:01:38.475224  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 405035us
W20260709 13:01:38.465938  2282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.010s	sys 0.010s
I20260709 13:01:38.406906  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:37.936932  1666 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:37.900431  3307 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:38.182874  3348 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.144s	user 0.012s	sys 0.002s
W20260709 13:01:37.932018  3321 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.008s	user 0.016s	sys 0.000s
I20260709 13:01:38.562058  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.487125  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:38.575074  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:38.576241  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89231.1us
W20260709 13:01:38.561025  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 154227us
I20260709 13:01:38.101327  3192 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:38.537153  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.457437  3193 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:38.516888  3294 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:38.583931  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:37.918787  3394 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:37.924405  3359 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.315842  3354 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.245110  3337 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:37.920800  3386 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.398564  3365 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.577666  3399 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:38.478472  3338 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:38.267923  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:38.587250  3268 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:38.461380  3259 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.541208  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1223) took 1390 ms. Trace:
I20260709 13:01:38.525791  1668 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:38.542610  1665 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:38.461676  3388 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.452407  3162 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:38.522385  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.627758  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:37.925115  3404 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.535068  1659 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:38.580206  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.580636  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.581059  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.629205  1664 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
W20260709 13:01:38.629077  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1828) took 1925 ms (client timeout 1960 ms). Trace:
W20260709 13:01:38.527948  1689 debug-util.cc:398] Leaking SignalData structure 0x7b0800369c80 after lost signal to thread 1692
I20260709 13:01:38.629768  1735 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:38.581687  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:37.935906  3334 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.583570  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.584132  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.614877  1736 rpcz_store.cc:276] 0709 13:01:37.150683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.153422 (+  2739us) service_pool.cc:224] Handling call
0709 13:01:37.194287 (+ 40865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.407903 (+1213616us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.540833 (+132930us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400404554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:38.541112 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2471316352}
I20260709 13:01:38.631047  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.528231  3271 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.607941  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.632041  3271 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.591427  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:38.592504  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.589093  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.460544  3407 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.602499  3281 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.532068  3274 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:38.588900  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:38.597445  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:38.560228  3412 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.310144  3387 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:37.936412  3396 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:38.593550  3356 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.008s	sys 0.009s
I20260709 13:01:37.939319  3390 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.582649  3421 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:38.460767  3371 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.469637  3395 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.477859  3357 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.587416  3413 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:38.588379  3294 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:37.930238  3349 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:37.938805  3322 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.602356  3397 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:38.469763  3411 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.602790  3280 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:38.636776  3359 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:38.905838  3316 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.647326  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.644753  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.893402  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.912516  1742 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:38.895449  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1249) took 1245 ms. Trace:
I20260709 13:01:38.629153  1737 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:38.913930  1662 rpcz_store.cc:276] 0709 13:01:37.650876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.702793 (+ 51917us) service_pool.cc:224] Handling call
0709 13:01:37.703503 (+   710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:37.939589 (+236086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:37.939635 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:37.939681 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:37.939719 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.895056 (+955337us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:38.895130 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:38.895357 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542364032}
I20260709 13:01:38.919456  2313 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:01:38.630242  1689 debug-util.cc:398] Leaking SignalData structure 0x7b0800369ca0 after lost signal to thread 1693
I20260709 13:01:38.798955  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:38.630044  1659 rpcz_store.cc:269] 0709 13:01:36.703563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:36.704762 (+  1199us) service_pool.cc:224] Handling call
0709 13:01:38.628950 (+1924188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:38.923349  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.010180  1737 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:38.926924  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.923028  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:38.664593  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75773.7us
I20260709 13:01:39.012401  1742 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:39.012151  1736 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.017215  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:39.014003  1733 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:38.617820  3310 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.925s	user 0.005s	sys 0.011s
I20260709 13:01:38.647722  3327 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:38.734107  3290 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:38.930236  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:38.614523  3354 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:38.635397  3389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.008s	sys 0.040s
W20260709 13:01:38.920805  3356 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.013s	sys 0.021s
I20260709 13:01:39.017478  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:39.039932  1662 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:38.744984  3392 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:39.041005  3392 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:39.022884  1742 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:38.922969  3314 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:38.615341  3338 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.750939  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:38.988986  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67463.9us
I20260709 13:01:38.632176  1735 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:38.606215  3337 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:38.909125  3341 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.013772  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:39.057742  1742 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:39.048005  1667 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:38.912421  3348 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.029482  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:39.026336  2314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 531) took 1999 ms (client timeout 2013 ms). Trace:
I20260709 13:01:38.745131  3284 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:38.907567  3345 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.078137  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:38.647452  3259 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:38.613849  3222 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:38.917344  3274 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.031523  1739 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:38.649565  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 191) took 1108 ms. Trace:
I20260709 13:01:38.646477  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:38.645666  3268 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:39.036346  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 187) took 1552 ms (client timeout 1819 ms). Trace:
I20260709 13:01:38.634431  3410 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.014534  3400 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:39.111889  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:39.100286  1665 rpcz_store.cc:276] 0709 13:01:37.541576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.541842 (+   266us) service_pool.cc:224] Handling call
0709 13:01:38.649373 (+1107531us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:38.649480 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2164841344}
I20260709 13:01:39.041730  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:38.642957  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1220) took 1495 ms. Trace:
I20260709 13:01:38.822815  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:39.053241  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:38.919762  3421 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:38.943141  3280 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:39.030772  3310 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:39.058363  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44718.9us
W20260709 13:01:39.061767  2314 rpcz_store.cc:269] 0709 13:01:37.028036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.028225 (+   189us) service_pool.cc:224] Handling call
0709 13:01:39.016794 (+1988569us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54003ee4d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:39.026253 (+  9459us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2036683136}
I20260709 13:01:39.030174  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1951) took 1298 ms. Trace:
I20260709 13:01:39.132215  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:39.064771  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:39.133836  2313 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:39.016331  3412 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:38.798627  3187 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:39.038591  2311 raft_consensus.cc:2368] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:39.073897  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.075801  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:39.075924  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:39.165668  1587 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:39.079391  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 366075us
I20260709 13:01:38.649310  3399 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:38.922317  3388 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.054801  3392 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:39.047672  3389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.295s	user 0.015s	sys 0.050s
I20260709 13:01:38.909549  3426 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:39.091498  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 96662.9us
I20260709 13:01:38.922319  3281 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.102097  1735 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:39.105242  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1241) took 1465 ms. Trace:
W20260709 13:01:39.101293  1663 rpcz_store.cc:269] 0709 13:01:37.483888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.487174 (+  3286us) service_pool.cc:224] Handling call
0709 13:01:39.036125 (+1548951us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54002f2954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:39.036276 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005027584}
I20260709 13:01:39.116954  3427 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.008142  3431 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:39.070245  1689 debug-util.cc:398] Leaking SignalData structure 0x7b080035ada0 after lost signal to thread 1766
I20260709 13:01:39.056208  3286 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.048903  3408 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:39.114630  1741 rpcz_store.cc:276] 0709 13:01:37.147634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.174460 (+ 26826us) service_pool.cc:224] Handling call
0709 13:01:37.207545 (+ 33085us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:37.459787 (+252242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:37.459828 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:37.459881 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:37.459920 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:38.642614 (+1182694us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:38.642681 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:38.642872 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":579606272}
I20260709 13:01:39.125473  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.126614  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.220722  1742 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
W20260709 13:01:39.127988  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 305327us
I20260709 13:01:39.079833  3387 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:38.798194  3404 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:39.082165  3353 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:39.038807  3290 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.080134  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:39.151954  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 82502.6us
I20260709 13:01:39.136395  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:39.158840  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:38.922389  3397 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.009008  3428 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:39.044194  3354 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.132472  1590 rpcz_store.cc:276] 0709 13:01:37.722562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.880702 (+158140us) service_pool.cc:224] Handling call
0709 13:01:37.893968 (+ 13266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.524639 (+630671us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.021021 (+496382us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400431054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:39.021244 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1325119360}
I20260709 13:01:39.037297  3413 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:39.161984  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 534328us
I20260709 13:01:39.112843  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:39.294181  3423 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:39.294622  3407 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.092772  3402 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.295060  3423 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.100375  3359 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:39.296650  3359 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.113199  3274 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:39.122228  3222 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:39.176155  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.192452  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1250) took 1541 ms. Trace:
I20260709 13:01:39.301712  3222 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.301781  1661 rpcz_store.cc:276] 0709 13:01:37.651424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.711769 (+ 60345us) service_pool.cc:224] Handling call
0709 13:01:37.712090 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.032176 (+1320086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.032208 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:39.032254 (+    46us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:39.032295 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.192171 (+159876us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:39.192219 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.192384 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3035630976}
I20260709 13:01:39.303431  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.303740  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:39.304693  1661 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.187078  1664 rpcz_store.cc:276] 0709 13:01:37.642132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.642334 (+   202us) service_pool.cc:224] Handling call
0709 13:01:38.628928 (+986594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.629856 (+   928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.104884 (+475028us) spinlock_profiling.cc:243] Waited 960 ms on lock 0x7b540013ced0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:39.105135 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2207442304}
I20260709 13:01:39.056720  3314 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.128939  3262 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:39.085867  3348 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.130470  3421 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.080471  3341 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.082392  3284 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.226212  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:39.228226  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:39.169237  3316 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:39.203835  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:39.118357  3432 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:39.173748  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1236) took 1531 ms. Trace:
W20260709 13:01:39.347203  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 119386us
I20260709 13:01:39.267758  3350 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.263083  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:39.267741  3435 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.268081  3390 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.268651  3395 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.082258  3294 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.268790  3338 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.256706  3431 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.272114  3349 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.272768  1593 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:39.293345  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:39.296123  3192 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.698s	user 0.007s	sys 0.004s
I20260709 13:01:39.243708  3353 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:39.244122  3430 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:39.173053  3187 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.324101  1661 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:39.327592  3354 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.327796  3262 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.328703  2312 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:39.329663  3268 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.330700  3388 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.385645  3371 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.331379  3281 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.238984  3286 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.256155  3327 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:39.224166  3419 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:39.344743  1665 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 4
W20260709 13:01:39.248617  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 201764us
I20260709 13:01:39.258515  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.349164  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:39.138700  3280 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.233213  3426 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.243530  3427 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.273378  3408 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:39.360011  1738 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:39.363173  1689 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd600 after lost signal to thread 2342
I20260709 13:01:39.365442  1737 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:39.160126  3410 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:39.381623  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 346459us
I20260709 13:01:39.344040  3316 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:39.370234  3350 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:39.396767  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:39.367444  3432 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:39.261224  3404 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:39.514708  3397 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.517436  3404 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.334437  3428 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.520049  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:39.401048  3294 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.450007  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:39.416014  3429 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:39.473811  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 124743us
I20260709 13:01:39.383407  3435 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:39.472429  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 333218us
I20260709 13:01:39.413153  3423 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.480515  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:39.533272  1590 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.390841  3192 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:39.434419  3337 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.411113  3431 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.489065  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.484793  3434 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:39.496479  3286 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.488219  3222 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.456525  1659 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:39.498132  3442 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:39.556479  3284 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.471280  3281 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:39.558480  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1847) took 1705 ms. Trace:
I20260709 13:01:39.559012  1659 rpcz_store.cc:276] 0709 13:01:37.851806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.379353 (+1527547us) service_pool.cc:224] Handling call
0709 13:01:39.404296 (+ 24943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.557123 (+152827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.557439 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:39.472872  3310 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.502074  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1846) took 1651 ms. Trace:
I20260709 13:01:39.562364  3187 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.560494  1668 rpcz_store.cc:276] 0709 13:01:37.851186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.321961 (+1470775us) service_pool.cc:224] Handling call
0709 13:01:39.332368 (+ 10407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:39.332732 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.332761 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:39.333311 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 13:01:39.471790 (+138479us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:39.471849 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.471924 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:39.501786 (+ 29862us) raft_consensus.cc:1685] finished
0709 13:01:39.501811 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.501987 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4901,"raft.queue_time_us":3659,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":13889,"thread_start_us":123886,"threads_started":1,"wal-append.queue_time_us":164746}
I20260709 13:01:39.564524  3402 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:39.502178  3413 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.502528  3412 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.473434  3353 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.397904  3430 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.497596  3327 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.504885  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.406924  3356 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.066s
I20260709 13:01:39.507220  1661 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.428275  3357 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.449322  3440 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:39.526502  1667 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:39.496735  3399 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.432940  3405 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.311s
I20260709 13:01:39.533519  1662 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:39.534077  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:39.529655  2310 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
W20260709 13:01:39.536706  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 86779.5us
W20260709 13:01:39.589979  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55986.9us
I20260709 13:01:39.506047  3316 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.539563  3411 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:39.538614  1736 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:39.541611  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:39.529081  3429 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.522013  3432 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:39.571913  1741 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
W20260709 13:01:39.570719  3375 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.108s	user 0.016s	sys 0.001s
I20260709 13:01:39.603907  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.526510  3350 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.619899  1588 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:01:39.347586  1660 rpcz_store.cc:276] 0709 13:01:37.638211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:37.676587 (+ 38376us) service_pool.cc:224] Handling call
0709 13:01:37.698054 (+ 21467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:38.266262 (+568208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:38.266315 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:38.266388 (+    73us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:38.266446 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.122554 (+856108us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:39.129460 (+  6906us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.169379 (+ 39919us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":535,"raft.queue_time_us":646521,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":43296,"spinlock_wait_cycles":430156160,"thread_start_us":377633,"threads_started":1}
I20260709 13:01:39.550136  3437 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:01:39.621948  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80600.9us
I20260709 13:01:39.584344  1665 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:39.600874  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.602970  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.599768  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.625741  1591 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:39.626120  1592 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:39.601704  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.578364  3348 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.590286  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:39.626814  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.627975  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 195) took 1044 ms. Trace:
I20260709 13:01:39.628957  1665 rpcz_store.cc:276] 0709 13:01:38.583796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.160020 (+576224us) service_pool.cc:224] Handling call
0709 13:01:39.627906 (+467886us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:39.549186  3192 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.578923  3430 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.633092  1587 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:39.634984  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.627441  1591 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:39.639003  1668 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:01:39.639719  1593 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:39.639719  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1255) took 1048 ms. Trace:
I20260709 13:01:39.629287  1661 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:39.603461  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.640805  1668 rpcz_store.cc:276] 0709 13:01:38.591470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.587690 (+996220us) service_pool.cc:224] Handling call
0709 13:01:39.638558 (+ 50868us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.639452 (+   894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.639655 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:39.641682  1589 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
W20260709 13:01:39.642513  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52353.6us
I20260709 13:01:39.640316  3336 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:39.630089  1590 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:39.657955  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:39.656466  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.640651  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:39.478166  3419 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.811486  3428 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.657702  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:40.041216  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:39.677238  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:40.042232  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.042363  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:39.690728  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:39.755932  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.021s	sys 0.053s
I20260709 13:01:39.645007  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.697431  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.050967  1589 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:40.049575  1592 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:39.944578  1733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.719s	user 0.042s	sys 0.060s
W20260709 13:01:40.051834  1733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.043s	sys 0.065s
I20260709 13:01:39.669696  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.052340  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.657018  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:39.831324  2317 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.015s	sys 0.013s
W20260709 13:01:40.054430  2317 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.015s	sys 0.020s
W20260709 13:01:40.055271  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 378191us
I20260709 13:01:39.637393  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.056902  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
W20260709 13:01:40.049607  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.882s	user 0.025s	sys 0.063s
I20260709 13:01:40.058049  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.058779  1587 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:40.059801  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.060508  1587 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:39.866312  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 573062us
I20260709 13:01:40.053622  1586 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:40.062338  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:39.886051  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.000s	sys 0.029s
W20260709 13:01:40.063587  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.000s	sys 0.034s
W20260709 13:01:40.064522  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 829503us
I20260709 13:01:39.696050  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1841) took 1849 ms. Trace:
W20260709 13:01:40.055583  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.069129  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.069943  1592 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:01:40.071194  2281 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.634s (SENT) [suppressed 8 similar messages]
I20260709 13:01:40.064713  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1241) took 1490 ms. Trace:
I20260709 13:01:39.605963  3284 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.049083  1590 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.053371  1591 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:40.079623  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:40.068442  1663 rpcz_store.cc:276] 0709 13:01:37.846187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.230849 (+1384662us) service_pool.cc:224] Handling call
0709 13:01:39.245826 (+ 14977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.426774 (+180948us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.426828 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:39.426893 (+    65us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:39.426944 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:39.695742 (+268798us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400010c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:39.695800 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:39.695965 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397094272}
I20260709 13:01:40.102281  2312 raft_consensus.cc:2368] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
I20260709 13:01:39.608690  3341 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.588585  3440 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:40.072022  1735 rpcz_store.cc:276] 0709 13:01:38.576241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.061857 (+485616us) service_pool.cc:224] Handling call
0709 13:01:39.099081 (+ 37224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.186691 (+ 87610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.063725 (+877034us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:40.064648 (+   923us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6041344}
W20260709 13:01:40.035005  3392 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.843s	user 0.014s	sys 0.000s
I20260709 13:01:40.104552  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:40.105880  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:40.106055  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.128767  2314 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:01:40.129240  1668 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:39.594974  3274 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.106938  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.100210  3284 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:40.107596  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:39.691574  3429 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.140406  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:40.141073  1735 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:39.621691  3375 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.109881  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:40.110790  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49001.5us
I20260709 13:01:40.111235  3359 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.111595  3412 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.114118  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:40.114506  2311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 486) took 1529 ms (client timeout 1820 ms). Trace:
I20260709 13:01:40.121984  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.042745  3411 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:40.066826  3338 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:40.027365  3387 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.771s	user 0.005s	sys 0.008s
I20260709 13:01:40.103420  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.149909  1663 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:39.592976  3262 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.133042  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:39.941363  3280 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.936473  3426 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:39.668982  3430 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.141088  3284 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.108937  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:40.141269  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 500713us
I20260709 13:01:40.143532  3442 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:39.624497  3437 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:40.150153  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.150318  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:39.667896  3423 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.474s
I20260709 13:01:40.153131  2313 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:39.987125  3435 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.192521  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:40.193114  1664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.006s	sys 0.038s
I20260709 13:01:40.148974  3411 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.154258  3387 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:40.135864  1585 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.127806  3440 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.158397  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:40.193526  1664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.006s	sys 0.038s
I20260709 13:01:40.167227  1739 raft_consensus.cc:2368] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:39.654966  3350 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:39.696041  3410 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.144654  1741 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.147092  3412 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.182775  3419 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.186098  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:40.192324  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:40.170249  3432 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.001s
W20260709 13:01:40.171696  3421 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.795s	user 0.003s	sys 0.011s
I20260709 13:01:39.817104  3427 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.193321  1668 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:40.217975  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.139441  3392 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:39.747020  3408 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.173333  3430 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:40.149736  2311 rpcz_store.cc:269] 0709 13:01:38.584832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.588054 (+  3222us) service_pool.cc:224] Handling call
0709 13:01:40.114282 (+1526228us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400573ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:40.114435 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":675541888}
I20260709 13:01:40.192380  3437 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.171547  3310 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.208559  1666 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
W20260709 13:01:40.195967  3327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.007s	sys 0.024s
I20260709 13:01:40.205886  3435 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.214123  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:40.154624  3294 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:01:40.224771  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.293586  1741 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.233904  3430 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.238430  3274 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.254015  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1840) took 2408 ms. Trace:
I20260709 13:01:40.195776  3411 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.201664  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.301985  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:40.207346  3412 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:40.308708  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.952s	user 0.016s	sys 0.039s
W20260709 13:01:40.304920  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 109060us
W20260709 13:01:40.274124  3327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.007s	sys 0.024s
I20260709 13:01:40.346074  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:40.357811  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.358590  1666 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.320711  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.211752  3187 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.308907  1660 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:40.204842  3359 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260709 13:01:40.368387  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.305850  1667 rpcz_store.cc:276] 0709 13:01:37.845567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.224760 (+1379193us) service_pool.cc:224] Handling call
0709 13:01:39.225437 (+   677us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.580485 (+355048us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.253578 (+673093us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b540013d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:40.253901 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":677555968}
I20260709 13:01:40.375773  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.361299  1668 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:40.378720  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.379803  1668 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:40.382102  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.352910  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 217) took 1227 ms. Trace:
W20260709 13:01:40.306118  3268 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.012s	sys 0.041s
I20260709 13:01:40.274027  3310 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:40.330885  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 129270us
I20260709 13:01:40.390795  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:40.453558  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62938.9us
I20260709 13:01:40.458185  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.303654  3435 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.374439  1741 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:01:40.219455  3432 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.219395  3421 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.364557  3187 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:40.347921  3448 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:01:40.386826  3268 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.013s	sys 0.041s
I20260709 13:01:40.393641  1593 rpcz_store.cc:276] 0709 13:01:39.074812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.077938 (+  3126us) service_pool.cc:224] Handling call
0709 13:01:40.302293 (+1224355us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:40.302480 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":445642496}
W20260709 13:01:40.355340  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.999s	user 0.017s	sys 0.044s
W20260709 13:01:40.354775  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141081us
I20260709 13:01:40.392526  3372 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:40.384044  3442 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.351121  3452 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.469717  3387 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:40.461148  3435 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.377403  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:40.459712  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:40.385581  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.100s	user 0.038s	sys 0.072s
W20260709 13:01:40.486661  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.201s	user 0.038s	sys 0.072s
I20260709 13:01:40.467130  3187 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:40.468225  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.20525e+06us
I20260709 13:01:40.468809  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:40.465575  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 213) took 1426 ms. Trace:
I20260709 13:01:40.491299  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:40.492285  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.446419  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1238) took 1876 ms. Trace:
I20260709 13:01:40.470568  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:40.466413  3316 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.816s	user 0.012s	sys 0.002s
W20260709 13:01:40.480109  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1230) took 1916 ms (client timeout 1651 ms). Trace:
I20260709 13:01:40.479743  3442 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.464514  3419 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:40.399996  3280 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
W20260709 13:01:40.471194  3336 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.824s	user 0.016s	sys 0.000s
W20260709 13:01:40.489634  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.392750  3310 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.452797  3294 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:40.499068  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:40.528080  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:40.499907  1741 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:40.482656  3271 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.481423  3452 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:40.497334  1734 rpcz_store.cc:269] 0709 13:01:38.554300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.561116 (+  6816us) service_pool.cc:224] Handling call
0709 13:01:40.470193 (+1909077us) spinlock_profiling.cc:243] Waited 597 ms on lock 0x7b54003516d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:40.470350 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1372400512}
I20260709 13:01:40.493537  1733 rpcz_store.cc:276] 0709 13:01:38.570063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.023710 (+453647us) service_pool.cc:224] Handling call
0709 13:01:39.027517 (+  3807us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:39.082572 (+ 55055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.082618 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:39.121104 (+ 38486us) log.cc:844] Serialized 26 byte log entry
0709 13:01:40.052452 (+931348us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:40.052498 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.052575 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:40.445945 (+393370us) raft_consensus.cc:1685] finished
0709 13:01:40.445977 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.446323 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":267477,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":127,"spinlock_wait_cycles":43136,"thread_start_us":887005,"threads_started":2,"wal-append.queue_time_us":1193308}
I20260709 13:01:40.482832  3387 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.486371  3375 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.530728  3294 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:40.531174  3421 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.523147  3336 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:40.532524  3443 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:40.504647  3442 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:40.537134  3162 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.977s	user 0.036s	sys 0.073s
I20260709 13:01:40.548532  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:40.505034  3419 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:40.486061  1667 raft_consensus.cc:2376] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2. Re-sending same reply.
I20260709 13:01:40.552670  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.549883  3294 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.545636  3271 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.561414  1734 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.550110  3392 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.543292  3387 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.563616  3443 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:40.583175  3443 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.591662  3322 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:40.556685  3448 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:40.550349  3451 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:40.539594  3452 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.561537  1668 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:40.617833  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.576872  1735 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:40.562804  3162 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.036s	sys 0.074s
I20260709 13:01:40.546375  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.578392  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.568547  3294 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.564339  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1768) took 1210 ms. Trace:
I20260709 13:01:40.600414  1734 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:40.565223  3419 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:40.619253  1668 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:40.624369  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 190) took 1500 ms. Trace:
I20260709 13:01:40.624815  1734 rpcz_store.cc:276] 0709 13:01:39.123651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.548039 (+1424388us) service_pool.cc:224] Handling call
0709 13:01:40.624279 (+ 76240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.625264  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 192) took 1501 ms. Trace:
I20260709 13:01:40.573761  3271 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.500821  3316 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.625906  3271 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.625957  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.626394  1665 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:40.628367  3316 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.630849  1735 rpcz_store.cc:276] 0709 13:01:39.124197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.131860 (+1007663us) service_pool.cc:224] Handling call
0709 13:01:40.625201 (+493341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.632310  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.633291  1735 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.634024  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.635057  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1776) took 1169 ms. Trace:
I20260709 13:01:40.635697  1741 rpcz_store.cc:276] 0709 13:01:39.465637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.545869 (+1080232us) service_pool.cc:224] Handling call
0709 13:01:40.634833 (+ 88964us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.637004  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.637594  1733 rpcz_store.cc:276] 0709 13:01:39.353613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.549621 (+1196008us) service_pool.cc:224] Handling call
0709 13:01:40.564251 (+ 14630us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.641862  1667 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:40.481313  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.600061  3448 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:40.606621  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 114425us
I20260709 13:01:40.634119  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1781) took 1062 ms. Trace:
W20260709 13:01:40.647758  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 166540us
W20260709 13:01:40.651750  2317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.000s	sys 0.016s
W20260709 13:01:40.656800  2317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.000s	sys 0.018s
I20260709 13:01:40.652164  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:40.651810  1632 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.006s	sys 0.011s
W20260709 13:01:40.659158  1632 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.006s	sys 0.011s
W20260709 13:01:40.651357  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 305579us
W20260709 13:01:40.659583  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 528187us
W20260709 13:01:40.659777  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.654803  1735 rpcz_store.cc:276] 0709 13:01:39.572007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.631936 (+1059929us) service_pool.cc:224] Handling call
0709 13:01:40.634042 (+  2106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.663280  1666 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:40.667130  1733 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:40.662273  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:40.671492  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:40.493911  1584 rpcz_store.cc:276] 0709 13:01:39.038046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.038206 (+   160us) service_pool.cc:224] Handling call
0709 13:01:40.350579 (+1312373us) spinlock_profiling.cc:243] Waited 8.38 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8f8c 00007fcdeaeb8ef6 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:40.364576 (+ 13997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:40.392385 (+ 27809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:40.400335 (+  7950us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:40.402791 (+  2456us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:40.430756 (+ 27965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.442209 (+ 11453us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.464060 (+ 21851us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19284736}
I20260709 13:01:40.675016  1632 leader_election.cc:400] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
I20260709 13:01:40.651424  3316 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:40.676352  3336 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.699090  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.740051  1735 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.654503  3448 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.740931  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1772) took 1347 ms. Trace:
W20260709 13:01:40.724908  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 233721us
I20260709 13:01:40.742501  1735 rpcz_store.cc:276] 0709 13:01:39.393739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.662885 (+1269146us) service_pool.cc:224] Handling call
0709 13:01:40.740852 (+ 77967us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.676605  3421 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:40.698515  3271 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.644882  3451 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:40.695757  3427 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.734747  1632 leader_election.cc:403] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:40.737042  2316 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:40.678004  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:40.724220  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 196261us
I20260709 13:01:40.724236  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.725126  3327 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:40.750998  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 76151.6us
I20260709 13:01:40.729983  3432 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.768810  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:40.751550  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:40.751637  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1782) took 1161 ms. Trace:
W20260709 13:01:40.754935  3454 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 13:01:40.747749  3421 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:40.756137  3427 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.741240  3336 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.786306  3387 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.787109  3336 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.787704  3290 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.788780  3290 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:40.766283  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1237) took 2196 ms. Trace:
I20260709 13:01:40.795909  1736 rpcz_store.cc:276] 0709 13:01:38.569756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.022735 (+452979us) service_pool.cc:224] Handling call
0709 13:01:39.027517 (+  4782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:39.638177 (+610660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.765910 (+1127733us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b5400404a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:40.766194 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175068544}
I20260709 13:01:40.770550  1733 rpcz_store.cc:276] 0709 13:01:39.590229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.674720 (+1084491us) service_pool.cc:224] Handling call
0709 13:01:40.751456 (+ 76736us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:40.771878  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83650.1us
I20260709 13:01:40.767982  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.740437  3316 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:40.782300  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 183) took 1702 ms. Trace:
I20260709 13:01:40.766187  3451 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:40.805022  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53620.1us
I20260709 13:01:40.758924  3392 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
I20260709 13:01:40.791368  3427 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:40.791253  3294 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.831749  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:40.792940  3458 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:01:40.794612  3421 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.805006  3396 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.846043  3396 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:40.846485  3396 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.794095  3310 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.795132  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1844) took 2945 ms. Trace:
I20260709 13:01:40.868765  1664 rpcz_store.cc:276] 0709 13:01:37.849884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.321186 (+1471302us) service_pool.cc:224] Handling call
0709 13:01:39.326500 (+  5314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:39.378153 (+ 51653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.378201 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:39.396011 (+ 17810us) log.cc:844] Serialized 26 byte log entry
0709 13:01:40.198737 (+802726us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:40.198915 (+   178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.727621 (+528706us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b5400012a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:40.727672 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:40.794815 (+ 67143us) raft_consensus.cc:1685] finished
0709 13:01:40.794837 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.795070 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108770048,"thread_start_us":745735,"threads_started":1,"wal-append.queue_time_us":1149577}
I20260709 13:01:40.802471  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.870682  1593 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.870780  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1850) took 1841 ms. Trace:
I20260709 13:01:40.801052  2316 rpcz_store.cc:276] 0709 13:01:39.076618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.077859 (+  1241us) service_pool.cc:224] Handling call
0709 13:01:40.727476 (+1649617us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:40.727729 (+   253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:40.766890 (+ 39161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.778577 (+ 11687us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8567552}
I20260709 13:01:40.804526  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:40.805058  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 104180us
I20260709 13:01:40.871184  1661 rpcz_store.cc:276] 0709 13:01:39.029670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.641679 (+612009us) service_pool.cc:224] Handling call
0709 13:01:39.642043 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:40.130821 (+488778us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:40.130869 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:40.130931 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:40.130979 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.870495 (+739516us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:40.870546 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.870715 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1123619456}
I20260709 13:01:40.783231  3271 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.872640  3271 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:01:40.874303  3310 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.875034  3310 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.875422  3372 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:40.875504  3310 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.876039  3372 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.806756  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.807575  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.877952  1733 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.747498  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 194) took 2164 ms. Trace:
I20260709 13:01:40.810796  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.879179  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.879684  1590 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:40.880107  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1765) took 1712 ms. Trace:
I20260709 13:01:40.879999  3419 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.808107  3162 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:40.808563  3432 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.811463  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.880573  1733 rpcz_store.cc:276] 0709 13:01:39.168258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.878857 (+1710599us) service_pool.cc:224] Handling call
0709 13:01:40.880040 (+  1183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:40.881284  1589 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.882716  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.884196  2283 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:40.884630  2283 leader_election.cc:403] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:40.789270  3387 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.813629  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.902529  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.903820  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.779541  3327 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.817845  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 179) took 1767 ms. Trace:
I20260709 13:01:40.905121  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.907528  1735 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:40.907903  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.821324  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 185) took 1631 ms. Trace:
I20260709 13:01:40.824491  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.910296  1588 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:40.825145  3187 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.826206  3411 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:40.826851  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 1.33722e+06us
W20260709 13:01:40.827687  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1239) took 2257 ms (client timeout 1590 ms). Trace:
W20260709 13:01:40.816305  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47605.2us
I20260709 13:01:40.828570  3384 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:40.803838  3284 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.620s	user 0.008s	sys 0.005s
I20260709 13:01:40.779793  3410 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:40.922837  1706 leader_election.cc:400] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:40.923302  1706 leader_election.cc:403] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:40.805225  3316 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:40.855958  3294 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:40.864224  3412 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.508s	user 0.011s	sys 0.002s
W20260709 13:01:40.868217  3430 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.007s
I20260709 13:01:40.805554  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:40.899607  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.909353  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.830477  3467 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:40.912148  1738 rpcz_store.cc:269] 0709 13:01:38.570356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.576406 (+  6050us) service_pool.cc:224] Handling call
0709 13:01:40.827446 (+2251040us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:40.827567 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1010120576}
I20260709 13:01:40.909950  2317 rpcz_store.cc:276] 0709 13:01:39.189898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.204388 (+ 14490us) service_pool.cc:224] Handling call
0709 13:01:39.204887 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:40.055112 (+850225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:40.055146 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:40.055656 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 13:01:40.657330 (+601674us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:40.657383 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.657471 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:40.821096 (+163625us) raft_consensus.cc:1685] finished
0709 13:01:40.821117 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.821261 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":181230,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":9293,"thread_start_us":1450323,"threads_started":2,"wal-append.queue_time_us":691814}
W20260709 13:01:40.903846  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 105614us
I20260709 13:01:40.919588  3310 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.940502  3448 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.832741  1592 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:40.936275  1733 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.926151  3410 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:40.941934  3410 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.942509  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.943204  1592 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:40.944125  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.911801  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.945194  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.947628  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.934763  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:40.825367  3442 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.899379  3432 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:40.946754  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.947640  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:40.957237  3472 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:40.969134  2317 raft_consensus.cc:2368] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:40.969763  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:40.969466  3472 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.971009  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.971664  1733 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:40.972291  3472 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:40.972544  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:40.946514  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:40.920868  3405 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:40.979225  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.980489  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:40.982023  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:40.957631  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.977304  1733 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:40.963593  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:40.923573  3162 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:40.931767  3437 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.620s	user 0.006s	sys 0.006s
I20260709 13:01:40.989154  1666 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:40.934978  3339 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:40.998319  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.922991  3375 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 13:01:40.985327  1593 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:41.001883  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 187) took 1721 ms. Trace:
I20260709 13:01:40.988087  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.983649  1661 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:40.920008  2310 rpcz_store.cc:276] 0709 13:01:39.050466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.056974 (+  6508us) service_pool.cc:224] Handling call
0709 13:01:40.817642 (+1760668us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:40.817773 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":520716160}
I20260709 13:01:40.992434  3372 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.044010  3477 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.044546  3477 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:41.044984  3477 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.990820  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:40.992256  3345 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:40.990824  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.049777  1667 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:40.929781  3427 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:40.977332  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:40.994975  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:40.906777  3259 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:40.999136  1734 raft_consensus.cc:2368] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:40.994827  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:41.074397  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79681us
I20260709 13:01:40.920336  3435 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.975256  3432 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:41.002089  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1763) took 1872 ms (client timeout 1552 ms). Trace:
I20260709 13:01:41.072149  1592 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:40.987092  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:41.079301  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:41.079689  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92940.9us
I20260709 13:01:41.080060  1592 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:41.080796  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.081269  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:41.003654  2313 rpcz_store.cc:276] 0709 13:01:39.281070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.281402 (+   332us) service_pool.cc:224] Handling call
0709 13:01:39.281915 (+   513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:40.193731 (+911816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.001620 (+807889us) spinlock_profiling.cc:243] Waited 871 ms on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:41.001820 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2002966784}
I20260709 13:01:41.082929  3372 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:41.017045  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:41.018723  1740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1243) took 2441 ms (client timeout 1831 ms). Trace:
I20260709 13:01:40.928828  3423 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.064s
I20260709 13:01:41.016248  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.028002  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1848) took 2420 ms. Trace:
I20260709 13:01:41.028556  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.968266  3467 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:40.958143  3430 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.985489  3290 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:40.958362  3412 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:40.992826  3465 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:41.033674  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 186) took 1911 ms (client timeout 1713 ms). Trace:
I20260709 13:01:40.878842  1659 rpcz_store.cc:276] 0709 13:01:38.583515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.561391 (+977876us) service_pool.cc:224] Handling call
0709 13:01:39.619750 (+ 58359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:39.766185 (+146435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:39.809121 (+ 42936us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:39.935067 (+125946us) log.cc:844] Serialized 26 byte log entry
0709 13:01:40.370798 (+435731us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:40.383715 (+ 12917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:40.392295 (+  8580us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:40.747034 (+354739us) raft_consensus.cc:1685] finished
0709 13:01:40.747070 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:40.747403 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7927,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":132,"thread_start_us":302195,"threads_started":1,"wal-append.queue_time_us":594030}
W20260709 13:01:41.035037  3440 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.815s	user 0.013s	sys 0.000s
I20260709 13:01:41.035739  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.036448  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:40.989754  3413 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:41.039291  1586 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:41.040313  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.040829  3327 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.993462  3294 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:01:41.015528  3405 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.020638  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.050683  1735 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:40.944335  3445 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260709 13:01:40.925861  3384 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.055732  3345 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:41.059183  1661 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:40.926067  3284 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.061708  3187 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:40.999624  3437 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.063751  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.083649  1589 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:41.208746  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.110517  3375 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.098729  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.075009  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.215255  1741 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:41.214953  3375 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.215391  1667 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:41.216378  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.216912  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:41.077616  1738 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
W20260709 13:01:41.162142  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.217725  1593 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:41.217819  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1767) took 1866 ms. Trace:
I20260709 13:01:41.218111  3430 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:41.218261  1738 rpcz_store.cc:276] 0709 13:01:39.351436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.992069 (+1640633us) service_pool.cc:224] Handling call
0709 13:01:41.217760 (+225691us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.219368  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.219872  1664 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:41.220422  3405 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.217217  3375 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:41.065737  1736 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:41.223784  1590 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:41.222744  3470 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:41.224151  3284 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.224521  3470 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:41.224720  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.223140  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.226002  1666 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:41.225919  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.226454  3290 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:41.226383  1739 rpcz_store.cc:269] 0709 13:01:39.122071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.125090 (+  3019us) service_pool.cc:224] Handling call
0709 13:01:41.016996 (+1891906us) spinlock_profiling.cc:243] Waited 946 ms on lock 0x7b5400435654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.033567 (+ 16571us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2176893184}
I20260709 13:01:41.227082  3290 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.214635  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.227195  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.214635  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.225589  3470 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.218700  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.227267  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.220906  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.225036  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.231810  1587 raft_consensus.cc:2368] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:41.145638  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.229831  1667 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:41.230387  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:41.133809  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52781.8us
I20260709 13:01:41.230975  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.228034  3375 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:41.225934  3413 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:41.230427  3470 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.064927  1588 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:41.242110  3440 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.243364  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.244207  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.244985  1736 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:41.245383  3467 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.096552  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.245927  3413 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.245637  3375 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.246263  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:41.306123  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1777) took 1828 ms. Trace:
I20260709 13:01:41.306537  1733 rpcz_store.cc:276] 0709 13:01:39.478227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.015106 (+1536879us) service_pool.cc:224] Handling call
0709 13:01:41.306034 (+290928us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.245941  3345 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:41.246716  1737 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:41.247500  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.312829  1667 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:41.248904  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.231217  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:41.078361  1742 rpcz_store.cc:269] 0709 13:01:39.129594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.221600 (+ 92006us) service_pool.cc:224] Handling call
0709 13:01:41.001886 (+1780286us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003520d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.002013 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624002432}
I20260709 13:01:41.314284  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:41.252615  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:41.253769  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 173062us
W20260709 13:01:41.226236  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 186012us
I20260709 13:01:41.227824  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.252677  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.221158  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.320698  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.083707  3372 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:41.288019  1735 raft_consensus.cc:2368] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:41.394863  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.395539  1735 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.288831  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.227783  1589 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:41.290297  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.232091  1586 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:41.243371  1660 rpcz_store.cc:276] 0709 13:01:38.607458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.621099 (+1013641us) service_pool.cc:224] Handling call
0709 13:01:39.621481 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:40.364824 (+743343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.024329 (+659505us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b5400014ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:41.027906 (+  3577us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1580084608}
I20260709 13:01:41.400532  1660 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:01:41.295320  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:41.249589  1740 rpcz_store.cc:269] 0709 13:01:38.576901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.923117 (+346216us) service_pool.cc:224] Handling call
0709 13:01:41.018511 (+2095394us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400404f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.018628 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1169749248}
I20260709 13:01:41.404582  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.405439  1740 raft_consensus.cc:2376] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4. Re-sending same reply.
I20260709 13:01:41.299633  1592 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:41.094578  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.047667  3473 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.310321  1632 leader_election.cc:400] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:41.408063  1632 leader_election.cc:403] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:41.239122  2310 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.313470  1668 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:41.319103  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.346097  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:41.411456  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 92440.4us
I20260709 13:01:41.293119  3354 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:41.412209  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.033211  3452 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.241228  3412 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.326392  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.391481  1584 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:41.379540  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.380386  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.381855  3440 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.384859  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.397953  3314 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.434573  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1788) took 1216 ms. Trace:
I20260709 13:01:41.311486  3375 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:41.434957  3448 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.302153  3187 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.049350  3339 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.390791  1593 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:41.317754  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.319383  3284 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:41.411813  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 113585us
W20260709 13:01:41.433633  3396 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.587s	user 0.006s	sys 0.006s
I20260709 13:01:41.413332  1585 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:41.392760  1659 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:41.389364  3310 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.418114  1667 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:41.293787  3437 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.431481  1739 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:41.435878  2313 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 13:01:41.436446  1666 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:41.434999  1742 rpcz_store.cc:276] 0709 13:01:40.217956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.318383 (+1100427us) service_pool.cc:224] Handling call
0709 13:01:41.434474 (+116091us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.438225  3442 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:41.410573  3473 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:41.443624  1660 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:41.416287  3387 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
W20260709 13:01:41.451265  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 133614us
W20260709 13:01:41.451392  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 362788us
I20260709 13:01:41.451699  1738 raft_consensus.cc:2376] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3. Re-sending same reply.
I20260709 13:01:41.454495  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:41.352345  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.458554  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.472033  1664 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:41.459300  1735 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:41.467139  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.476027  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1764) took 2344 ms. Trace:
I20260709 13:01:41.476454  1735 rpcz_store.cc:276] 0709 13:01:39.131599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.475269 (+2343670us) service_pool.cc:224] Handling call
0709 13:01:41.475560 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.475745 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.475767 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.475808 (+    41us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:41.475838 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.475885 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.475972 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.479736  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.471925  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.480387  2314 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:41.484165  1736 raft_consensus.cc:2410] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:01:41.472057  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.450996  3187 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.494951  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1773) took 2089 ms. Trace:
I20260709 13:01:41.495366  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1774) took 2064 ms. Trace:
I20260709 13:01:41.493324  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:41.488963  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.489125  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.495383  1739 rpcz_store.cc:276] 0709 13:01:39.405276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.494077 (+2088801us) service_pool.cc:224] Handling call
0709 13:01:41.494426 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.494625 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.494650 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.494697 (+    47us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:41.494730 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.494789 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.494892 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:41.504674  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1839) took 3646 ms (client timeout 3256 ms). Trace:
W20260709 13:01:41.496882  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0853174189999777s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
I20260709 13:01:41.495817  1736 rpcz_store.cc:276] 0709 13:01:39.430826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.494639 (+2063813us) service_pool.cc:224] Handling call
0709 13:01:41.494894 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.495072 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.495097 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.495136 (+    39us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:41.495168 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.495215 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.495310 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:41.522293  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0277174189999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1831s for a response
W20260709 13:01:41.523028  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110921us
I20260709 13:01:41.523136  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.523955  1585 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:41.395961  3162 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.479432  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:41.490962  3410 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.002s
W20260709 13:01:41.579986  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 56669.9us
I20260709 13:01:41.603487  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.529198  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 191) took 2405 ms. Trace:
I20260709 13:01:41.591619  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.547478  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.613966  1742 rpcz_store.cc:276] 0709 13:01:39.123921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.470558 (+2346637us) service_pool.cc:224] Handling call
0709 13:01:41.528493 (+ 57935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.528731 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.528775 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.528841 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:41.528894 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.528971 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.529117 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.479270  3423 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:41.526371  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:41.612234  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1775) took 2162 ms. Trace:
I20260709 13:01:41.523952  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:41.505321  1662 rpcz_store.cc:269] 0709 13:01:37.844903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.041221 (+1196318us) service_pool.cc:224] Handling call
0709 13:01:41.491270 (+2450049us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.491435 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":963513088}
I20260709 13:01:41.616767  1739 rpcz_store.cc:276] 0709 13:01:39.449781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.611294 (+2161513us) service_pool.cc:224] Handling call
0709 13:01:41.611693 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.611885 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.611907 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.611944 (+    37us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:41.611980 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.612034 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.612170 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.623553  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1785) took 1496 ms. Trace:
I20260709 13:01:41.624191  1739 rpcz_store.cc:276] 0709 13:01:40.127382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.622373 (+1494991us) service_pool.cc:224] Handling call
0709 13:01:41.622826 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.623103 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.623141 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.623204 (+    63us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:41.623256 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.623330 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.623474 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.626832  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1786) took 1461 ms. Trace:
I20260709 13:01:41.493917  3372 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:41.604645  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1770) took 2232 ms. Trace:
I20260709 13:01:41.628134  1739 rpcz_store.cc:276] 0709 13:01:40.165777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.625706 (+1459929us) service_pool.cc:224] Handling call
0709 13:01:41.626139 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.626401 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.626438 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.626498 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:41.626546 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.626620 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.626758 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:41.610698  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48756.9us
I20260709 13:01:41.631619  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:41.633270  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 109426us
W20260709 13:01:41.584121  3479 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 13:01:41.627592  2313 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.499219  3187 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.483309  3430 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.479693  3473 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:41.584931  3316 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.619s	user 0.005s	sys 0.005s
W20260709 13:01:41.588943  3421 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.748s	user 0.010s	sys 0.001s
I20260709 13:01:41.658254  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.658980  1590 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:41.644891  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:41.493762  3472 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:41.529433  3387 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:41.461028  3345 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:41.634776  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.631222  1735 rpcz_store.cc:276] 0709 13:01:39.372204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.477583 (+2105379us) service_pool.cc:224] Handling call
0709 13:01:41.603676 (+126093us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.603955 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.603992 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:41.604061 (+    69us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:41.604113 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.604367 (+   254us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:41.604542 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.499135  3448 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:41.629227  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 142192us
I20260709 13:01:41.628692  3410 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.632894  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.520624  3437 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.585860  3435 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.520536  3442 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.480823  3310 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.651371  3440 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:01:41.653826  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 164800us
I20260709 13:01:41.470649  3396 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.655521  1661 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:41.534143  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.619709  3423 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:41.745244  3423 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.752346  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:41.657141  3483 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.097s
W20260709 13:01:41.754864  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1778) took 2240 ms (client timeout 1769 ms). Trace:
I20260709 13:01:41.755589  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:41.755520  1734 rpcz_store.cc:269] 0709 13:01:39.521038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.625613 (+1104575us) service_pool.cc:224] Handling call
0709 13:01:41.754568 (+1128955us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b5400435b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.754762 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":847913216}
I20260709 13:01:41.676839  3347 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:41.759469  3443 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.759858  3347 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:41.759905  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1766) took 2567 ms (client timeout 1709 ms). Trace:
I20260709 13:01:41.760938  3347 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.761574  1740 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:41.667316  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:41.762229  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:41.762665  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 95426.2us
I20260709 13:01:41.763284  1584 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.764549  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:41.640089  1742 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:41.670835  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.671500  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.766074  1590 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:41.766239  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.673060  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.767114  1739 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
W20260709 13:01:41.760571  1737 rpcz_store.cc:269] 0709 13:01:39.192525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:39.488681 (+296156us) service_pool.cc:224] Handling call
0709 13:01:41.759589 (+2270908us) spinlock_profiling.cc:243] Waited 1.71 s on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.759802 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3925726080}
I20260709 13:01:41.768255  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.671118  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.643486  3482 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:41.685959  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:41.771638  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 100894us
I20260709 13:01:41.679920  3401 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 13:01:41.771384  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.683904  3314 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 13:01:41.779354  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.731175  3410 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.698889  3339 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.732154  3490 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:41.779633  3484 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.787683  3392 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:41.685981  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.787683  3484 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:41.780443  3357 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:41.788403  3484 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.788372  3392 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:41.669204  3452 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:41.733848  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49077us
I20260709 13:01:41.731196  3310 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.738319  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:41.795737  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:41.739168  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:41.797644  1586 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:41.737226  2310 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:41.740281  1589 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:41.698176  3384 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.684245  3442 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.674500  3287 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:01:41.677003  3427 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:41.665092  3421 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.661494  3316 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.668116  3412 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:41.767212  1591 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:01:41.763235  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 188) took 2640 ms. Trace:
I20260709 13:01:41.481089  3284 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.831658  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:41.769671  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98624.6us
I20260709 13:01:41.644142  3430 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.835379  3430 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.771214  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1250) took 2104 ms. Trace:
I20260709 13:01:41.845784  1742 rpcz_store.cc:276] 0709 13:01:39.666389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.628949 (+1962560us) service_pool.cc:224] Handling call
0709 13:01:41.636279 (+  7330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.770885 (+134606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.771152 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:41.847546  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.848448  1742 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:41.771844  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:41.730772  3396 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.776899  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:41.861912  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 85102.2us
I20260709 13:01:41.861775  3392 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.862746  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.863662  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:41.864794  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 127094us
I20260709 13:01:41.815876  3451 raft_consensus.cc:2764] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 13:01:41.869894  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:41.787395  3490 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.676343  3448 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:41.664517  3387 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:41.792287  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.880549  1667 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:41.881374  1737 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:41.795369  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:41.883949  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88578.4us
I20260709 13:01:41.737619  1593 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:01:41.772102  3482 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:41.759927  3443 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:41.676469  3437 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.780650  3483 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:41.819861  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1853) took 1775 ms. Trace:
I20260709 13:01:41.822873  1588 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:41.905678  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.951660  1588 raft_consensus.cc:606] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Received request to transfer leadership to TS ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:41.952170  1588 raft_consensus.cc:614] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:41.788751  3357 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:41.828373  1740 rpcz_store.cc:276] 0709 13:01:39.123033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.413767 (+2290734us) service_pool.cc:224] Handling call
0709 13:01:41.761279 (+347512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.762200 (+   921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.763041 (+   841us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54000e2c50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.763175 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":798365440}
I20260709 13:01:41.830725  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.757344  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:41.855577  3432 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.555s	user 0.014s	sys 0.001s
W20260709 13:01:41.855922  3290 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.551s	user 0.000s	sys 0.010s
W20260709 13:01:41.856716  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.953301  1663 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:41.954355  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:41.954807  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:41.858047  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 194) took 1152 ms. Trace:
W20260709 13:01:41.860517  3477 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.815s	user 0.009s	sys 0.003s
I20260709 13:01:41.676851  3472 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:41.815150  3440 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:41.865948  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94226.2us
I20260709 13:01:41.900345  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:41.918218  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.864830  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.866953  1739 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:01:41.865262  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1769) took 2506 ms (client timeout 1994 ms). Trace:
I20260709 13:01:41.830305  3427 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:41.892271  3387 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:41.899279  3443 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.990288  3448 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.896906  3482 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:41.995122  1742 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:41.821607  3372 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:41.955613  1665 rpcz_store.cc:276] 0709 13:01:40.044737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.044932 (+   195us) service_pool.cc:224] Handling call
0709 13:01:41.819093 (+1774161us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.819796 (+   703us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101782272}
I20260709 13:01:42.000067  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 1792) took 1196 ms. Trace:
I20260709 13:01:41.933943  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:41.871068  3411 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.955745  1738 rpcz_store.cc:276] 0709 13:01:40.706004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.219060 (+513056us) service_pool.cc:224] Handling call
0709 13:01:41.857751 (+638691us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.857874 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532668160}
I20260709 13:01:42.006594  1663 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:41.879736  3359 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:41.869326  3492 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.893674  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:41.967317  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102563us
I20260709 13:01:41.909086  2313 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:41.971845  3427 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.973282  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.979017  3316 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:41.962519  3290 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:41.987974  2310 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:01:42.000460  3259 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.731s	user 0.013s	sys 0.000s
I20260709 13:01:41.983335  3442 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:41.959757  3357 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:41.985612  3338 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:42.026927  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:41.988235  3284 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:41.827543  3412 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:41.995869  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:41.892874  3339 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.000531  1587 rpcz_store.cc:276] 0709 13:01:40.804103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.804288 (+   185us) service_pool.cc:224] Handling call
0709 13:01:41.999846 (+1195558us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540031c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.999993 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":827269376}
I20260709 13:01:42.001279  1660 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:41.892549  3455 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:41.895836  3396 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:01:42.005182  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 172220us
I20260709 13:01:41.951886  3483 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:41.965735  3477 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:41.964960  1741 rpcz_store.cc:269] 0709 13:01:39.358984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:40.636626 (+1277642us) service_pool.cc:224] Handling call
0709 13:01:41.864888 (+1228262us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54001095d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:41.865157 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1313568512}
I20260709 13:01:42.009277  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:41.891453  3487 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.059962  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:42.012168  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:42.015079  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 121613us
W20260709 13:01:42.061111  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49080.7us
I20260709 13:01:42.026176  3359 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:41.996186  3472 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:42.027437  2311 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:42.073411  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:01:42.027789  3259 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:41.963405  3432 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:42.015030  3413 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.626s	user 0.007s	sys 0.005s
I20260709 13:01:41.987891  3421 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.041405  1742 raft_consensus.cc:2368] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
W20260709 13:01:42.048302  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.058990  3487 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.007074  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.094815  1738 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:42.096180  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.023422  3280 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:42.057461  1661 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:42.015517  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:41.899812  3489 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.152s
I20260709 13:01:42.063580  1706 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.025966  3310 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:42.064229  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.065013  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.042865  3338 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.071009  3359 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.027472  3372 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.052280  3412 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.051275  3339 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.074110  1664 raft_consensus.cc:606] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:42.057381  3483 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.082772  1665 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:42.086585  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.167546  1740 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:42.067909  3350 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20260709 13:01:42.168826  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.087327  3354 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.052554  1634 leader_election.cc:400] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:42.169476  1740 raft_consensus.cc:2376] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4. Re-sending same reply.
I20260709 13:01:42.067742  3448 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:42.092247  3495 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:01:42.093667  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.174743  1742 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
W20260709 13:01:42.098316  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82897.4us
I20260709 13:01:42.111021  3487 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.111286  1585 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:42.115866  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52372.1us
I20260709 13:01:42.125034  3432 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:42.126065  3472 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.127830  3490 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:42.129447  3477 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.072916  3442 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.067485  3455 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:42.133025  1738 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.150085  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:42.169668  3470 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:42.159863  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95739.8us
I20260709 13:01:42.058859  3473 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.170063  1634 leader_election.cc:403] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 13:01:42.204375  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 152003us
I20260709 13:01:42.055624  3499 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:01:42.165874  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:42.109771  3488 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:42.093705  3413 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.173044  3448 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.177345  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:42.172995  3354 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:42.161713  3310 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.160197  3338 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:42.161439  3435 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.167745  3339 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.189626  1660 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:42.193476  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 143788us
I20260709 13:01:42.194748  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.232182  2317 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:42.194893  3359 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:42.199131  2313 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:42.158595  1741 raft_consensus.cc:2368] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:42.158385  3485 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:42.202733  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.239734  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.240460  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.241395  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.241737  3470 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:42.242093  3432 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:42.173650  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.242868  3432 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.243139  3470 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:42.243477  3432 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:42.243763  3470 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.244344  3470 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:42.165035  3440 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260709 13:01:42.207753  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58788.5us
W20260709 13:01:42.216863  3501 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:01:42.198513  3442 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:42.218209  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1270) took 1149 ms. Trace:
I20260709 13:01:42.252185  1668 rpcz_store.cc:276] 0709 13:01:41.068950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.750271 (+681321us) service_pool.cc:224] Handling call
0709 13:01:41.750688 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:41.750925 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:41.750950 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:41.751540 (+   590us) log.cc:844] Serialized 26 byte log entry
0709 13:01:41.818072 (+ 66532us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:41.818131 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:41.818238 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:42.217962 (+399724us) raft_consensus.cc:1685] finished
0709 13:01:42.217987 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:42.218138 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2944,"thread_start_us":66068,"threads_started":1,"wal-append.queue_time_us":154957}
W20260709 13:01:42.218465  3187 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.000s
I20260709 13:01:42.226055  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.227813  1661 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:42.206565  3490 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:42.269721  3187 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.272531  3187 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.206184  3396 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:42.235278  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:42.244215  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.165606  3356 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 13:01:42.247494  2316 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:42.289183  1588 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:42.241451  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:42.214133  3448 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:42.256374  3375 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.821s	user 0.011s	sys 0.003s
I20260709 13:01:42.282615  1740 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
W20260709 13:01:42.282734  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47616.1us
W20260709 13:01:42.287503  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44189.6us
W20260709 13:01:42.294588  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 100683us
I20260709 13:01:42.236306  3437 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.273065  3354 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.234674  3435 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.218221  3455 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.342429  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.387547  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 189) took 1295 ms. Trace:
I20260709 13:01:42.490473  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.494339  2317 rpcz_store.cc:276] 0709 13:01:41.092204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.096148 (+  3944us) service_pool.cc:224] Handling call
0709 13:01:42.387210 (+1291062us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54006e0fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.387419 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1831645440}
I20260709 13:01:42.259778  3442 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.328759  1740 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:01:42.492254  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 126008us
W20260709 13:01:42.484880  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 728068us
I20260709 13:01:42.420590  3466 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
I20260709 13:01:42.523471  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:42.351648  3356 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.173s
I20260709 13:01:42.524762  3466 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.528368  3466 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.528880  3466 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.531109  3490 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.531860  3490 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:42.532564  3356 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:42.533249  3356 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:42.535427  3490 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.415648  3396 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:42.539435  3423 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.793s	user 0.010s	sys 0.003s
I20260709 13:01:42.420681  3387 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.540021  3423 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.541301  1735 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:42.372257  3443 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.539523  1736 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:42.545305  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1249) took 2900 ms. Trace:
I20260709 13:01:42.545763  1736 rpcz_store.cc:276] 0709 13:01:39.665881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.538744 (+2872863us) service_pool.cc:224] Handling call
0709 13:01:42.539142 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.544943 (+  5801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:42.545226 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:42.418563  3484 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.004s
I20260709 13:01:42.526818  2281 leader_election.cc:400] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:42.547715  2281 leader_election.cc:403] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:42.547945  1740 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
W20260709 13:01:42.548749  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 251888us
I20260709 13:01:42.286217  3290 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.553982  1740 raft_consensus.cc:606] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:42.371793  1584 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:42.555519  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:42.295207  3327 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:42.295284  3187 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:42.534272  3356 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.567775  3187 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:42.312282  3440 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:42.381603  3496 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:42.566007  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:42.570339  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:42.460301  3473 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:42.567009  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 224277us
I20260709 13:01:42.279870  3259 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.274s
W20260709 13:01:42.571364  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:42.574414  1632 leader_election.cc:400] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:42.576319  1632 leader_election.cc:403] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:42.294716  3375 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.564543  3310 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:42.585635  3347 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:42.556777  3413 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.599380  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:42.567119  3443 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.348259  3430 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:42.595211  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 104814us
W20260709 13:01:42.570719  3482 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.578s	user 0.014s	sys 0.000s
I20260709 13:01:42.556470  3484 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.586692  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.601393  3488 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:42.601815  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:42.607380  3466 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.565356  2310 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:42.568981  3490 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.570245  3411 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:42.588945  3354 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.570402  3187 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.575814  3477 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:42.573292  3496 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:42.622219  1733 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.553117  3470 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:42.612133  3357 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.558s	user 0.011s	sys 0.003s
I20260709 13:01:42.637828  1666 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:42.621277  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.627200  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.599611  3489 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.573452  3499 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:42.628995  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:42.613654  3473 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.634225  1734 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.612622  3280 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.624676  3452 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.579406  3440 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.647670  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:42.630544  3284 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:42.634354  3413 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:42.633348  3411 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:42.623819  3482 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.640020  3387 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.653918  3280 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.654517  3473 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:42.642586  3357 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.659266  3284 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:42.661310  3521 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:42.635946  3467 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:01:42.655375  3498 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:42.671586  3473 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.655030  3518 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.678575  3375 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.636711  3513 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 13:01:42.658341  3452 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.681703  3435 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:42.682669  3518 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.682869  3521 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:42.674307  3488 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:42.693750  3513 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:42.697278  3484 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.686511  3498 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:42.684583  3496 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:42.708220  3487 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.004s
I20260709 13:01:42.718272  3482 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.718369  3487 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.718662  3496 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:42.704110  3492 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.666s	user 0.011s	sys 0.000s
I20260709 13:01:42.696285  3518 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.698694  3513 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:42.717247  3521 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:42.754959  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 179707us
I20260709 13:01:42.757849  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.788179  1659 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:42.665057  3338 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.790571  1737 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.797103  2318 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:42.669523  3280 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.685804  3411 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.804978  3411 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.806236  3435 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:42.807077  3435 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.807593  3435 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.675144  3284 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:42.811636  3419 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.681495  3423 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.823284  3440 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.823849  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 1982) took 1232 ms. Trace:
I20260709 13:01:42.826519  3487 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.827487  3492 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:42.835572  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:42.841296  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 1992) took 1105 ms. Trace:
W20260709 13:01:42.841871  3483 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.001s
I20260709 13:01:42.843039  3483 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.700686  3448 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.847011  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1873) took 1772 ms. Trace:
I20260709 13:01:42.847582  1667 rpcz_store.cc:276] 0709 13:01:41.074452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.462768 (+388316us) service_pool.cc:224] Handling call
0709 13:01:41.463232 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:41.881058 (+417826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:42.846297 (+965239us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:42.846725 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959279488}
I20260709 13:01:42.718703  3498 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.824787  1585 rpcz_store.cc:276] 0709 13:01:41.590736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.591233 (+   497us) service_pool.cc:224] Handling call
0709 13:01:42.822877 (+1231644us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b54003b10d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.823285 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1142193536}
I20260709 13:01:42.862339  1593 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:42.862638  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:42.820439  1488 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:01:42.863968  1664 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.841926  1590 rpcz_store.cc:276] 0709 13:01:41.735737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.766679 (+ 30942us) service_pool.cc:224] Handling call
0709 13:01:42.541851 (+775172us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:42.542091 (+   240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.763932 (+221841us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:42.764181 (+   249us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:42.764277 (+    96us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:42.764324 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:42.839963 (+ 75639us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54001def54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:42.840035 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:42.841024 (+   989us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":513182208}
W20260709 13:01:42.865094  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1806) took 1191 ms (client timeout 1524 ms). Trace:
W20260709 13:01:42.867278  1734 rpcz_store.cc:269] 0709 13:01:41.673127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.757011 (+ 83884us) service_pool.cc:224] Handling call
0709 13:01:42.857416 (+1100405us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b5400436054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.864758 (+  7342us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553680640}
I20260709 13:01:42.702457  3452 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:42.882532  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.884845  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:42.885851  3354 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:42.887828  3357 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.890908  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:42.893646  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:42.894438  3327 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:42.894886  1665 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:42.904345  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:42.906185  3524 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.183s	user 0.000s	sys 0.002s
I20260709 13:01:42.699857  3470 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:42.910650  3354 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:42.934409  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.893776  3430 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.912494  3350 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:42.941913  3350 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:42.930178  3354 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:42.721921  3513 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:42.987286  1590 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:42.907198  3383 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:43.070029  3280 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.071108  3345 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:42.942222  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:42.984200  1662 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:43.074469  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:42.951949  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:42.992208  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1808) took 1318 ms (client timeout 1670 ms). Trace:
W20260709 13:01:42.991020  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1789) took 2767 ms (client timeout 2642 ms). Trace:
I20260709 13:01:43.056042  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:42.986245  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:42.955155  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1810) took 1280 ms. Trace:
I20260709 13:01:43.083338  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.083086  3442 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.084196  1585 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.084292  3513 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:42.945590  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:42.951398  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:43.087440  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 147875us
W20260709 13:01:42.948624  3432 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.704s	user 0.005s	sys 0.010s
W20260709 13:01:42.994148  3528 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.269s	user 0.000s	sys 0.002s
W20260709 13:01:43.079372  1735 rpcz_store.cc:269] 0709 13:01:41.673806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.816257 (+142451us) service_pool.cc:224] Handling call
0709 13:01:42.991742 (+1175485us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b5400338e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.991975 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1630609792}
I20260709 13:01:43.093767  1736 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.990182  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:42.984294  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:43.081583  1737 rpcz_store.cc:276] 0709 13:01:41.674916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.954343 (+279427us) service_pool.cc:224] Handling call
0709 13:01:42.954894 (+1000551us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.955077 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1772886912}
I20260709 13:01:43.088611  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:43.080478  1733 rpcz_store.cc:269] 0709 13:01:40.223423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.307419 (+1083996us) service_pool.cc:224] Handling call
0709 13:01:42.990482 (+1683063us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400404554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:42.990771 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3006053120}
I20260709 13:01:43.087644  3353 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:42.909235  3327 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:42.984119  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.096077  1741 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
W20260709 13:01:43.096391  3531 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.100s	user 0.002s	sys 0.000s
I20260709 13:01:43.093034  1591 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:43.034839  3357 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:43.097306  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.073673  3383 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:43.099858  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:42.985052  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.055516  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:43.104722  2318 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:42.948175  3473 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.104199  3483 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:42.950037  3466 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.119372  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.134007  1588 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.134207  1592 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:01:43.123823  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 178612us
I20260709 13:01:43.124369  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.115989  3280 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:43.123937  1668 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:43.148056  3527 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:01:43.147671  3314 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:43.134508  3519 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:43.153055  1661 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:43.154322  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:43.156373  2313 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:43.137277  1667 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:43.116330  3442 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:43.101578  3432 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.160967  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.153273  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:43.133901  3339 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.852s	user 0.001s	sys 0.023s
W20260709 13:01:43.153384  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.157647  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:43.163970  3519 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:42.944610  3350 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.181980  3405 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:43.182612  3405 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:43.182864  3339 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.184638  1742 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:43.184093  3405 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.186013  3497 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:43.186467  3497 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:43.188357  1741 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.189095  3384 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:43.189519  3384 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:43.190876  3384 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.192205  3384 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:43.172322  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71924.5us
I20260709 13:01:43.187775  3497 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.189539  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.200680  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.196838  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.204282  3470 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:01:43.205904  3400 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:01:43.206169  3187 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.561s	user 0.016s	sys 0.000s
I20260709 13:01:43.206506  3400 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:43.206841  3187 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.207988  3400 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.209862  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.210635  3400 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.210937  1586 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:43.210369  3354 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.212198  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:43.212620  3427 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.175s	user 0.012s	sys 0.006s
I20260709 13:01:43.212826  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.221863  1741 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:01:43.213537  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1818) took 1535 ms (client timeout 1533 ms). Trace:
I20260709 13:01:43.175210  1584 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.175462  3448 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.173389  3327 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:43.171712  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.239077  1590 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
W20260709 13:01:43.227420  1739 rpcz_store.cc:269] 0709 13:01:41.678366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.768014 (+ 89648us) service_pool.cc:224] Handling call
0709 13:01:43.161871 (+1393857us) spinlock_profiling.cc:243] Waited 83 ms on lock 0x7b54003511d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:43.213253 (+ 51382us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":190863744}
I20260709 13:01:43.177904  3519 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:43.253664  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.928s	user 0.018s	sys 0.021s
W20260709 13:01:43.254256  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.018s	sys 0.021s
W20260709 13:01:43.254925  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101815us
I20260709 13:01:43.239849  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1280) took 1102 ms. Trace:
I20260709 13:01:43.245841  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.168223  3529 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
W20260709 13:01:43.265928  3489 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.586s	user 0.013s	sys 0.004s
I20260709 13:01:43.266700  3489 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:43.271147  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:43.273348  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.272781  1740 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.253728  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.256695  1666 rpcz_store.cc:276] 0709 13:01:41.997850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.001972 (+  4122us) service_pool.cc:224] Handling call
0709 13:01:42.002555 (+   583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.639325 (+636770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:42.994252 (+354927us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:43.100465 (+106213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1460542848}
I20260709 13:01:43.239626  3497 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:43.281895  1592 raft_consensus.cc:2368] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:43.282414  1738 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:43.284168  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.284802  3490 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.645s	user 0.017s	sys 0.000s
W20260709 13:01:43.285706  1660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.003s	sys 0.017s
I20260709 13:01:43.285583  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:43.166754  3357 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.268957  3284 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.309904  1664 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:43.303906  1737 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:43.213478  3427 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.299968  3405 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:43.314878  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.004s	sys 0.017s
I20260709 13:01:43.327929  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.331710  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 56964.8us
I20260709 13:01:43.306484  3470 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:43.326766  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.336305  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.313347  1733 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.341990  1585 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:43.312848  3490 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:43.347743  1586 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:43.350600  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.369601  1734 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:43.370831  3141 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.371447  3141 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.372895  3401 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:43.374899  3284 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.375885  3314 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:43.378001  3472 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.378895  3472 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.381119  3472 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.381649  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:43.382465  1592 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
W20260709 13:01:43.383500  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 1.18884e+06us
I20260709 13:01:43.384649  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:43.385663  3518 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.615s	user 0.014s	sys 0.000s
I20260709 13:01:43.385991  1661 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:43.386135  3518 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.388949  3518 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:43.360350  3339 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:43.389775  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57573.2us
I20260709 13:01:43.390159  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.390636  3350 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.391234  1738 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:43.391973  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.392661  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.398537  1667 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:43.393420  1738 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.402561  3432 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.406963  1666 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 1. Current term is 2. Ops: [1.3-1.3]
W20260709 13:01:43.407409  3435 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.006s
I20260709 13:01:43.407922  3435 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:43.412039  1735 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260709 13:01:43.414628  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:43.422487  3442 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:43.422986  3442 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:43.423429  3442 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:43.425937  3513 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:43.414628  3357 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.428160  3327 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:43.428644  3327 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:43.391152  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:43.390373  3339 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:43.430874  1737 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.411816  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 537) took 1984 ms. Trace:
I20260709 13:01:43.432188  1589 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:43.376000  3401 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:01:43.409220  3496 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.000s
I20260709 13:01:43.434012  2316 rpcz_store.cc:276] 0709 13:01:41.426906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.427335 (+   429us) service_pool.cc:224] Handling call
0709 13:01:41.438958 (+ 11623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.290614 (+851656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.411072 (+1120458us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:43.411492 (+   420us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1833674496}
I20260709 13:01:43.408988  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:43.417125  3338 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:43.450917  1741 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.434943  3401 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:43.454945  3401 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.456614  3401 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.446344  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.465948  1591 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:01:43.470080  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79110.4us
W20260709 13:01:43.471139  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 205764us
I20260709 13:01:43.457415  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 235) took 1180 ms. Trace:
I20260709 13:01:43.430117  3470 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:43.480271  1668 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:43.479471  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:43.481065  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.487231  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.484030  1665 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:43.485153  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.470635  3339 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:43.497740  1740 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:43.498468  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:43.495458  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 211) took 1248 ms. Trace:
I20260709 13:01:43.500284  1668 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:43.499230  1666 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:43.480427  3472 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:43.508780  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.494567  3412 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.259s	user 0.010s	sys 0.014s
I20260709 13:01:43.509207  3427 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:43.515290  1738 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.516111  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:43.525604  1739 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.526268  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.449286  3327 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:43.526485  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1815) took 1849 ms (client timeout 2241 ms). Trace:
I20260709 13:01:43.513865  3473 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.494786  3314 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:01:43.513998  3357 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260709 13:01:43.523866  3411 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.005s
I20260709 13:01:43.766835  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:43.538231  3412 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.651778  1584 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:43.766814  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:43.551250  1666 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:01:43.500612  1660 rpcz_store.cc:276] 0709 13:01:42.246709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.247004 (+   295us) service_pool.cc:224] Handling call
0709 13:01:42.247535 (+   531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.248029 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:42.447120 (+199091us) log.cc:844] Serialized 14 byte log entry
0709 13:01:43.362826 (+915706us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:43.446781 (+ 83955us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:43.470450 (+ 23669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.482832 (+ 12382us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:43.495369 (+ 12537us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":244919,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":9859,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":47667,"thread_start_us":1030257,"threads_started":2}
I20260709 13:01:43.781268  3466 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.782315  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:43.782538  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:43.772224  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:43.645699  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 174807us
I20260709 13:01:43.538870  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:43.550395  1740 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.641165  3435 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:43.534556  1742 rpcz_store.cc:269] 0709 13:01:41.677524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.204294 (+526770us) service_pool.cc:224] Handling call
0709 13:01:43.509904 (+1305610us) spinlock_profiling.cc:243] Waited 925 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:43.526421 (+ 16517us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2127457536}
I20260709 13:01:43.475242  1587 rpcz_store.cc:276] 0709 13:01:42.276647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.276962 (+   315us) service_pool.cc:224] Handling call
0709 13:01:42.277447 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:42.563692 (+286245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:42.563721 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:42.571203 (+  7482us) log.cc:844] Serialized 26 byte log entry
0709 13:01:42.950081 (+378878us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:42.950221 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:43.456732 (+506511us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b540031db54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:43.456788 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:43.456851 (+    63us) raft_consensus.cc:1685] finished
0709 13:01:43.456869 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:43.457158 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6258,"raft.queue_time_us":12127,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":8000,"spinlock_wait_cycles":632596352,"thread_start_us":336942,"threads_started":1,"wal-append.queue_time_us":385794}
I20260709 13:01:43.577231  3496 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:43.515774  3490 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:43.587877  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.706637  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:43.621405  1632 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.524s (SENT)
I20260709 13:01:43.787098  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:43.768512  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:43.706837  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.777932  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:43.761281  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.804601  1735 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.693080  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.786927  3526 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.648s	user 0.002s	sys 0.000s
I20260709 13:01:43.803820  1589 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 2: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:43.807232  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:43.807528  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.807942  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:43.808399  1742 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:43.808734  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 196412us
I20260709 13:01:43.804461  2311 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:43.809751  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:43.810343  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:43.811033  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:43.811442  1668 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 3
W20260709 13:01:43.811894  1632 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.670s (SENT)
I20260709 13:01:43.812768  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:43.813135  1632 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.758s (SENT)
I20260709 13:01:43.813851  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:43.811887  1666 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:43.814924  1634 leader_election.cc:400] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:43.815272  1634 leader_election.cc:403] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:01:43.816162  1666 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:43.810951  1664 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:43.818110  3508 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:43.818753  1660 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:43.818856  3508 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.819360  3473 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:01:43.820175  1632 leader_election.cc:336] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.820s (SENT)
I20260709 13:01:43.820566  3473 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:43.820966  3473 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:43.821879  3372 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:43.821923  3392 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:43.822543  3392 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:43.822850  3533 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.823472  3533 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.823565  3392 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.825174  3392 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.825765  3427 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:43.826524  1666 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:43.827167  3427 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:43.827600  3427 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:43.827952  1666 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 13:01:43.828858  1557 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.831s (SENT)
I20260709 13:01:43.828621  3427 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.809881  3496 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:43.831202  3496 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.833287  1668 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.819381  3508 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:43.836015  3508 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:43.836457  3519 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.538s	user 0.008s	sys 0.003s
I20260709 13:01:43.821333  3473 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:43.839541  1659 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:01:43.841699  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.842262  1664 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:43.791644  1739 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.829349  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:43.850576  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:43.859735  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.860276  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:43.860862  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46132.4us
I20260709 13:01:43.865422  3533 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:43.865996  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.866235  3338 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:43.866696  3338 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:43.866968  1737 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.867089  3338 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:43.868216  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.871088  1733 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.871941  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.872514  1733 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.875854  1741 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:43.876731  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.621367  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.38063e+06us
I20260709 13:01:43.878208  1738 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:43.706926  3538 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:43.880098  1588 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.710s	user 0.006s	sys 0.005s
I20260709 13:01:43.880445  1588 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:43.707288  3327 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.669377  3284 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.837024  3519 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.892239  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.892846  1739 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:43.896337  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.899091  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.902539  2318 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:43.905231  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.907455  1733 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:43.885865  3472 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:43.886636  3412 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:43.906153  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:43.930226  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:43.931558  1741 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.933872  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:43.912118  3400 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.701s	user 0.006s	sys 0.010s
I20260709 13:01:43.935622  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:43.935598  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45476.1us
I20260709 13:01:43.898540  3435 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:43.938629  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:43.899596  3384 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:43.915555  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:43.944099  1742 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:01:43.944427  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115612us
I20260709 13:01:43.689078  3339 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:43.952958  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91396.8us
I20260709 13:01:43.955008  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:43.958791  1588 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 13:01:43.965653  3519 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.965811  3435 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.969033  3435 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.965690  3427 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:43.897840  3466 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:43.984323  3466 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:43.985283  3350 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.000s
I20260709 13:01:43.989099  3141 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.764736  3383 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.026126  1737 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 1.
I20260709 13:01:44.005825  3401 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.008038  3187 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:43.994273  3451 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:43.991796  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.067782  1660 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:43.999002  1659 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:44.005326  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.003604  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.074009  1661 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.015573  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1881) took 1101 ms. Trace:
I20260709 13:01:44.034423  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:43.621934  3470 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.074980  1665 rpcz_store.cc:276] 0709 13:01:42.938947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.951559 (+ 12612us) service_pool.cc:224] Handling call
0709 13:01:44.015301 (+1063742us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:44.015458 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432920320}
I20260709 13:01:44.082804  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 1817) took 1089 ms. Trace:
I20260709 13:01:44.083432  1590 rpcz_store.cc:276] 0709 13:01:43.029634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:43.086454 (+ 56820us) service_pool.cc:224] Handling call
0709 13:01:44.074514 (+988060us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54001fa254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:44.082716 (+  8202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188825472}
W20260709 13:01:44.085227  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:43.936575  3354 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:43.906920  3538 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:44.060578  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.034968  3497 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:44.076081  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 122012us
I20260709 13:01:44.106344  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.048929  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.108119  1733 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:43.936380  3284 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.104899  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.120364  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 210) took 1146 ms. Trace:
I20260709 13:01:44.120838  1734 rpcz_store.cc:276] 0709 13:01:42.929479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:42.936247 (+  6768us) service_pool.cc:224] Handling call
0709 13:01:42.936899 (+   652us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:43.370143 (+433244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.069316 (+699173us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b540012d9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:44.075990 (+  6674us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961829120}
I20260709 13:01:44.120302  1738 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:44.114660  3401 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.129642  3392 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:44.130151  3392 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:44.130545  3392 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.133948  3413 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:01:44.134604  1589 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:01:44.135026  1589 pending_rounds.cc:85] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:01:44.135318  1589 pending_rounds.cc:107] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:01:44.135602  1589 raft_consensus.cc:2889] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:44.136109  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.137245  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.152781  1740 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:44.154251  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:44.156247  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.067348  3451 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:44.075516  3280 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:44.158667  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.159834  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.160648  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:44.158614  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:44.090118  3508 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.152613  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.162827  1735 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:44.163086  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80538.5us
I20260709 13:01:44.174698  1667 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:44.163451  1591 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:44.164626  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.178195  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:44.165537  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 344326us
I20260709 13:01:44.139335  1734 raft_consensus.cc:2376] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3. Re-sending same reply.
I20260709 13:01:44.167030  1742 raft_consensus.cc:2376] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3. Re-sending same reply.
I20260709 13:01:44.169917  1739 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:44.154290  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.173596  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:44.180743  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.182252  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:44.184064  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.184831  1733 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:44.186306  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.189033  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:44.190709  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72674.9us
I20260709 13:01:44.191459  2317 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 2: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:44.162436  3472 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:44.197851  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.199532  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1256) took 1207 ms. Trace:
I20260709 13:01:44.201557  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.202261  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:44.202622  1665 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:44.206746  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.207535  1734 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:44.208910  1734 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:44.209686  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 252) took 9913 ms. Trace:
I20260709 13:01:44.210163  2312 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:44.211133  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.211225  3401 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.211666  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:44.193948  3141 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260709 13:01:44.194530  3383 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.213196  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.201094  1738 rpcz_store.cc:276] 0709 13:01:42.992402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.151630 (+1159228us) service_pool.cc:224] Handling call
0709 13:01:44.199435 (+ 47805us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:44.219759  1667 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:44.216476  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.222783  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.223489  1665 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.213124  3141 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:44.215924  3497 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.229748  3141 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:44.230834  3141 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.231992  3497 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.232513  3141 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.232824  1736 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:44.200466  3508 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.235302  3508 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.238432  3451 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.239642  3451 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.232600  3280 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:44.242347  3280 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:44.242630  1738 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:01:44.242730  3280 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:44.244504  1738 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:44.244140  3280 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.234119  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 230) took 2265 ms. Trace:
I20260709 13:01:44.245709  3284 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:44.213425  1584 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:37610
I20260709 13:01:44.247077  1584 raft_consensus.cc:606] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.247485  1584 raft_consensus.cc:614] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:44.227161  3187 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:44.248853  3187 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.249969  3187 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.250686  1734 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:44.115885  3354 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:44.217712  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51935.2us
I20260709 13:01:44.254216  3354 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:44.256662  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.229408  3401 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:44.245891  1593 rpcz_store.cc:276] 0709 13:01:41.968456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:41.968684 (+   228us) service_pool.cc:224] Handling call
0709 13:01:42.861490 (+892806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:42.863202 (+  1712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.233891 (+1370689us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b54003b10d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:44.234039 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1977828096}
I20260709 13:01:44.228149  1739 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:44.264379  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1829) took 1070 ms. Trace:
I20260709 13:01:44.264765  1739 rpcz_store.cc:276] 0709 13:01:43.201662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.188697 (+987035us) service_pool.cc:224] Handling call
0709 13:01:44.264315 (+ 75618us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:44.265825  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62328.1us
I20260709 13:01:44.272018  1740 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:44.273813  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:44.276878  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88583us
I20260709 13:01:44.289217  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.277405  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.279443  1736 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:44.259861  3280 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:44.277209  1632 leader_election.cc:336] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.994s (SENT)
I20260709 13:01:44.273149  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:44.242900  3538 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.280530  1735 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:01:44.263612  3497 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.281591  1739 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:44.293828  1733 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:44.299749  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.280867  3513 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:44.300076  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83885.5us
I20260709 13:01:44.304040  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1258) took 1304 ms. Trace:
I20260709 13:01:44.304555  1735 rpcz_store.cc:276] 0709 13:01:42.993093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.203203 (+1210110us) service_pool.cc:224] Handling call
0709 13:01:44.226746 (+ 23543us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:44.296926 (+ 70180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.297174 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:44.311525  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:44.313735  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1251) took 1313 ms. Trace:
I20260709 13:01:44.314319  1733 rpcz_store.cc:276] 0709 13:01:42.990320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.196070 (+1205750us) service_pool.cc:224] Handling call
0709 13:01:44.275482 (+ 79412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:44.303251 (+ 27769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.303551 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:44.210158  1489 rpcz_store.cc:276] 0709 13:01:34.210467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:34.210651 (+   184us) service_pool.cc:224] Handling call
0709 13:01:34.224849 (+ 14198us) spinlock_profiling.cc:243] Waited 8.69 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb90b2 00007fcdf3e32a89 00007fcdf3dcada6 00007fcdf3d9c088 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc
0709 13:01:34.228113 (+  3264us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8e9c 00007fcdeaeb90f6 00007fcdf3e32a89 00007fcdf3dcada6 00007fcdf3d9c088 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc
0709 13:01:44.092823 (+9864710us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:37.295946 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:37.341193 (+ 45247us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:37.382270 (+ 41077us) write_op.cc:435] Acquired schema lock
0709 13:01:37.410140 (+ 27870us) tablet.cc:662] Decoding operations
0709 13:01:37.484749 (+ 74609us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:37.495479 (+ 10730us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:37.503187 (+  7708us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:37.561351 (+ 58164us) tablet.cc:701] Row locks acquired
0709 13:01:37.570321 (+  8970us) write_op.cc:260] PREPARE: finished
0709 13:01:37.589891 (+ 19570us) write_op.cc:270] Start()
0709 13:01:37.605371 (+ 15480us) write_op.cc:276] Timestamp: P: 1783602097586828 usec, L: 0
0709 13:01:37.607723 (+  2352us) op_driver.cc:348] REPLICATION: starting
0709 13:01:37.720901 (+113178us) log.cc:844] Serialized 599 byte log entry
0709 13:01:39.318746 (+1597845us) op_driver.cc:464] REPLICATION: finished
0709 13:01:40.117465 (+798719us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfda32a 00007fcdedf5ff14 00007fcdedf5fec3 00007fcdedf5fe38 00007fcdedf5fdf3 00007fcdedf5f0dc 00007fcdedf88e16 00007fcdedf70cbb 00007fcdedf5b39d 00007fcdedf5ed75
0709 13:01:40.393106 (+275641us) write_op.cc:301] APPLY: starting
0709 13:01:40.395951 (+  2845us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:40.431297 (+ 35346us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:40.434536 (+  3239us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:40.472113 (+ 37577us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:40.472301 (+   188us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:40.472343 (+    42us) write_op.cc:312] APPLY: finished
0709 13:01:40.472663 (+   320us) log.cc:844] Serialized 23 byte log entry
0709 13:01:40.733225 (+260562us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:40.738829 (+  5604us) write_op.cc:454] Released schema lock
0709 13:01:40.738882 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:40.792146 (+ 53264us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:40.792191 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1065324,"reported_tablets":5,"spinlock_wait_cycles":25122048,"child_traces":[["op",{"apply.queue_time_us":1011854,"apply.run_cpu_time_us":14296,"apply.run_wall_time_us":401448,"num_ops":1,"prepare.queue_time_us":353388,"prepare.run_cpu_time_us":72672,"prepare.run_wall_time_us":1925380,"raft.queue_time_us":420114,"raft.run_cpu_time_us":36590,"raft.run_wall_time_us":1224610,"replication_time_us":1625877,"spinlock_wait_cycles":705783040,"thread_start_us":2358638,"threads_started":5,"wal-append.queue_time_us":994824,"wal-append.run_cpu_time_us":13259,"wal-append.run_wall_time_us":2184076}]]}
W20260709 13:01:44.315953  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:44.317553  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51067.4us
I20260709 13:01:44.318715  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.320451  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:44.325873  1741 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:01:44.326942  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1263) took 1131 ms. Trace:
I20260709 13:01:44.324304  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.328998  3490 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.329722  3490 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.327418  1741 rpcz_store.cc:276] 0709 13:01:43.195905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.204243 (+1008338us) service_pool.cc:224] Handling call
0709 13:01:44.278699 (+ 74456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:44.326540 (+ 47841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.326856 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:44.334290  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:44.341053  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52054us
I20260709 13:01:44.342674  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.332989  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.344653  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.273140  3314 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:01:44.346906  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.351097  1667 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:44.348811  1736 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:44.350517  1661 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:44.356767  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:44.361692  1667 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:44.367082  3384 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:44.318346  3497 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.353811  3411 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:44.379453  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.380896  1667 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:44.381141  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.367426  3372 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:01:44.385118  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107936us
I20260709 13:01:44.367635  3401 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.365025  3538 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:44.391053  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.397758  3496 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:01:44.398475  3442 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.974s	user 0.000s	sys 0.018s
I20260709 13:01:44.398738  3496 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:44.399066  3442 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.401333  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:44.402225  3473 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.001s
I20260709 13:01:44.402323  1667 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:44.402832  3473 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.409832  1733 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:44.411002  3497 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.414175  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:44.421002  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.411720  3401 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.423830  3401 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:44.424499  3372 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.425683  3372 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.429917  1665 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:01:44.430665  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88532.8us
I20260709 13:01:44.431717  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:44.432847  1632 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.136s (SENT)
I20260709 13:01:44.438618  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:44.439656  1665 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:44.445736  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.449651  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:44.451620  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 566543us
I20260709 13:01:44.453954  1665 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:44.437378  3442 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:44.437809  3427 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:44.461450  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:44.463620  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 117006us
I20260709 13:01:44.451530  3497 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.469087  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:44.458937  2310 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 13:01:44.484947  3443 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.485572  3443 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:44.485981  3443 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.486817  3392 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:44.466634  3338 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.599s	user 0.008s	sys 0.007s
I20260709 13:01:44.490056  1662 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:44.489921  3338 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.491539  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.492328  3338 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:44.492877  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:44.493492  1662 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.495082  1742 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 5. Ops: []
W20260709 13:01:44.495667  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.456529  3473 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:44.498014  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66442.2us
I20260709 13:01:44.499496  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:44.467198  3383 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.511971  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:44.512712  2314 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:44.509662  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116820us
I20260709 13:01:44.471772  3497 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:44.517796  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 199239us
W20260709 13:01:44.515713  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102553us
I20260709 13:01:44.516288  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.517659  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:44.507445  3546 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.120s	user 0.002s	sys 0.000s
I20260709 13:01:44.493675  3392 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:44.524099  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:44.525049  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:44.472697  2310 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:44.528554  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.438125  3490 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.539373  1665 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:44.548607  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.547900  1591 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:01:44.520817  3497 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.553990  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.555001  1587 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:44.561383  1588 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:44.561931  1589 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:44.541347  3187 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:44.576666  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56998.1us
I20260709 13:01:44.527094  3473 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:44.574765  1634 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
W20260709 13:01:44.575448  1632 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.241s (SENT)
I20260709 13:01:44.578423  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.578810  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.578434  1634 leader_election.cc:403] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:01:44.578184  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:44.581401  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76468.7us
I20260709 13:01:44.582267  1661 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:44.584491  3553 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 13:01:44.586130  3554 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:01:44.520175  3383 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.593966  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:44.594893  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:44.595778  1591 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.596418  1660 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:44.589658  1593 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:44.603461  1661 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:44.598229  3451 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:01:44.606936  3451 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:44.607317  3451 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:44.608359  3451 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.573810  3466 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.521181  3519 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.077s
I20260709 13:01:44.609745  3466 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:44.610257  3466 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.610806  3519 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:44.611258  3519 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:44.611692  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.613355  3473 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.615689  3473 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:44.616341  1664 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:44.617856  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.536831  3427 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 13:01:44.628093  3427 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:44.628654  3427 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:44.591671  3187 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.618422  3372 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:44.609758  3405 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:01:44.679240  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49794.2us
I20260709 13:01:44.679675  3405 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:44.680261  3427 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.680800  3405 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.681740  3427 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.684307  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.685381  3339 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:44.620319  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78867.1us
I20260709 13:01:44.684466  3280 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 13:01:44.620361  3498 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:44.622260  3451 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:44.622151  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68581.5us
I20260709 13:01:44.687732  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:44.687876  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.621019  3562 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:44.597874  3455 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:44.624339  1663 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:44.689826  3392 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:44.691442  3392 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:44.692162  3392 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.692662  3392 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:44.594691  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:44.635512  3556 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 13:01:44.626418  3383 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.696441  1632 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3.
I20260709 13:01:44.697646  3372 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:44.698705  3508 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:44.699159  3508 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.699652  3562 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:44.604877  3549 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.275s	user 0.001s	sys 0.003s
I20260709 13:01:44.700320  3508 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.551446  3538 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
I20260709 13:01:44.713248  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.598125  3442 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.637758  3384 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:44.686275  1739 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:01:44.701357  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:44.577466  3412 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.666240  1664 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:44.703995  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.708279  1662 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:44.712235  1591 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:44.693055  3392 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.681777  1737 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:44.721386  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45068.5us
W20260709 13:01:44.723547  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 145467us
I20260709 13:01:44.724766  2314 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:44.726424  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.727982  1666 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:44.714599  3327 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.812s	user 0.002s	sys 0.013s
I20260709 13:01:44.735610  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:44.746095  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44898.9us
I20260709 13:01:44.743541  1632 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
I20260709 13:01:44.735729  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.754136  1632 leader_election.cc:403] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:44.753877  1664 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
W20260709 13:01:44.758559  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138624us
I20260709 13:01:44.745808  3533 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:44.760854  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.758874  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.759929  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.766407  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.765801  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:44.796185  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:44.797254  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61774.5us
I20260709 13:01:44.796998  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:44.797688  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.737804  3451 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.737387  3187 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:44.805945  1668 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:44.804838  1734 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:44.742347  3568 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 13:01:44.764230  3411 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4.
W20260709 13:01:44.823706  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58007.9us
I20260709 13:01:44.770087  3354 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:44.823179  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.822909  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:44.833881  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74068.7us
W20260709 13:01:44.834767  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92191.8us
I20260709 13:01:44.824201  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.841652  1739 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:44.843847  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:44.804240  3490 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:44.853730  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:44.854758  1734 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 2: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:44.855049  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:44.859647  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.764782  3432 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.764562  3412 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:44.802013  3508 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:44.749022  3489 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 13:01:44.851632  1586 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:44.842005  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:44.802450  3442 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:44.829847  3339 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.866788  1738 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 13:01:44.867223  1738 pending_rounds.cc:85] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 1
I20260709 13:01:44.867702  1738 pending_rounds.cc:107] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:01:44.867942  1738 raft_consensus.cc:2889] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:44.868799  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71908us
I20260709 13:01:44.830260  3451 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:44.871878  2314 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:44.822870  3350 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:44.844609  3141 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:44.888193  3372 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:44.839485  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.764079  3555 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.125s
W20260709 13:01:44.899132  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 140359us
I20260709 13:01:44.862684  3327 raft_consensus.cc:2764] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:01:44.861877  3538 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:01:44.905529  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71915.6us
I20260709 13:01:44.906672  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.908129  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.908149  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:44.911183  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:44.892372  3141 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:44.895870  3473 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:44.909585  3350 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.931387  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1265) took 1645 ms. Trace:
I20260709 13:01:44.894563  3497 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.918876   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 13:01:44.873811  3432 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:44.937937  1742 rpcz_store.cc:276] 0709 13:01:43.286426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.227976 (+941550us) service_pool.cc:224] Handling call
0709 13:01:44.257910 (+ 29934us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:44.498535 (+240625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:44.931096 (+432561us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b54003fa7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:44.931306 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545011072}
I20260709 13:01:44.910156  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:44.938884  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48832.2us
I20260709 13:01:44.899377  3496 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 13:01:44.896135  3455 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
W20260709 13:01:44.945384  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79264.9us
I20260709 13:01:44.921706  3141 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:44.960139  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53615.8us
I20260709 13:01:44.961061  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:44.964030  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:44.918313  3392 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.993345  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.952097  3354 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:44.997123  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:44.998332  1741 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:45.002794  3571 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 13:01:45.010409  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.011010  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.011441  1736 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:45.011812  1739 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
W20260709 13:01:45.013482  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50505.1us
I20260709 13:01:45.023687  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:44.959946  3490 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:44.935137  3533 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20260709 13:01:44.974159  3327 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:44.942667  3413 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20260709 13:01:45.005962  3443 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.007s
I20260709 13:01:44.944109  3350 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:45.026358  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56608.5us
W20260709 13:01:45.041301  3581 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.080s	user 0.002s	sys 0.000s
I20260709 13:01:45.042049   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:01:45.042240  3567 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 13:01:45.024616  3568 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 13:01:45.024309  3472 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:45.056754  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 95797.9us
I20260709 13:01:45.004415  3575 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:45.072741  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:45.069152  1736 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 13:01:44.998155  3354 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:45.046455  3455 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260709 13:01:45.073956  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50383.9us
I20260709 13:01:45.084308  1740 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:01:45.074172  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 205095us
I20260709 13:01:45.048077  3586 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:45.053932  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:45.025913  3327 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:45.098857  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45054.9us
W20260709 13:01:45.094440  3561 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 13:01:45.096068  3573 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 13:01:45.048866  3443 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.097034  3578 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:45.100208  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:01:45.101897  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 132711us
I20260709 13:01:44.998330  3519 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:44.999348  3451 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.075922  3472 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:45.071637  3582 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:45.112294  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:45.035876  3412 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:45.114022  1557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.786s (SENT) [suppressed 53 similar messages]
I20260709 13:01:45.117683  3357 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:45.119364  3347 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:45.087944  3538 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:45.122252  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.047797  3442 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.106510  3473 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.105710  3568 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.121569  3496 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:45.135811  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:45.138592  3470 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:01:45.139060  3570 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.154s	user 0.002s	sys 0.000s
I20260709 13:01:45.140062  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1303) took 1172 ms. Trace:
I20260709 13:01:45.154681  1737 rpcz_store.cc:276] 0709 13:01:43.967557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.090687 (+123130us) service_pool.cc:224] Handling call
0709 13:01:45.139852 (+1049165us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b5400352ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:45.139998 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":674083712}
I20260709 13:01:45.115236  3490 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.128250  3357 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.128013  3347 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:45.143031  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:45.105711  3383 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:45.108539  3577 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 13:01:45.154206  3496 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.153365  3405 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.177975  3586 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:45.246150  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1893) took 1220 ms. Trace:
I20260709 13:01:45.295336  1665 rpcz_store.cc:276] 0709 13:01:44.046888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.515017 (+468129us) service_pool.cc:224] Handling call
0709 13:01:44.515414 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:44.539855 (+ 24441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.245772 (+705917us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:45.246052 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54516480}
I20260709 13:01:45.244704  3347 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:45.249745  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 195497us
I20260709 13:01:45.247671  3338 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:45.245831  3595 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.169592  3568 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.237259  3392 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:45.202755  3432 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.177275  3497 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:45.203045  3496 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.168496  3357 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:45.167934  3383 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.127331  3413 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 13:01:45.181042  3533 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 13:01:45.163616  3538 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:45.163360  3470 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:45.171455  3597 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
W20260709 13:01:45.320961  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 248368us
I20260709 13:01:45.372851  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:45.334002  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 191064us
I20260709 13:01:45.373827  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:45.335055  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.340368  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:45.343943  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.351264  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1296) took 1111 ms. Trace:
I20260709 13:01:45.320570  3338 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:01:45.362556  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 253353us
I20260709 13:01:45.375803  1666 rpcz_store.cc:276] 0709 13:01:44.240035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.590078 (+350043us) service_pool.cc:224] Handling call
0709 13:01:44.727740 (+137662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:44.728536 (+   796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:45.350969 (+622433us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400118350. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:45.351167 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315719040}
I20260709 13:01:45.349442  3350 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.338184  3347 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.312704  3405 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.357465  3383 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:45.361819  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 226152us
I20260709 13:01:45.402323  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.410264  3609 raft_consensus.cc:2764] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:01:45.411527  3601 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:45.378340  3578 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:45.376699  3432 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:45.383904  3327 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:45.414083  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.419023  3589 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.402912  3350 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:45.399260  3443 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:45.417214  1705 leader_election.cc:400] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:45.429618  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:45.432379  1705 leader_election.cc:403] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 13:01:45.435464  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:45.432925  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60190.3us
I20260709 13:01:45.411016  3338 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.433130  3594 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:45.434003  3591 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:01:45.431545  3596 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:45.446393  1737 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:45.414475  3601 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:45.450958  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:45.451185  3338 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:45.438624  3582 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.345573  3595 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.437440  3350 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.475780  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:45.484333  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.432484  3432 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.428090  3578 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.487207  3568 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:45.443389  3141 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.420113  3586 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.487332  3600 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:45.452239  3538 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.437183  3589 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.503505  3608 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:45.504056  3405 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:45.506078  3427 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.824s	user 0.009s	sys 0.006s
I20260709 13:01:45.508452  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:45.525226  1665 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 13:01:45.533831  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.523840  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:45.529390  1587 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:45.520428  1592 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.536139  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:45.537734  1660 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:45.540949  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65282.6us
I20260709 13:01:45.542513  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:45.506752  3384 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.780s	user 0.010s	sys 0.009s
I20260709 13:01:45.497354  3596 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.546036  1585 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:45.548512  1661 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:01:45.511489  3613 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 13:01:45.515076  3466 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.904s	user 0.005s	sys 0.009s
I20260709 13:01:45.452654  3594 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.510540  3609 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.517006  3589 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:45.510118  3612 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:01:45.566044  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.552706  3568 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 13:01:45.578738  1666 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:45.560528  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 186784us
I20260709 13:01:45.448532  3470 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.552497  3489 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.573756  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:45.583187  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.545251  3600 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:45.473356  3582 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.553013  3384 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.585868  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:45.563827  3327 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:45.586916  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.585989  3600 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:45.588326  1593 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:01:45.589736  1590 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:45.590097  3610 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:45.590520  3607 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:45.596540  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54126.5us
I20260709 13:01:45.599709  1632 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:45.587502  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:45.601289  1666 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.600775  1665 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:45.600417  3610 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.585638  3473 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.585728  3612 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.570667  3466 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.576368  3577 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 13:01:45.606020  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:45.576454  3427 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
W20260709 13:01:45.613962  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.613888  1661 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:45.619994  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:45.621115  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:45.622313  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:45.622725  1668 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:01:45.597750  3600 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.626778  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:45.602921  3327 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.595549  3412 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 13:01:45.593716  3489 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.636152  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:45.641561  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.643869  1590 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:45.631819  3577 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:45.595832  3350 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.628387  3614 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.613977  3384 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:01:45.664907  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.665596  1735 raft_consensus.cc:2376] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2. Re-sending same reply.
I20260709 13:01:45.666456  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.635977  3427 consensus_queue.cc:1243] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:01:45.638633  3473 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:45.578033  3601 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:45.671062  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:45.674477  1557 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.762s (SENT)
W20260709 13:01:45.675431  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 239985us
I20260709 13:01:45.675684  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.676189  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.678297  2314 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:45.607226  3607 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.678229  1737 raft_consensus.cc:2376] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3. Re-sending same reply.
I20260709 13:01:45.631474  3284 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:01:45.680774  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.709746  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.682042  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.657696  3577 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:45.663417  3508 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.793s	user 0.017s	sys 0.004s
I20260709 13:01:45.658082  3392 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.720028  2312 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:45.723781  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:45.760037  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.760694  1737 raft_consensus.cc:2376] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5. Re-sending same reply.
W20260709 13:01:45.758862  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76877.3us
I20260709 13:01:45.715734  3605 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:45.661747  3383 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.768988  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.771736  1735 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:45.692503  3411 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260709 13:01:45.720301  3606 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.679872  3568 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260709 13:01:45.693831  3427 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 13:01:45.755650  3432 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.672576  3611 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.203s
W20260709 13:01:45.780722  3599 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260709 13:01:45.780478  3618 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20260709 13:01:45.780714  3598 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.149s
I20260709 13:01:45.725993  3442 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 13:01:45.790241  2316 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:45.771040  3605 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.639700  3610 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.791734  1592 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:45.778761  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:45.793560  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 157550us
I20260709 13:01:45.767373  3578 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.638499  3596 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.803826  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.803933  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:45.805568  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:45.757449  3327 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.797940  3605 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:45.817149  3580 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:45.817844  3580 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.821799  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:45.803347  3466 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:45.807760  3284 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:45.798955  3621 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.830734  3498 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:45.830852  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.831233  3498 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:45.831689  2312 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:45.832201  3498 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.811790  3582 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.833478  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:45.834290  3489 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:45.833276  3498 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:45.834863  3622 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:45.835549  3384 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:45.837491  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.832558  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.840543  3470 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:45.815016  3508 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260709 13:01:45.842849  3472 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.678s	user 0.006s	sys 0.007s
I20260709 13:01:45.843330  3472 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.808997  3625 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 13:01:45.846331  3280 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:45.846592  3625 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:45.847380  3339 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:45.847196  3625 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.847797  3339 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.852433  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:45.853579  3555 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:45.854039  3555 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:45.860585  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:45.864326  2313 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:45.810467  3578 raft_consensus.cc:2764] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:01:45.814448  3392 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:01:45.878561  3384 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:45.874054  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.779062  3606 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.821069  3610 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:45.829016  3284 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:45.803893  3577 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:45.879109  3611 consensus_queue.cc:1243] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260709 13:01:45.875123  3357 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.505s	user 0.006s	sys 0.005s
I20260709 13:01:45.845433  3472 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.894613  1733 raft_consensus.cc:2376] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3. Re-sending same reply.
I20260709 13:01:45.903146  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:45.968597  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 65595.9us
I20260709 13:01:45.903509  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:45.911048  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.905362  3473 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.884503  3618 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:45.915292  3508 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.904551  3327 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:45.908442  3284 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:45.971628  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.993453  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:45.957861  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 181297us
I20260709 13:01:45.980243  1666 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:01:45.919989  3496 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.002s
I20260709 13:01:45.907577  3489 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.800534  3612 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:45.907361  3582 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.914211  3625 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:45.995486  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:45.998332  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.901181  3339 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:46.011978  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.012161  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.024300  1734 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:46.036705  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.037523  1739 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:46.037448  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:45.984299  3392 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.017092  3496 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.038753  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.031003  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 220) took 1057 ms. Trace:
I20260709 13:01:46.042373  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:45.932237  3357 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.044699  1632 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:45.981297  3585 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:45.984802  3577 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.050621  1739 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:45.984576  3327 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:46.046597  1740 rpcz_store.cc:276] 0709 13:01:44.973841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.999034 (+ 25193us) service_pool.cc:224] Handling call
0709 13:01:45.010408 (+ 11374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.084966 (+ 74558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.030640 (+945674us) spinlock_profiling.cc:243] Waited 65.2 ms on lock 0x7b54002030d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:46.030917 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149921536}
I20260709 13:01:46.058933  1734 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:46.010383  3621 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:45.986253  3611 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:46.024672  3607 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.041457  3442 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.069895  2310 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:46.072753  3496 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.073321  2317 raft_consensus.cc:2368] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5.
I20260709 13:01:46.074580  3600 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.077401  3600 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.081561  3580 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.082988  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.083441  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.083933  1740 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:46.084805  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.085410  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.087194  3580 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:46.088076  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.088091  3635 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:46.088901  3635 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:46.025985  3582 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.090009  3635 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.092077  3594 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.092410  3582 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:46.072932  3632 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 13:01:46.093425  3427 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:46.094297  3602 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:46.094466  3427 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:46.094844  3602 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:45.983495  3473 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.070024  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.066906  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.101171  3442 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.105774  3427 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.107522  3427 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.011684  3508 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.060335  3585 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.062286  3383 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.110126  3607 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:46.099556  3564 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:46.110987  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:46.114272  3338 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.002s
I20260709 13:01:46.106308  3442 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.115175  3594 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:01:46.116382  3606 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:46.029681  3489 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.133566  3587 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:01:46.115273  3339 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.135738  3489 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.136333  3473 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.136763  3594 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:46.137248  3594 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:46.138514  3508 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.139402  3327 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.140115  3583 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:46.141026  3327 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.098209  3611 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.129982  3564 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.143913  3611 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.111038  3555 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.146355  3564 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.109300  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 217) took 1137 ms. Trace:
I20260709 13:01:46.146793  3555 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:46.122800  3623 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.314s	user 0.000s	sys 0.001s
I20260709 13:01:46.106518  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.053922  3384 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260709 13:01:46.100412  3608 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.554s	user 0.010s	sys 0.003s
I20260709 13:01:46.108330  3357 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.127893  1593 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:46.130023  1740 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.186571  3608 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:46.190205  1740 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 13:01:46.191318  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.191907  1740 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:46.192777  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.115110  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.193900  1733 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.068564  3621 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.195514  1733 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:46.138170  3594 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.207229  3594 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:46.125113  3607 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:46.209064  3607 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:46.210917  3519 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:46.215188  3621 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:46.224630  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.121564  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51659.3us
I20260709 13:01:46.228029  2314 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.150282  1739 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 13:01:45.985684  3618 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:46.168316  3627 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.306s	user 0.003s	sys 0.000s
I20260709 13:01:46.147056  1742 rpcz_store.cc:276] 0709 13:01:44.972065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.972252 (+   187us) service_pool.cc:224] Handling call
0709 13:01:45.002189 (+ 29937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:45.002494 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:45.002521 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:45.009369 (+  6848us) log.cc:844] Serialized 26 byte log entry
0709 13:01:45.070075 (+ 60706us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:45.184869 (+114794us) log.cc:844] Serialized 14 byte log entry
0709 13:01:46.108699 (+923830us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18002b5940. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:01:46.108859 (+   160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.108948 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.108997 (+    49us) raft_consensus.cc:1685] finished
0709 13:01:46.109093 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.109238 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23753,"prepare.queue_time_us":249452,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":234666,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":12851,"spinlock_wait_cycles":27945856,"thread_start_us":282215,"threads_started":3,"wal-append.queue_time_us":96476}
I20260709 13:01:46.168401  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.169371  3383 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.173903  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.235850  1585 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:46.174710  3470 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
W20260709 13:01:46.174002  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67579.9us
W20260709 13:01:46.196576  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.198791  1740 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.206107  1733 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:46.235009  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:46.250768  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.237916  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:46.251585  2312 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.223322  3187 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:46.237911  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:46.259492  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:46.234985  3608 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.223799  3339 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:46.261485  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:46.374330  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.375090  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.384356  2318 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.368849  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.254558  3508 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:46.255945  3338 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:46.379119  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.393556  1742 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:46.393883  1666 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:46.375257  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 56945.5us
I20260709 13:01:46.257781  3187 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:46.396315  1666 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:46.262164  3392 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:46.397825  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 136428us
I20260709 13:01:46.406669  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:46.369866  3473 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.383014  3586 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:46.252454  3383 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.376592  3314 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:46.419152  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.417526  1666 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:46.420140  1660 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.416051  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.420859  1741 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.391939  3489 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.423089  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.424360  1742 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:46.425753  1666 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:46.391481  3470 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:46.427212  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.253648  3605 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:46.432281  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.433053  3618 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.434170  2315 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEARNER]: Advancing to term 4
W20260709 13:01:46.434547  3630 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.453s	user 0.000s	sys 0.000s
I20260709 13:01:46.447742  3585 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.449966  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.450590   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:46.408149  3338 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.455361  3338 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.456827  3490 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:46.457396  3383 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.394804  3608 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.459744  3607 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.419166  3314 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:46.462109  3470 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.462359  3314 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.463537  3314 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.462654  3608 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.464149  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.464824  1739 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:46.416435  3578 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:46.437897  3466 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:46.470679  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.471978  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.472898  1733 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.416901  3473 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:46.473150  3606 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:01:46.473623  3473 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.471294  3383 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.427574  3489 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.474574  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:46.474844  3489 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.477772  3622 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.477944  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.478235  3622 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.478866  3622 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.479843  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 205) took 1369 ms. Trace:
I20260709 13:01:46.480304  2310 rpcz_store.cc:276] 0709 13:01:45.110379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.112870 (+  2491us) service_pool.cc:224] Handling call
0709 13:01:45.163385 (+ 50515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.070551 (+907166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.479515 (+408964us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b54005ae454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:46.479754 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2050926336}
I20260709 13:01:46.482287  3605 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.429388  1584 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:46.483028  3586 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:46.474622  3383 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.484265  3187 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.489174  3187 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.489704  3602 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.491545  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 251) took 1640 ms. Trace:
W20260709 13:01:46.493045  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.502048  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.494192  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.503273  1739 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.499351  1664 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:46.457254  3490 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:46.503177  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.505578  2317 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.505967  3357 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.505698  3490 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.506510  3357 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.507324  3490 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.454835  3609 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:46.510376  3595 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.510741  3609 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:46.511848  3609 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.463662  3607 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.513311  3347 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 13:01:46.516111  3612 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:46.516714  3612 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.518141  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 257) took 1075 ms. Trace:
I20260709 13:01:46.518525  1587 rpcz_store.cc:276] 0709 13:01:45.447025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.447264 (+   239us) service_pool.cc:224] Handling call
0709 13:01:46.517953 (+1070689us) spinlock_profiling.cc:243] Waited 74.6 ms on lock 0x7b5400443754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:46.518079 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171535232}
I20260709 13:01:46.519706  3612 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:46.520394  3284 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.506s	user 0.007s	sys 0.003s
I20260709 13:01:46.520807  3284 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.522780  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.484246  3586 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.501655  1591 rpcz_store.cc:276] 0709 13:01:44.851152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.852617 (+  1465us) service_pool.cc:224] Handling call
0709 13:01:44.869676 (+ 17059us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:45.104831 (+235155us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:45.104855 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:45.107440 (+  2585us) log.cc:844] Serialized 26 byte log entry
0709 13:01:45.600586 (+493146us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:45.600644 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.490836 (+890192us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b540025c954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:46.490897 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.490951 (+    54us) raft_consensus.cc:1685] finished
0709 13:01:46.490965 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.491465 (+   500us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50298,"raft.queue_time_us":81940,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"spinlock_wait_cycles":216426880,"thread_start_us":521736,"threads_started":2,"wal-append.queue_time_us":529539}
I20260709 13:01:46.524583  2318 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4.
I20260709 13:01:46.521442  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.525157  1737 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:46.526090  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.526714  1737 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:46.527421  1660 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:46.531282  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.532656  1742 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:01:46.505638  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.538419  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.532716  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:46.408967  3339 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.539184  2316 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:46.551515  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48439.3us
I20260709 13:01:46.558912  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.563498  1588 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.557706  1586 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:46.567344  1737 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:46.569577  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.500777  3538 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:46.570828  1662 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:46.568377  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.571924  1585 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.487967  3383 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.568893  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:46.554736  1733 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:46.579648  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.580681  1733 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:46.584589  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.585752  1588 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.600132  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.600864  1586 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:46.535387  3357 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.605398  1739 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:46.538030  3466 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:46.551314  3585 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.612599  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 225) took 1511 ms. Trace:
I20260709 13:01:46.564558  3401 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.563144  3411 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:46.622066  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.614462  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.615645  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.572742  3609 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.627799  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.628772  1739 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:46.596055  3583 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:46.595108  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:46.606657  3602 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:46.628541  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.659171  1665 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:46.653402  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.634510  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:46.654781  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92488.2us
W20260709 13:01:46.625610  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.660472  1585 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:46.630075  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:46.595686  3597 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:46.623950  1588 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:46.612020  3508 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:46.608189  3582 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.609377  3284 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.621604  3472 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:46.627207  3354 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.621771  3600 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.557752  3622 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.676208  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.670082  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:46.678580  1586 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:46.682543  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.626983  3411 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.664243  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:46.684319  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.667066  1739 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:46.697294  1665 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
W20260709 13:01:46.704919  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.704957  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.685364  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.688386  2317 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 3. (term mismatch)
I20260709 13:01:46.659018  3615 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:46.673301  3587 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:46.709702  1742 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:46.712736  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.661600  3598 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 13:01:46.677775  3582 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.698014  3472 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.724074  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:46.706743  3383 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.707002  3411 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:46.730227  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:46.709407  3555 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:46.629024  3585 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:46.720265  3519 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.004s
I20260709 13:01:46.697713  3354 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.749907  3498 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:46.750476  3498 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:46.751438  3498 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.737221  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.737269  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.752955  1742 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.753059  1662 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:46.753850  3472 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.754140  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.754500  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.754894  1662 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:46.755198  1668 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:46.755683  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.755971  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.756584  1668 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:46.732043  2315 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 13:01:46.739384  3568 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.005s
W20260709 13:01:46.742565  3649 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 13:01:46.764657  2315 pending_rounds.cc:85] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 2
I20260709 13:01:46.744858  3585 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.698781  3284 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:01:46.746142  3657 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:01:46.698637  3600 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.707037  2317 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 2
I20260709 13:01:46.765901  1662 raft_consensus.cc:2368] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:46.738306  3383 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.781003  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.783376  1662 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 5
I20260709 13:01:46.752424  3354 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.765537  3577 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:46.731841  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74359.7us
W20260709 13:01:46.752492  3327 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.000s
I20260709 13:01:46.735038  3582 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.765031  2315 pending_rounds.cc:107] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:01:46.757494  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.826764  1668 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:46.801471  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.806411  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.720655  3466 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.835891  3598 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 13:01:46.826282  2315 raft_consensus.cc:2889] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:46.825601  3585 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:46.829754  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.829550  3658 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 13:01:46.831534  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1836) took 1927 ms. Trace:
I20260709 13:01:46.779443  2317 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260709 13:01:46.730373  3587 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.844750  2317 raft_consensus.cc:2917] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
I20260709 13:01:46.845655  2317 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:46.844225  1589 rpcz_store.cc:276] 0709 13:01:44.904270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:44.904527 (+   257us) service_pool.cc:224] Handling call
0709 13:01:44.919464 (+ 14937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:45.518614 (+599150us) spinlock_profiling.cc:243] Waited 1.62 ms on lock 0x7b18001c9240. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfd5bfb 00007fcdedfe064b
0709 13:01:45.518850 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:45.518890 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.452731 (+933841us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b2c00032118. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:01:46.480790 (+ 28059us) log.cc:844] Serialized 26 byte log entry
0709 13:01:46.654505 (+173715us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:46.654541 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.654614 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.831284 (+176670us) raft_consensus.cc:1685] finished
0709 13:01:46.831312 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.831465 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16141,"raft.queue_time_us":109640,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8296,"spinlock_wait_cycles":32422528,"thread_start_us":303709,"threads_started":2,"wal-append.queue_time_us":238871}
W20260709 13:01:46.847604  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.819420  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:46.809072  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 142474us
I20260709 13:01:46.821141  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.842533  3585 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.785699  3383 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.820356  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:46.851248  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.852357  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:46.857702  1742 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:46.838032  3489 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.818012  3577 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:46.837301  3339 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:46.862238  1663 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:46.857493  1662 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:46.804255  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:46.867254  3427 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.741s	user 0.007s	sys 0.003s
I20260709 13:01:46.868639  1740 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:01:46.868990  1740 pending_rounds.cc:85] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 1
I20260709 13:01:46.869208  1740 pending_rounds.cc:107] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:01:46.869469  1740 raft_consensus.cc:2889] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:46.867170  2310 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 13:01:46.870424  2310 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 4
I20260709 13:01:46.870702  2310 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:01:46.870947  2310 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:46.872135  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:46.872880  1742 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.873519  3605 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:46.874010  3606 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:01:46.876554  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72428.4us
I20260709 13:01:46.876933  1668 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.879798  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:46.874255  3605 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.864544  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:46.884857  3605 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:46.884769  3187 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:46.891594  3578 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.892129  3578 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:46.887548  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.888444  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.907414  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 256) took 1480 ms. Trace:
I20260709 13:01:46.875638  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.908324  1588 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:01:46.909176  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.910280  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:46.874799  3658 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:46.915989  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.917652  1738 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.933076  1738 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:46.921321  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:46.938912  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.942823  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.946000  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:46.909325  3598 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.945253  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.951817  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:46.908170  1592 rpcz_store.cc:276] 0709 13:01:45.426630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.426795 (+   165us) service_pool.cc:224] Handling call
0709 13:01:46.895024 (+1468229us) spinlock_profiling.cc:243] Waited 87.9 ms on lock 0x7b540025d354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:46.907313 (+ 12289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":202060672}
I20260709 13:01:46.947818  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:46.950640  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86239.1us
I20260709 13:01:46.953544  1588 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:46.953779  1662 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:46.953848  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:46.955271  3607 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:46.934948  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47468.1us
I20260709 13:01:46.902604  3622 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:46.957300  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:46.914151  3606 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:46.958184  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 228055us
I20260709 13:01:46.916613  3401 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.916976  3646 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:46.928848  3575 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:46.978016  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.941784  3187 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:46.983004  3600 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:46.950461  3605 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:46.929126  3658 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.987731  3401 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:46.982496  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:46.992317  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:46.994096  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:46.996948  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:47.016433  1741 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:46.980104  3472 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.008208  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:46.951144  3284 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260709 13:01:46.958392  3598 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.981870  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:47.012060  1742 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:46.976964  3640 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.983176  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.019508  1588 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:01:46.950906  3642 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:46.944453  3473 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.021749  2280 leader_election.cc:400] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
I20260709 13:01:46.996392  3575 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:46.952535  3411 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.025234  2280 leader_election.cc:403] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:01:47.026332  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44603.5us
I20260709 13:01:47.028960  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:47.029973  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.014223  3658 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:47.030004  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:47.024724  1589 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:47.014437  3383 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.048432  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.029734  3622 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:47.031988  3642 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.029901  3640 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.024426  3598 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.027279  3575 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.038784  3614 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:47.039461  3519 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:47.040644  3586 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:47.030926  3187 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:47.041706  3607 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:47.042905  3589 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:47.043118  3473 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.043897  3655 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:47.044952  3587 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:47.045356  3660 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.012907  3646 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:47.045663  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:47.050765  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:46.998534  3508 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 13:01:47.051416  3622 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:47.031019  1742 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 13:01:47.056787  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 204530us
I20260709 13:01:47.031353  3605 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:47.041792  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.050246  3568 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEARNER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3.
W20260709 13:01:47.055373  3470 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.003s	sys 0.016s
I20260709 13:01:47.051419  3383 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.025089  3472 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.054203  3578 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:47.091055  1589 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:47.095799  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50207.5us
W20260709 13:01:47.089699  3490 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.582s	user 0.012s	sys 0.003s
I20260709 13:01:47.055738  3609 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
W20260709 13:01:47.112892  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.079548  3187 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:47.114529  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:47.118423  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:47.101580  3622 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.120914  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.056030  3606 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.101084  3605 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:47.104821  3660 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.322556  1591 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:01:47.153512  1557 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4.
I20260709 13:01:47.324234  1557 leader_election.cc:403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:47.289332  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 514) took 1466 ms. Trace:
W20260709 13:01:47.324981  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 201549us
I20260709 13:01:47.300644  3314 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:47.325781  3314 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:47.326992  3562 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.326713  3314 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.327543  3562 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:47.328032  3314 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:47.124752  3578 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:47.329298  3578 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:47.329841  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.325249  2314 rpcz_store.cc:276] 0709 13:01:45.823277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.823427 (+   150us) service_pool.cc:224] Handling call
0709 13:01:47.289067 (+1465640us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b54003ef154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:47.289217 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":609567488}
I20260709 13:01:47.330332  3578 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.288498  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.331295  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.331656  1592 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:47.217423  3354 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:47.317051  1663 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:01:47.332669  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:47.334311  1663 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:47.334713  1739 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.335239  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:47.290968  3667 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
W20260709 13:01:47.337787  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 219500us
I20260709 13:01:47.338706  3496 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:47.339067  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:47.337716  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1856) took 1331 ms. Trace:
I20260709 13:01:47.339308  3496 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.339664  1740 rpcz_store.cc:276] 0709 13:01:46.006117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.407768 (+401651us) service_pool.cc:224] Handling call
0709 13:01:46.423635 (+ 15867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:46.869939 (+446304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.337439 (+467500us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400403b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:47.337653 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1022621056}
I20260709 13:01:47.339650  3625 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:47.341115  3625 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:47.341418  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:47.342183  1740 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:47.291615  3347 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:47.342469  3602 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.672s	user 0.010s	sys 0.003s
I20260709 13:01:47.342731  3347 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:47.343664  3347 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.344594  3347 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.344743  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1310) took 1770 ms. Trace:
I20260709 13:01:47.345191  1661 rpcz_store.cc:276] 0709 13:01:45.574639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.574782 (+   143us) service_pool.cc:224] Handling call
0709 13:01:45.575127 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.619897 (+ 44770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.344496 (+1724599us) spinlock_profiling.cc:243] Waited 38.4 ms on lock 0x7b54000a9bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:47.344679 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88312704}
I20260709 13:01:47.345902  3496 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.346276  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:47.202368  1662 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:47.347038  1661 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:01:47.355176  1661 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:01:47.352136  1592 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:01:47.138023  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.380676  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:47.113252  3470 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.003s	sys 0.017s
I20260709 13:01:47.120988  3585 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.232331  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.384720  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.381589  1740 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:47.384229  3472 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.392244  1661 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:46.617683  1736 rpcz_store.cc:276] 0709 13:01:45.100841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.101108 (+   267us) service_pool.cc:224] Handling call
0709 13:01:45.111334 (+ 10226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:45.154600 (+ 43266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:45.154635 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.032842 (+878207us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b2c00077ef8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:01:46.033474 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 13:01:46.376326 (+342852us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:46.586538 (+210212us) log.cc:844] Serialized 14 byte log entry
0709 13:01:46.586965 (+   427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.587083 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:46.587143 (+    60us) raft_consensus.cc:1685] finished
0709 13:01:46.612392 (+ 25249us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:46.612540 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19453,"prepare.queue_time_us":234900,"raft.queue_time_us":58871,"raft.run_cpu_time_us":449,"raft.run_wall_time_us":67780,"spinlock_wait_cycles":26177408,"thread_start_us":473326,"threads_started":2,"wal-append.queue_time_us":353443}
I20260709 13:01:47.396944  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:47.398583  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 239) took 1001 ms. Trace:
I20260709 13:01:47.398648  3401 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.399179  1664 rpcz_store.cc:276] 0709 13:01:46.397695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.410224 (+ 12529us) service_pool.cc:224] Handling call
0709 13:01:47.398384 (+988160us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:47.398518 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10099584}
I20260709 13:01:47.206127  3646 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:47.290220  3663 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:01:47.370608  1739 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:47.390633  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:47.279539  3668 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 13:01:47.414871  1660 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:47.391290  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:47.310535  3538 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.000s
W20260709 13:01:47.180112  3653 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.355s	user 0.000s	sys 0.003s
I20260709 13:01:47.392665  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:47.426712  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.161106  3141 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.444589  1740 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:47.444607  1593 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.325191  3466 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4.
I20260709 13:01:47.117900  3587 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:47.403829  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:47.410779  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:47.454627  1739 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:47.453665  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:47.456887  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53158.2us
W20260709 13:01:47.386861  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.302645  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.390444  3496 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:47.383307  3621 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.291131  3187 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.451941  3538 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:47.278393  3642 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.402840  3401 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:47.436798  1588 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:47.477149  3621 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.487639  3577 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.167819  3564 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:47.488329  3621 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:47.488631  3564 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:47.454056  3587 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:47.489114  3564 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:47.489351  3587 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:47.490111  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.478609  3489 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:47.490803  1591 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.459937  3314 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:47.488373  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:47.491415  3314 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:47.416167  1589 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:47.443706  1488 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 125) took 8829 ms. Trace:
W20260709 13:01:47.446941  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.408213  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:47.447831  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.190409  3555 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:47.424263  3660 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.183245  3589 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.266s
I20260709 13:01:47.460353  1736 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 4 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 13:01:47.416258  3594 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.207361  3640 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.168370  3586 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.287s
I20260709 13:01:47.381428  3605 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:47.483412  3582 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.633s	user 0.012s	sys 0.000s
I20260709 13:01:47.484694  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1307) took 1961 ms. Trace:
I20260709 13:01:47.415390  1586 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:47.423419  3671 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260709 13:01:47.462266  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:47.777036  1742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.013s	sys 0.013s
W20260709 13:01:48.022537  1742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.900s	user 0.019s	sys 0.016s
I20260709 13:01:47.492729  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:47.570245  1591 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:47.494223  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1862) took 1486 ms. Trace:
I20260709 13:01:47.478255  3664 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.026984  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 244) took 1589 ms. Trace:
W20260709 13:01:47.765717  3583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.011s	sys 0.010s
I20260709 13:01:47.541761  1587 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:48.027767  1666 rpcz_store.cc:276] 0709 13:01:46.437948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.438084 (+   136us) service_pool.cc:224] Handling call
0709 13:01:46.438440 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.481764 (+ 43324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.481805 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.482403 (+   598us) log.cc:844] Serialized 26 byte log entry
0709 13:01:46.674872 (+192469us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:46.674912 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.674987 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.179421 (+504434us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:47.669420 (+489999us) raft_consensus.cc:1685] finished
0709 13:01:48.026561 (+357141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.026888 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14958,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"spinlock_wait_cycles":45149952,"thread_start_us":192023,"threads_started":1,"wal-append.queue_time_us":323386}
I20260709 13:01:48.029163  3582 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:47.476171  3657 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:47.915719  3584 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.010s	sys 0.025s
I20260709 13:01:48.030251  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:48.030694  3584 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s	user 0.011s	sys 0.027s
I20260709 13:01:48.030413  3657 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.031242  1586 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:48.031311  3392 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s	user 0.026s	sys 0.040s
I20260709 13:01:48.031657  3582 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:47.820830  3347 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:47.884003  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.014s	sys 0.026s
W20260709 13:01:48.032740  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.014s	sys 0.032s
I20260709 13:01:48.032578  3347 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.033241  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.033188  3555 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.034070  3594 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.034706  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.035835  1736 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:47.474006  3614 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:47.498831  3496 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:47.493742  3600 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:48.028172  3583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.021s	sys 0.013s
I20260709 13:01:47.835335  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.038427  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:48.038388  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.007s	sys 0.023s
W20260709 13:01:47.642244  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:47.671501  3284 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.037220  1667 rpcz_store.cc:276] 0709 13:01:45.523337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.534571 (+ 11234us) service_pool.cc:224] Handling call
0709 13:01:45.534993 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:45.535285 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:45.672299 (+137014us) log.cc:844] Serialized 14 byte log entry
0709 13:01:47.484020 (+1811721us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b18003ccd60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a
0709 13:01:47.484254 (+   234us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:47.484312 (+    58us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:47.484370 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.484462 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.484606 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":238591,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":4029,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"spinlock_wait_cycles":7155456,"thread_start_us":238979,"threads_started":2,"wal-append.queue_time_us":200918,"wal-append.run_cpu_time_us":2348,"wal-append.run_wall_time_us":1204890}
I20260709 13:01:48.032173  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.041489  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.037801  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.026021  1737 rpcz_store.cc:276] 0709 13:01:46.007778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.878468 (+870690us) service_pool.cc:224] Handling call
0709 13:01:46.878858 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.879260 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.879292 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.910153 (+ 30861us) log.cc:844] Serialized 26 byte log entry
0709 13:01:47.467585 (+557432us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:47.467657 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.467768 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.493980 (+ 26212us) raft_consensus.cc:1685] finished
0709 13:01:47.494003 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:47.494163 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3663,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"thread_start_us":487351,"threads_started":1,"wal-append.queue_time_us":522027}
I20260709 13:01:48.076670  1586 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:48.035053  3347 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:47.686580  3405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.008s	sys 0.028s
I20260709 13:01:48.047646  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 236) took 1552 ms. Trace:
W20260709 13:01:47.981784  2310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.003s	sys 0.010s
W20260709 13:01:48.031714  3392 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.980s	user 0.026s	sys 0.040s
I20260709 13:01:48.032231  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1864) took 2023 ms. Trace:
W20260709 13:01:47.739471  1659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.013s	sys 0.007s
W20260709 13:01:48.023613  3646 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.009s	sys 0.037s
I20260709 13:01:47.738981  3642 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.038645  3598 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.965s	user 0.015s	sys 0.000s
I20260709 13:01:48.063534  3664 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:48.065208  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.008s	sys 0.024s
W20260709 13:01:48.106208  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 647122us
I20260709 13:01:48.072221  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.076675  1591 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
W20260709 13:01:48.066861  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 604731us
I20260709 13:01:48.145573  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:47.787001  3314 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.036442  3594 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.086318  1742 rpcz_store.cc:276] 0709 13:01:46.495298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.108156 (+612858us) service_pool.cc:224] Handling call
0709 13:01:47.108539 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.108795 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:47.121763 (+ 12968us) log.cc:844] Serialized 14 byte log entry
0709 13:01:48.047071 (+925308us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:48.047147 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:48.047250 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.047359 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.047553 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13251,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2605,"thread_start_us":901138,"threads_started":1}
I20260709 13:01:48.091085  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.149092  1589 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:48.149585  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2004) took 1156 ms. Trace:
I20260709 13:01:48.149995  1589 rpcz_store.cc:276] 0709 13:01:46.992982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.090748 (+1097766us) service_pool.cc:224] Handling call
0709 13:01:48.149475 (+ 58727us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:47.476845  3585 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.593s
I20260709 13:01:48.092856  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.055656  3640 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.089375  1593 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.634s	user 0.007s	sys 0.000s
I20260709 13:01:48.160593  1593 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:48.082057  3658 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.025s	user 0.013s	sys 0.000s
I20260709 13:01:48.036114  3555 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:48.167527  1586 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:48.177908  1593 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:48.179081  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2013) took 1007 ms. Trace:
I20260709 13:01:48.179533  1593 rpcz_store.cc:276] 0709 13:01:47.171423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.177154 (+1005731us) service_pool.cc:224] Handling call
0709 13:01:48.177662 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.178673 (+  1011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.178993 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:48.095026  1734 rpcz_store.cc:276] 0709 13:01:46.008397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.911128 (+902731us) service_pool.cc:224] Handling call
0709 13:01:46.913232 (+  2104us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.920964 (+  7732us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.920998 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.921843 (+   845us) log.cc:844] Serialized 26 byte log entry
0709 13:01:47.352852 (+431009us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:47.352916 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:47.353035 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:47.791598 (+438563us) raft_consensus.cc:1685] finished
0709 13:01:47.791624 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.032158 (+240534us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7576,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":14426,"thread_start_us":353193,"threads_started":1,"wal-append.queue_time_us":707355}
W20260709 13:01:48.086140  3405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s	user 0.015s	sys 0.038s
W20260709 13:01:48.119660  1708 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.014s	sys 0.030s
W20260709 13:01:48.182566  1708 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.015s	sys 0.038s
W20260709 13:01:48.182981  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 790970us
I20260709 13:01:48.183375  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:48.183917  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.539s (SENT)
I20260709 13:01:48.184170  1591 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:48.114517  3598 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.188575  3598 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:48.097676  3646 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.010s	sys 0.037s
I20260709 13:01:48.165930  3594 raft_consensus.cc:2764] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 13:01:48.171810  3640 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.194918  3614 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.195699  3640 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.196735  3555 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.197736  3555 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.200678  3555 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:48.151193  3682 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:01:48.105199  3642 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.098352  1659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.995s	user 0.019s	sys 0.011s
I20260709 13:01:48.206872  3657 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:48.085691  3582 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.208336  3642 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.207993  3664 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.083031  3660 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:48.213227  3660 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:48.213816  3660 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.036484  1488 rpcz_store.cc:276] 0709 13:01:38.614384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:38.626869 (+ 12485us) service_pool.cc:224] Handling call
0709 13:01:40.775112 (+2148243us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf47090 00007fcdedfe5f3a 00007fcdf3d9c1cc 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc 00007fcdf235a392 00007fcdec31848c 00007fcdec3178c6
0709 13:01:47.443601 (+6668489us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:44.550514 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:44.621036 (+ 70522us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:44.621099 (+    63us) write_op.cc:435] Acquired schema lock
0709 13:01:44.621126 (+    27us) tablet.cc:662] Decoding operations
0709 13:01:44.621588 (+   462us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:44.621669 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:44.621698 (+    29us) tablet.cc:685] Acquiring locks for 1 operations
0709 13:01:44.622101 (+   403us) tablet.cc:701] Row locks acquired
0709 13:01:44.622150 (+    49us) write_op.cc:260] PREPARE: finished
0709 13:01:44.622450 (+   300us) write_op.cc:270] Start()
0709 13:01:44.622620 (+   170us) write_op.cc:276] Timestamp: P: 1783602104622262 usec, L: 0
0709 13:01:44.622646 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 13:01:44.623404 (+   758us) log.cc:844] Serialized 536 byte log entry
0709 13:01:45.801334 (+1177930us) op_driver.cc:464] REPLICATION: finished
0709 13:01:46.027897 (+226563us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdeaed30ac 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdf1a13ef0
0709 13:01:46.568147 (+540250us) write_op.cc:301] APPLY: starting
0709 13:01:46.606492 (+ 38345us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:46.631015 (+ 24523us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:46.662090 (+ 31075us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:46.742932 (+ 80842us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:46.743356 (+   424us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 13:01:46.743410 (+    54us) write_op.cc:312] APPLY: finished
0709 13:01:46.743765 (+   355us) log.cc:844] Serialized 23 byte log entry
0709 13:01:47.001144 (+257379us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:47.001273 (+   129us) write_op.cc:454] Released schema lock
0709 13:01:47.001354 (+    81us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:47.067038 (+ 65684us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:47.067069 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2083270,"reported_tablets":5,"spinlock_wait_cycles":1303066368,"child_traces":[["op",{"apply.queue_time_us":694275,"apply.run_cpu_time_us":26507,"apply.run_wall_time_us":481299,"num_ops":1,"prepare.queue_time_us":468951,"prepare.run_cpu_time_us":20724,"prepare.run_wall_time_us":292698,"raft.queue_time_us":388573,"raft.run_cpu_time_us":66723,"raft.run_wall_time_us":602408,"replication_time_us":1154208,"spinlock_wait_cycles":74679296,"thread_start_us":1337604,"threads_started":5,"wal-append.queue_time_us":648953,"wal-append.run_cpu_time_us":18447,"wal-append.run_wall_time_us":1797680}]]}
W20260709 13:01:48.111917  3575 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.044s	user 0.014s	sys 0.000s
I20260709 13:01:48.147541  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:48.096903  3347 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:47.528873  3625 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.136819  3622 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.796s	user 0.014s	sys 0.000s
I20260709 13:01:48.071161  3600 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.086541  3284 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.186123  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.588s (SENT)
I20260709 13:01:48.091125  3584 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:01:48.092000  2310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.004s	sys 0.011s
W20260709 13:01:48.190200  3677 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
W20260709 13:01:48.425815  3662 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.208s	user 0.005s	sys 0.000s
I20260709 13:01:48.190061  1668 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:48.425367  3584 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:48.427066  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1320) took 1554 ms. Trace:
I20260709 13:01:48.427471  1668 rpcz_store.cc:276] 0709 13:01:46.872208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.014864 (+142656us) service_pool.cc:224] Handling call
0709 13:01:48.426868 (+1412004us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:48.427000 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2626548608}
I20260709 13:01:48.207974  3582 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.116801  1735 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 13:01:48.396327  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:48.364903  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1317) took 1493 ms. Trace:
I20260709 13:01:48.173753  3658 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.437616  1735 pending_rounds.cc:85] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:01:48.432253  3284 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.438002  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.437965  1735 pending_rounds.cc:107] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:01:48.439041  1735 raft_consensus.cc:2889] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:48.439877  3658 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.430064  3582 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:48.201417  3587 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.711s	user 0.006s	sys 0.001s
I20260709 13:01:48.220114  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2012) took 1049 ms. Trace:
I20260709 13:01:48.237146  3575 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.312978  1736 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:48.332645  3586 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:48.181486  1593 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:48.461594  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2018) took 1288 ms. Trace:
I20260709 13:01:48.415308  3187 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.324383  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.184631  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2011) took 1013 ms. Trace:
I20260709 13:01:48.334262  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1857) took 2327 ms. Trace:
I20260709 13:01:48.383728  2314 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:01:48.391709  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 983594us
W20260709 13:01:48.380462  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.022s	sys 0.076s
I20260709 13:01:48.315523  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1910) took 1787 ms. Trace:
W20260709 13:01:48.484007  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.026s	sys 0.080s
I20260709 13:01:48.413362  1665 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:48.228360  3496 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.486003  1734 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:01:48.489652  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:48.490483  1668 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:48.486105  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 240) took 1919 ms. Trace:
I20260709 13:01:48.486686  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.486948  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:48.493089  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.493834  3664 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.493965  1666 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:48.494539  3664 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:48.494697  3578 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:48.487373  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:48.484712  3187 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.495949  1664 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:48.495515  3664 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.419080  3347 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.491492  1738 rpcz_store.cc:276] 0709 13:01:46.566853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.228218 (+661365us) service_pool.cc:224] Handling call
0709 13:01:47.228739 (+   521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.261410 (+ 32671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:47.261462 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.382456 (+120994us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.485614 (+1103158us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:48.485666 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.485733 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.485771 (+    38us) raft_consensus.cc:1685] finished
0709 13:01:48.485783 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.486041 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":145,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":173,"spinlock_wait_cycles":27392,"thread_start_us":1031308,"threads_started":1,"wal-append.queue_time_us":1013771}
I20260709 13:01:48.497030  3694 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:48.497378  3187 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.497968  3187 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.498272  3347 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.423134  3622 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.498451  3187 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:48.424930  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 238868us
W20260709 13:01:48.500418  1708 leader_election.cc:336] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.595s (SENT)
I20260709 13:01:48.500826  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.501176  3187 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.501874  3610 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:48.335932  3564 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.846s	user 0.008s	sys 0.004s
I20260709 13:01:48.430323  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1846) took 1875 ms. Trace:
I20260709 13:01:48.488013  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.427304  3584 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.502074  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.429412  3600 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.416145  3598 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:01:48.502036  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.602s (SENT)
I20260709 13:01:48.452935  3411 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.455564  1586 rpcz_store.cc:276] 0709 13:01:47.171117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.166787 (+995670us) service_pool.cc:224] Handling call
0709 13:01:48.167263 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.219486 (+ 52223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.220040 (+   554us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:48.437178  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2008) took 1336 ms. Trace:
I20260709 13:01:48.507032  1587 rpcz_store.cc:276] 0709 13:01:47.101150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.331020 (+229870us) service_pool.cc:224] Handling call
0709 13:01:48.436979 (+1105959us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:48.437111 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482297088}
I20260709 13:01:48.507392  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:48.434682  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.508100  1586 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:48.462007  1593 rpcz_store.cc:276] 0709 13:01:47.172837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.180800 (+1007963us) service_pool.cc:224] Handling call
0709 13:01:48.181217 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.461228 (+280011us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.461493 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:48.182633  3684 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:48.508983  3684 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:48.509585  3702 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.509953  3684 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:48.508294  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.511102  3684 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:48.512193  3654 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.677s	user 0.002s	sys 0.000s
I20260709 13:01:48.512269  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.513214  1587 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.502760  3564 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.516927  3564 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.509214  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.462718  3682 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.462934  3585 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.485375  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.526865  3585 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.528591  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.529085  3585 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.529367  1661 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:48.530522  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.531546  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:48.532294  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.532791  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.533464  1593 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:48.414875  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 267500us
I20260709 13:01:48.415087  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:48.534351  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.535023  2311 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:48.535045  1593 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:48.536549  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.541570  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:48.449355  3696 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:01:48.503502  3584 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:48.443274  3625 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:48.504220  1667 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:48.503796  3598 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.502830  1590 rpcz_store.cc:276] 0709 13:01:46.555361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.555535 (+   174us) service_pool.cc:224] Handling call
0709 13:01:46.583292 (+ 27757us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.618797 (+ 35505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.618852 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.652787 (+ 33935us) log.cc:844] Serialized 26 byte log entry
0709 13:01:46.920122 (+267335us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:46.920175 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.920248 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.425199 (+1504951us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:48.429994 (+  4795us) raft_consensus.cc:1685] finished
0709 13:01:48.430021 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.430258 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4799,"raft.queue_time_us":141,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":21093,"spinlock_wait_cycles":23209856,"thread_start_us":225403,"threads_started":1,"wal-append.queue_time_us":448773}
I20260709 13:01:48.465411  1741 rpcz_store.cc:276] 0709 13:01:46.006386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.421437 (+415051us) service_pool.cc:224] Handling call
0709 13:01:46.476906 (+ 55469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:47.016821 (+539915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.333961 (+1317140us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b5400338954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:48.334194 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1233344896}
I20260709 13:01:48.457645  3583 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.487540  1736 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:48.457880  3605 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.510105  3702 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.559983  1588 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:48.455523  1663 rpcz_store.cc:276] 0709 13:01:46.871420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.335188 (+463768us) service_pool.cc:224] Handling call
0709 13:01:47.335515 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.335885 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:47.335911 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.336465 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.033245 (+696780us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:48.033289 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.033361 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.364591 (+331230us) raft_consensus.cc:1685] finished
0709 13:01:48.364617 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.364820 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2195,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":205,"thread_start_us":696383,"threads_started":1,"wal-append.queue_time_us":962119}
W20260709 13:01:48.520073  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1914) took 1961 ms (client timeout 1662 ms). Trace:
W20260709 13:01:48.561755  1660 rpcz_store.cc:269] 0709 13:01:46.558823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.585304 (+ 26481us) service_pool.cc:224] Handling call
0709 13:01:47.335093 (+749789us) spinlock_profiling.cc:243] Waited 5.75 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:48.519884 (+1184791us) spinlock_profiling.cc:243] Waited 78.8 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:48.519994 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":194531072}
I20260709 13:01:48.563155  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:48.564060  1660 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:48.564941  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:48.435890  3672 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.257s	user 0.001s	sys 0.000s
I20260709 13:01:48.567716  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1859) took 1749 ms. Trace:
W20260709 13:01:48.568133  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.364569  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:48.568984  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 349958us
I20260709 13:01:48.460647  3698 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.569851  3698 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.568135  1585 rpcz_store.cc:276] 0709 13:01:46.818673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.828275 (+  9602us) service_pool.cc:224] Handling call
0709 13:01:46.828687 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.843428 (+ 14741us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:46.843467 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:46.843983 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 13:01:46.999505 (+155522us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:46.999565 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:46.999639 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.435198 (+1435559us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:48.567366 (+132168us) raft_consensus.cc:1685] finished
0709 13:01:48.567392 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.567647 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14139,"raft.queue_time_us":17458,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":7067392,"thread_start_us":155131,"threads_started":1,"wal-append.queue_time_us":330380}
I20260709 13:01:48.571527  3314 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.572142  3314 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:48.572687  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.484375  3699 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260709 13:01:48.573105  3314 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.574620  3598 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.485242  3658 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.576506  3658 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.481660  3339 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.577772  3339 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.578665  3658 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.526139  3575 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.533702  1668 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:48.579957  3339 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.482597  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 564) took 1106 ms. Trace:
I20260709 13:01:48.581139  1668 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:48.539155  3405 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:48.582149  3622 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.582415  3405 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.539935  1664 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
W20260709 13:01:48.420680  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 353180us
I20260709 13:01:48.583715  1660 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:48.558430  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.584512  1590 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:48.537788  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:48.542204  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.465344  1591 rpcz_store.cc:276] 0709 13:01:47.170820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.156475 (+985655us) service_pool.cc:224] Handling call
0709 13:01:48.184562 (+ 28087us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:48.585614  1740 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:48.586777  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49087.8us
I20260709 13:01:48.587720  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:48.525725  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.544023  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:48.522243  3612 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:48.546038  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44063.5us
I20260709 13:01:48.546303  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1876) took 1257 ms. Trace:
I20260709 13:01:48.603531  1739 rpcz_store.cc:276] 0709 13:01:47.289313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.455293 (+165980us) service_pool.cc:224] Handling call
0709 13:01:47.455637 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.908514 (+452877us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:47.908554 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.909171 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.136059 (+226888us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:48.136138 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.136233 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.545547 (+409314us) raft_consensus.cc:1685] finished
0709 13:01:48.545567 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.546218 (+   651us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33003,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":678798,"threads_started":2,"wal-append.queue_time_us":390989}
I20260709 13:01:48.543560  1661 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:48.508001  3582 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:48.461608  3587 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.506248  1666 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:01:48.608158  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.608692  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.556643  3583 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.558374  3660 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.561210  3702 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.620913  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.539798  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:48.490187  3284 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.584956  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.636641  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:48.600661  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:48.601109  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.520987  3606 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 13:01:48.590379  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:48.609392  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.650766  1668 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.651062  1587 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:48.612069  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.612823  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.613901  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.651885  1740 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:48.652181  2314 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:48.621377  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:48.623459  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83762.1us
I20260709 13:01:48.653951  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.654942  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.655831  1587 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:48.573772  1585 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:48.632356  3699 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.660054  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.633154  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.633240  3622 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:48.634001  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.634326  3614 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:48.635259  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.469385  3586 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.544438  3703 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:48.661860  2313 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:01:48.662554  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49864.7us
I20260709 13:01:48.674403  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.667382  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.675318  1587 raft_consensus.cc:2410] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:01:48.670207  2317 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:01:48.675222  1591 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:48.677045  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.677901  1590 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:48.679029  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.679610  1590 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.665555  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:48.613027  3640 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.666546  1584 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 6. Ops: []
I20260709 13:01:48.612367  3575 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:48.617659  3339 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.675113  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:48.705880  1668 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:48.707093  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53511.5us
I20260709 13:01:48.708515  3716 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:01:48.711195  2318 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:48.711867  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:48.712431  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.713058  1736 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:48.714449  1593 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:01:48.719091  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44111.8us
I20260709 13:01:48.719537  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.720654  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.726573  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.728494  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:48.660921  3601 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:48.737838  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.738525  2314 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:48.730132  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62625.9us
I20260709 13:01:48.730573  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.740099  1585 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:48.740003  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.732818  1736 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:48.741222  1740 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:48.733858  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.741945  1733 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
I20260709 13:01:48.744354  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:48.536566  3682 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.666400  3658 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:48.747458  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.747691  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:48.747932  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.672926  3614 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.672822  3700 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:48.673975  3670 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:48.750888  2312 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:48.751590  3614 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.750799  1740 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:48.689697  3610 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:48.666587  3690 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:48.669574  3597 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:48.617374  3582 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:48.660771  3284 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.685316  3702 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
I20260709 13:01:48.650254  3612 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.702904  3600 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.103s
W20260709 13:01:48.777819  1557 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.325s (SENT)
I20260709 13:01:48.754966  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:48.843138  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65434us
W20260709 13:01:48.759230  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93810.1us
I20260709 13:01:48.843461  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.845286  1591 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:48.782507  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.806360  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.846712  1740 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:48.846664  1587 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:48.825700  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.817580  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:48.848929  1733 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:48.809983  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.750736  1738 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:48.704869  3555 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:48.886948  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 77052.6us
I20260709 13:01:48.888943  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:48.714913  3392 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:48.703279  3575 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.895856  1733 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:48.898181  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.900866  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:48.744853  3496 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:48.725978  3640 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:48.907832  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:48.917642  1591 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:48.918048  1587 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
I20260709 13:01:48.918702  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2020) took 1745 ms. Trace:
I20260709 13:01:48.745249  3314 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:48.919009  1587 rpcz_store.cc:276] 0709 13:01:47.173403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.917380 (+1743977us) service_pool.cc:224] Handling call
0709 13:01:48.917775 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:48.918429 (+   654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.918645 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:48.919622  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.906914  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.920256  1742 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:48.673089  3699 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.709851  3339 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.852340  3646 raft_consensus.cc:670] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:48.786710  3564 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:48.912396  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:48.942732  1591 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:01:48.853399  3605 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.869736  3658 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.704831  3703 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.924610  3718 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.823665  3562 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:48.934742  3405 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:48.937104  1738 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:48.899786  3606 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 13:01:48.943162  1591 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
W20260709 13:01:48.975764  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105058us
I20260709 13:01:48.976338  1591 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:01:48.904688  3596 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:48.902184  3575 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:48.580613  2310 rpcz_store.cc:276] 0709 13:01:47.375822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.390211 (+ 14389us) service_pool.cc:224] Handling call
0709 13:01:47.392750 (+  2539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:47.425760 (+ 33010us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:47.425798 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.426384 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.425741 (+999357us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:48.425799 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.425890 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.455255 (+ 29365us) raft_consensus.cc:1685] finished
0709 13:01:48.457605 (+  2350us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.465268 (+  7663us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11447,"raft.queue_time_us":60134,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":19423,"thread_start_us":998898,"threads_started":1,"wal-append.queue_time_us":889340}
I20260709 13:01:48.922166  3314 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.904961  3392 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:48.962669  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:48.986217  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.895313  3610 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:48.938531  3717 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
W20260709 13:01:48.971633  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82582.5us
I20260709 13:01:48.976711  1591 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:48.996230  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2021) took 1802 ms. Trace:
I20260709 13:01:48.996632  1591 rpcz_store.cc:276] 0709 13:01:47.193536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.918329 (+1724793us) service_pool.cc:224] Handling call
0709 13:01:48.942376 (+ 24047us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.995927 (+ 53551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.996167 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:48.962225  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 233799us
I20260709 13:01:48.952463  3658 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:48.484196  1659 rpcz_store.cc:276] 0709 13:01:46.524175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.532131 (+  7956us) service_pool.cc:224] Handling call
0709 13:01:46.626335 (+ 94204us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:46.944071 (+317736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:46.950376 (+  6305us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:47.049298 (+ 98922us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.207222 (+1157924us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:48.207294 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.207403 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:48.292453 (+ 85050us) raft_consensus.cc:1685] finished
0709 13:01:48.292527 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:48.311599 (+ 19072us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25337,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":8873,"thread_start_us":1341825,"threads_started":2,"wal-append.queue_time_us":1160028}
I20260709 13:01:48.909798  3555 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:48.993577  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.003275  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.972816  3564 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:48.885188  3690 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:48.978583  3596 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:48.894088  3597 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:48.981328  3718 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:48.983731  3670 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:48.982587  3702 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:49.015264  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:48.986490  3314 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.045888  3564 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:49.046186  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 249) took 2340 ms. Trace:
I20260709 13:01:49.046528  3582 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.047170  3564 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:49.049160  3708 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.662s	user 0.002s	sys 0.000s
I20260709 13:01:48.983317  3720 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:48.985247  3405 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 13:01:49.042507  3610 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:49.043794  3712 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 13:01:49.033332  3670 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.024938  1736 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:49.051465  3714 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.345s	user 0.000s	sys 0.001s
I20260709 13:01:49.053411  3690 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:49.146533  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.055313  1590 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:49.055429  1585 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:49.148144  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2031) took 1579 ms. Trace:
I20260709 13:01:49.148594  1585 rpcz_store.cc:276] 0709 13:01:47.569159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.054616 (+1485457us) service_pool.cc:224] Handling call
0709 13:01:49.055041 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.147655 (+ 92614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.147886 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:49.042407  3699 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:49.057020  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.150040  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.151912  1587 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:49.059794  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2017) took 1887 ms. Trace:
I20260709 13:01:49.061822  3640 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.065837  3612 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:49.067153  3339 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:49.161864  1634 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:49.066861  3721 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:49.110450  3670 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:49.068200  3586 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.068405  3598 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.067765  3314 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:49.069554  3392 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:49.023478  3596 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.070520  3555 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:49.002068  3660 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:49.070963  3707 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.438s	user 0.001s	sys 0.000s
I20260709 13:01:49.071393  3718 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:49.074879  3589 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:49.075829  3584 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.525s	user 0.009s	sys 0.000s
I20260709 13:01:49.076333  3702 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.077051  3284 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:49.082018  3605 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:49.083101  3583 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:49.084554  3711 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 13:01:49.085007  3704 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.716s	user 0.001s	sys 0.001s
I20260709 13:01:49.037894  3700 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 13:01:49.086256  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:49.087602  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:49.088748  1664 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:49.090075  3568 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:49.216063  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:49.239702  3568 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:49.240952  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:49.090271  3682 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.241104  3568 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:49.090986  3722 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:01:49.092105  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.242190  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:49.242645  3568 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:49.242974  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.243044  3405 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:49.243520  3405 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:49.243765  3584 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:49.244197  3584 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:49.046731  1665 rpcz_store.cc:276] 0709 13:01:46.705744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.319310 (+613566us) service_pool.cc:224] Handling call
0709 13:01:47.319706 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.484615 (+1164909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.045922 (+561307us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:49.046120 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2493356928}
I20260709 13:01:49.244300  3405 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:49.044646  3709 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:01:49.245529  3405 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.105286  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:49.153728  1584 rpcz_store.cc:276] 0709 13:01:47.172574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.171415 (+998841us) service_pool.cc:224] Handling call
0709 13:01:48.568268 (+396853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:48.691136 (+122868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.058707 (+367571us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b5400443254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:49.059564 (+   857us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400443250. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:49.059726 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1046573184}
I20260709 13:01:49.248363  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.248974  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:49.067575  3587 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.164774  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.167117  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2027) took 1715 ms. Trace:
I20260709 13:01:49.249815  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.250761  1741 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:49.250041  1591 rpcz_store.cc:276] 0709 13:01:47.452213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.997459 (+1545246us) service_pool.cc:224] Handling call
0709 13:01:49.023028 (+ 25569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.166431 (+143403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.166479 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:49.166553 (+    74us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:49.166714 (+   161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.166820 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.167036 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":157171,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":6749,"thread_start_us":142817,"threads_started":1}
W20260709 13:01:49.251623  3665 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 2.041s	user 0.003s	sys 0.000s
I20260709 13:01:49.252393  3564 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:01:49.188777  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 496325us
W20260709 13:01:49.192436  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 195942us
I20260709 13:01:49.255024  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:49.255653  1734 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.156265  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:49.164361  3585 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:49.205185  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:49.258896  3598 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.205777  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.165957  3187 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:49.160920  3562 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:01:49.219357  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.222234  1740 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:49.218765  1586 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:49.168592  3496 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:49.195267  3729 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:49.259235  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2016) took 2087 ms (client timeout 2111 ms). Trace:
I20260709 13:01:49.259598  1589 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:49.247062  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1880) took 1920 ms. Trace:
I20260709 13:01:49.245087  3584 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.204584  1736 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:49.211123  3589 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:49.232724  3684 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.721s	user 0.012s	sys 0.006s
I20260709 13:01:49.251653  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.263739  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:49.261387  1738 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:49.361652  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:49.320820  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:49.368224  1666 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:49.264196  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:49.371098  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:49.373426  1742 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:49.365530  1737 rpcz_store.cc:276] 0709 13:01:47.327010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.085092 (+758082us) service_pool.cc:224] Handling call
0709 13:01:48.085479 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:48.380775 (+295296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:48.380812 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:48.381553 (+   741us) log.cc:844] Serialized 26 byte log entry
0709 13:01:48.665996 (+284443us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:48.666037 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:48.666114 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:49.246835 (+580721us) raft_consensus.cc:1685] finished
0709 13:01:49.246855 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.246996 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115469,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":578646,"threads_started":2,"wal-append.queue_time_us":461025}
I20260709 13:01:49.364120  3589 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:49.378872  3589 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.381227  3597 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:49.382102  1659 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:49.383468  3729 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:49.385686  2315 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:49.382021  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:49.366709  1741 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
W20260709 13:01:49.363131  1588 rpcz_store.cc:269] 0709 13:01:47.172280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.038143 (+865863us) service_pool.cc:224] Handling call
0709 13:01:49.259008 (+1220865us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b54003b79d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:49.259170 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1138461568}
W20260709 13:01:49.337817  3712 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
W20260709 13:01:49.398453  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1315) took 2527 ms (client timeout 2030 ms). Trace:
W20260709 13:01:49.398989  1662 rpcz_store.cc:269] 0709 13:01:46.870858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:46.957039 (+ 86181us) service_pool.cc:224] Handling call
0709 13:01:49.398161 (+2441122us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b540013ddd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:49.398324 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":374150528}
I20260709 13:01:49.400831  3703 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:49.261116  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 96426.1us
I20260709 13:01:49.402334  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:49.403816  3314 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:49.403936  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.405104  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.405316  3600 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:49.405784  3411 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:49.406284  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.393448  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:49.405386  3614 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.653s	user 0.012s	sys 0.000s
I20260709 13:01:49.259268  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.406445  3600 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:49.407752  3614 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.408420  3600 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.408883  3600 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:49.406482  3411 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:49.410012  3614 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:49.362901  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73745.3us
I20260709 13:01:49.371294  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:49.415942  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:49.407770  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.473129  1588 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.430272  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.455397  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:49.456868  1665 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:01:49.473313  1737 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:49.383630  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:49.459735  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47399.9us
W20260709 13:01:49.462265  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.463595  1661 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:49.497267  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.463557  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:49.466416  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:49.498118  1738 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:49.364861  3187 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:49.471238  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.411330  1735 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:49.472703  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2026) took 2081 ms. Trace:
I20260709 13:01:49.300292  3682 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:49.479725  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.480787  1740 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:49.499729  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:49.523819  3187 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:49.527801  3735 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:01:49.532814  3702 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.464514  3598 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:49.524053  3597 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:49.533658  3598 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:49.534186  3598 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.534679  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:49.536708  1738 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:49.537824  1659 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:49.537861  3597 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:49.539114  3625 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.985s	user 0.013s	sys 0.000s
I20260709 13:01:49.540272  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:49.536701  3187 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.543222  1668 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:49.544255  3187 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:49.545190  1738 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:01:49.548902  3625 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:49.548362  3601 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.798s	user 0.013s	sys 0.000s
I20260709 13:01:49.549845  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:49.550658  1741 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:49.550858  1592 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:49.552070  1741 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 13:01:49.552677  1738 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:01:49.503471  3584 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:49.561585  1737 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 13:01:49.557924  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.022s	sys 0.017s
W20260709 13:01:49.562668  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.022s	sys 0.017s
I20260709 13:01:49.564085  1587 rpcz_store.cc:276] 0709 13:01:47.391201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.932131 (+1540930us) service_pool.cc:224] Handling call
0709 13:01:49.151707 (+219576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.152462 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.461632 (+309170us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540033aed0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:49.472631 (+ 10999us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437867776}
I20260709 13:01:49.499512  1734 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:49.481967  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:49.480509  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.481132  1665 pending_rounds.cc:85] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 1
I20260709 13:01:49.410566  3411 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.386469  3562 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:49.486805  1588 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:49.471869  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.592002  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.568116  1665 pending_rounds.cc:107] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:01:49.608984  1665 raft_consensus.cc:2889] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:49.576293  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 821453us
W20260709 13:01:49.576738  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.576942  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 215421us
I20260709 13:01:49.298233  3585 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:49.578871  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 152189us
I20260709 13:01:49.579710  1742 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:49.583206  3731 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.584321  3587 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.584915  3405 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:49.585933  3746 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:49.586863  3625 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:49.588351  3583 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:49.613860  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:49.615239  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:49.616604  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:49.839955  1584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.018s	sys 0.061s
W20260709 13:01:50.163200  1584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s	user 0.032s	sys 0.076s
I20260709 13:01:49.560747  3578 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:49.525792  3682 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:50.139755  1585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.004s	sys 0.012s
W20260709 13:01:50.168017  1585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.005s	sys 0.012s
I20260709 13:01:49.590257  3690 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 13:01:49.671442  1587 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:49.553357  3496 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:49.694993  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:50.182565  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:50.183276  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:49.701171  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81551.9us
I20260709 13:01:50.184734  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:49.945987  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1862) took 1619 ms. Trace:
I20260709 13:01:50.185680  1590 rpcz_store.cc:276] 0709 13:01:48.326271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.148293 (+822022us) service_pool.cc:224] Handling call
0709 13:01:49.148945 (+   652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.347606 (+198661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.347690 (+    84us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.363907 (+ 16217us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.376351 (+ 12444us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:49.376451 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.376561 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:49.937362 (+560801us) raft_consensus.cc:1685] finished
0709 13:01:49.937395 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:49.945749 (+  8354us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92843,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":209053,"threads_started":2,"wal-append.queue_time_us":570889}
I20260709 13:01:50.153687  1734 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:50.189935  1734 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:50.191586  1734 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:50.153668  1556 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.030s (SENT) [suppressed 29 similar messages]
W20260709 13:01:50.153170  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 513004us
W20260709 13:01:49.870980  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:50.042972  1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 1879) took 1596 ms (client timeout 1652 ms). Trace:
I20260709 13:01:50.141398  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:50.154935  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 673271us
W20260709 13:01:50.040565  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2025) took 2672 ms (client timeout 1609 ms). Trace:
I20260709 13:01:50.194142  2317 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 13:01:50.194324  1592 rpcz_store.cc:269] 0709 13:01:47.368369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:47.384404 (+ 16035us) service_pool.cc:224] Handling call
0709 13:01:50.040188 (+2655784us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b5400002654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:50.040408 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4927247616}
I20260709 13:01:50.196171  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 529) took 1509 ms. Trace:
W20260709 13:01:49.880908  3605 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.027s	sys 0.058s
I20260709 13:01:49.662948  1735 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:49.617926  1742 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:50.154524  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.211647  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:49.946584  1662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.005s	sys 0.008s
W20260709 13:01:50.212808  1662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.010s	sys 0.011s
W20260709 13:01:50.142223  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.170766  1660 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 13:01:50.174088  3718 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.006s	sys 0.016s
W20260709 13:01:49.748579  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.178040  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 278) took 1570 ms. Trace:
W20260709 13:01:50.178953  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 585615us
I20260709 13:01:49.593215  3187 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:01:50.193347  1586 rpcz_store.cc:269] 0709 13:01:48.446656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.508911 (+ 62255us) service_pool.cc:224] Handling call
0709 13:01:48.584827 (+ 75916us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:50.042673 (+1457846us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b5400036bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:50.042806 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1367840640}
W20260709 13:01:50.040094  3743 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 13:01:50.197211  2317 rpcz_store.cc:276] 0709 13:01:48.687258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.687443 (+   185us) service_pool.cc:224] Handling call
0709 13:01:49.203711 (+516268us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:50.141229 (+937518us) spinlock_profiling.cc:243] Waited 8.86 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8f8c 00007fcdeaeb8ef6 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:50.196066 (+ 54837us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25433600}
I20260709 13:01:50.156199  3583 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:50.199698  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.254868  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.256366  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.257094  1590 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:50.258122  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.258844  1590 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:50.073796  3562 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:50.100960  3706 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.699s	user 0.005s	sys 0.000s
I20260709 13:01:49.618402  1740 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:49.616391  3597 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.249006  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:50.262703  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1344) took 1045 ms. Trace:
I20260709 13:01:50.264308  1740 rpcz_store.cc:276] 0709 13:01:49.217688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.563937 (+346249us) service_pool.cc:224] Handling call
0709 13:01:49.618071 (+ 54134us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:50.262408 (+644337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.262626 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:50.246840  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.176949  3682 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:50.177342  3658 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.154s	user 0.011s	sys 0.002s
I20260709 13:01:50.250504  1587 rpcz_store.cc:276] 0709 13:01:48.607699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.565564 (+957865us) service_pool.cc:224] Handling call
0709 13:01:49.671151 (+105587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.177346 (+506195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.177953 (+   607us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:50.251884  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:49.880347  3721 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.196604  3347 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:50.101135  3729 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:50.246032  3718 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.006s	sys 0.016s
W20260709 13:01:50.210798  3605 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.952s	user 0.049s	sys 0.061s
I20260709 13:01:49.900748  3600 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.673755  3585 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.272133  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:49.801836  3699 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.460s
I20260709 13:01:49.612368  3596 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:49.717696  3555 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.548s
I20260709 13:01:50.181354  3578 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.269819  3625 raft_consensus.cc:1064] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:50.180305  3587 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.317086  3594 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:50.317896  3594 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.318450  3600 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.318352  3594 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.318823  3587 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.264945  3583 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:50.319947  3600 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:50.320518  3600 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:50.320760  3594 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:50.177294  3405 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.269523  1586 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
W20260709 13:01:50.282464  3670 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.088s	user 0.013s	sys 0.000s
I20260709 13:01:50.178153  3731 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.283402  3721 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.317915  1592 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:50.357189  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 281) took 1112 ms. Trace:
I20260709 13:01:50.357913  1592 rpcz_store.cc:276] 0709 13:01:49.244765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.298374 (+1053609us) service_pool.cc:224] Handling call
0709 13:01:50.317581 (+ 19207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.356748 (+ 39167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.357102 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:50.323235  3555 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260709 13:01:50.298974  3575 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.309s	user 0.014s	sys 0.000s
I20260709 13:01:50.267773  3597 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.368727  3555 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:50.369669  3555 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:50.218201  3746 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.367586  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1887) took 1653 ms. Trace:
I20260709 13:01:50.371212  1741 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 13:01:50.383122  1741 pending_rounds.cc:85] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:01:50.383816  1741 pending_rounds.cc:107] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 13:01:50.384239  1741 raft_consensus.cc:2889] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:50.383037  1587 rpcz_store.cc:276] 0709 13:01:48.714122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.314231 (+1600109us) service_pool.cc:224] Handling call
0709 13:01:50.366827 (+ 52596us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.367101 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:50.367140 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:50.367202 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:50.367252 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.367327 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.367496 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:50.281344  3658 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.356391  3721 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:50.327415  3405 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:50.386276  3589 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.007s	user 0.011s	sys 0.007s
I20260709 13:01:50.403113  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 217) took 1692 ms. Trace:
I20260709 13:01:50.366422  3690 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:50.331123  3670 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.423403  1667 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:01:50.405942  2313 rpcz_store.cc:276] 0709 13:01:48.710507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.735804 (+ 25297us) service_pool.cc:224] Handling call
0709 13:01:48.782341 (+ 46537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:48.826906 (+ 44565us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:48.826941 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:48.836924 (+  9983us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.202682 (+365758us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:49.202753 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.375637 (+1172884us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b5400719654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:50.375720 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.375796 (+    76us) raft_consensus.cc:1685] finished
0709 13:01:50.375816 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.403007 (+ 27191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113805,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":9196,"spinlock_wait_cycles":101467520,"thread_start_us":365183,"threads_started":1,"wal-append.queue_time_us":509273}
I20260709 13:01:50.466980  3670 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:50.467841  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1342) took 1205 ms. Trace:
I20260709 13:01:50.469594  3718 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.470196  3718 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.471325  3597 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:50.471897  3597 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.472328  3597 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.468430  1662 rpcz_store.cc:276] 0709 13:01:49.262615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.399916 (+137301us) service_pool.cc:224] Handling call
0709 13:01:49.400359 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.400782 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.400817 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.401464 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.213204 (+811740us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:50.213247 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.213314 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.467537 (+254223us) raft_consensus.cc:1685] finished
0709 13:01:50.467574 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.467762 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4263,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"thread_start_us":805856,"threads_started":1,"wal-append.queue_time_us":1062657}
I20260709 13:01:50.474116  3718 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.294998  3347 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:50.382197  3575 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.409169  3690 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:50.552461  1664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.025s	user 0.012s	sys 0.029s
W20260709 13:01:50.583657  1664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.056s	user 0.012s	sys 0.029s
I20260709 13:01:50.321478  3600 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.548007  3690 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:50.599742  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 983218us
I20260709 13:01:50.537827  3583 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:50.598413  3642 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.041s	sys 0.087s
W20260709 13:01:50.597082  3640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.006s	sys 0.028s
W20260709 13:01:50.619536  1593 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.027s	user 0.018s	sys 0.025s
W20260709 13:01:50.619967  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 1.00626e+06us
W20260709 13:01:50.642391  1593 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.018s	sys 0.025s
I20260709 13:01:50.644524  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1861) took 2319 ms. Trace:
W20260709 13:01:50.628768  3640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.006s	sys 0.028s
W20260709 13:01:50.626272  3584 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.067s	user 0.016s	sys 0.000s
I20260709 13:01:50.618579  3597 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:50.645823  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:50.647789  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.642194  3718 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:50.675402  3734 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:01:50.659883  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.692935  3597 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.679848  3670 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:01:50.619261  3347 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.705318  3585 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.718051  1740 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 13:01:50.735543  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:50.718888  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:50.739585  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.008s	sys 0.008s
W20260709 13:01:50.777438  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.008s	sys 0.008s
I20260709 13:01:50.777382  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:01:50.777899  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 593314us
I20260709 13:01:50.757685  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 282) took 1492 ms. Trace:
W20260709 13:01:50.781103  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.005s	sys 0.013s
W20260709 13:01:50.781574  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.010s	sys 0.009s
W20260709 13:01:50.782123  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.008s	sys 0.010s
W20260709 13:01:50.768183  1737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.006s	sys 0.012s
W20260709 13:01:50.774605  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.013s	sys 0.020s
I20260709 13:01:50.746049  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1337) took 1485 ms. Trace:
W20260709 13:01:50.712028  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 66400.5us
W20260709 13:01:50.793154  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.013s	sys 0.024s
W20260709 13:01:50.722999  1556 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.000s	sys 0.046s
W20260709 13:01:50.795295  1556 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.002s	sys 0.048s
W20260709 13:01:50.796744  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 643119us
I20260709 13:01:50.797528  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:50.794862  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1341) took 1586 ms. Trace:
I20260709 13:01:50.793424  1664 rpcz_store.cc:276] 0709 13:01:49.261077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.387321 (+126244us) service_pool.cc:224] Handling call
0709 13:01:49.459616 (+ 72295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.460048 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.460082 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.477922 (+ 17840us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.584274 (+1106352us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:50.584341 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.584448 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.745656 (+161208us) raft_consensus.cc:1685] finished
0709 13:01:50.745682 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.745954 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10660,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8801,"thread_start_us":1056802,"threads_started":1,"wal-append.queue_time_us":1200152}
W20260709 13:01:50.751510  1585 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.001s	sys 0.016s
W20260709 13:01:50.799659  1585 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.002s	sys 0.016s
I20260709 13:01:50.800628  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1867) took 2473 ms. Trace:
I20260709 13:01:50.799165  1738 rpcz_store.cc:276] 0709 13:01:49.208369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.557627 (+349258us) service_pool.cc:224] Handling call
0709 13:01:49.558862 (+  1235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.589350 (+ 30488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.589387 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.589935 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.794391 (+1204456us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:50.794441 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.794558 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.794607 (+    49us) raft_consensus.cc:1685] finished
0709 13:01:50.794620 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.794789 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7991,"raft.queue_time_us":9458,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":32112,"thread_start_us":585417,"threads_started":1,"wal-append.queue_time_us":545766}
W20260709 13:01:50.735613  3496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.009s	sys 0.007s
I20260709 13:01:50.780764  1586 rpcz_store.cc:276] 0709 13:01:49.265592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.328126 (+1062534us) service_pool.cc:224] Handling call
0709 13:01:50.328465 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.328696 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:50.329045 (+   349us) log.cc:844] Serialized 14 byte log entry
0709 13:01:50.757209 (+428164us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:50.757280 (+    71us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:50.757330 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.757423 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.757617 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":666,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":11442,"thread_start_us":427472,"threads_started":1,"wal-append.queue_time_us":428104}
I20260709 13:01:50.716758  3589 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:01:50.794592  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.008s	sys 0.010s
I20260709 13:01:50.719041  3746 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.721999  3658 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.735428  3771 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:50.801625  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.006s	sys 0.012s
W20260709 13:01:50.820044  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 625555us
W20260709 13:01:50.824522  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.007s	sys 0.013s
I20260709 13:01:50.825233  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:50.827268  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:50.821278  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.010s	sys 0.009s
I20260709 13:01:50.827692  1662 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260709 13:01:50.828168  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 573878us
I20260709 13:01:50.801036  1585 rpcz_store.cc:276] 0709 13:01:48.327670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.150820 (+823150us) service_pool.cc:224] Handling call
0709 13:01:49.163430 (+ 12610us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.163661 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:49.348841 (+185180us) log.cc:844] Serialized 14 byte log entry
0709 13:01:50.800263 (+1451422us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:50.800324 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:50.800381 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.800451 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.800573 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":478071,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":128762,"raft.run_cpu_time_us":423,"raft.run_wall_time_us":4029,"spinlock_wait_cycles":56361856,"thread_start_us":1615151,"threads_started":3,"wal-append.queue_time_us":825426}
W20260709 13:01:50.768271  3411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.009s	sys 0.009s
W20260709 13:01:50.793179  3710 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.114s	user 0.004s	sys 0.018s
W20260709 13:01:50.795364  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.005s	sys 0.013s
I20260709 13:01:50.754222  3764 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.729327  3759 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:50.793491  1737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.008s	sys 0.013s
I20260709 13:01:50.712251  3718 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:50.830108  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 228844us
W20260709 13:01:50.840085  3411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.014s	sys 0.009s
W20260709 13:01:50.780995  3673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.984s	user 0.002s	sys 0.016s
W20260709 13:01:50.868831  3710 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.200s	user 0.004s	sys 0.018s
I20260709 13:01:50.841949  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:50.841944  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:50.842988  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45584.5us
I20260709 13:01:50.907560  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:50.843957  3598 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.309s	user 0.013s	sys 0.002s
I20260709 13:01:50.722295  3597 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.843506  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:50.885946  3187 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.010s	sys 0.012s
W20260709 13:01:50.883246  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.014s	sys 0.015s
W20260709 13:01:50.916906  3187 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.010s	sys 0.012s
W20260709 13:01:50.917028  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.015s	sys 0.016s
I20260709 13:01:50.917367  3187 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:50.918694  3187 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.887425  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1343) took 1670 ms. Trace:
I20260709 13:01:50.921712  1737 rpcz_store.cc:276] 0709 13:01:49.217354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.562712 (+345358us) service_pool.cc:224] Handling call
0709 13:01:49.563145 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.605702 (+ 42557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.608801 (+  3099us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.718540 (+109739us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.887030 (+1168490us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:50.887074 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.887158 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.887204 (+    46us) raft_consensus.cc:1685] finished
0709 13:01:50.887216 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.887360 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20502,"raft.queue_time_us":14509,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":109,"thread_start_us":1042439,"threads_started":1,"wal-append.queue_time_us":951363}
I20260709 13:01:50.792987  3575 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:50.840153  3750 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.162s	user 0.001s	sys 0.001s
I20260709 13:01:50.838127  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:50.899473  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.902132  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.935701  1587 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:50.938177  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.907560  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:50.890707  3405 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.913336  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:50.956552  1660 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:50.958370  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:50.820590  3496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.009s	sys 0.007s
I20260709 13:01:50.885599  1664 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:50.926113  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:50.970830  1592 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:50.927280  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:50.886054  3605 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 13:01:50.938741  3562 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.018s	sys 0.022s
I20260709 13:01:50.912376  3589 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:50.910526  3673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.116s	user 0.003s	sys 0.017s
I20260709 13:01:50.941285  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:50.983585  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:50.984035  2314 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:50.842342  3771 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:50.984289  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:50.949555  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 262693us
I20260709 13:01:50.941588  1586 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:50.939064  1587 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:01:50.841939  3762 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 13:01:50.895911  3584 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.895993  3564 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.898808  3583 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:50.882594  3746 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:50.896234  3761 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:50.916430  3598 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:50.924381  3575 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:50.984858  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.031309  1660 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:51.032563  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:50.894088  3764 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.003937  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.035861  1664 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:50.898440  3759 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:50.980212  3562 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.019s	sys 0.023s
I20260709 13:01:51.014307  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:51.017560  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:50.914222  3721 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260709 13:01:50.982120  3658 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:50.976899  3605 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:51.018934  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 185479us
W20260709 13:01:51.022608  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 115135us
I20260709 13:01:50.939707  3411 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 13:01:51.033413  1660 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:51.037678  1592 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.051209  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:51.056138  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:51.038223  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 196383us
I20260709 13:01:51.042693  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1870) took 2711 ms. Trace:
W20260709 13:01:51.041958  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 134472us
I20260709 13:01:50.978202  3405 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.056814  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.004796  3589 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.016985  3564 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.007721  3584 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.059589  1660 raft_consensus.cc:2410] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 13:01:51.094383  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.127746  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.128702  1660 raft_consensus.cc:2410] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:01:51.080603  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.029044  3746 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.085426  1592 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:01:51.130026  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1349) took 1539 ms (client timeout 1929 ms). Trace:
I20260709 13:01:51.133226  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:51.133944  1660 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:51.134182  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:51.133620  1665 rpcz_store.cc:269] 0709 13:01:49.590553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.611336 (+ 20783us) service_pool.cc:224] Handling call
0709 13:01:51.080427 (+1469091us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:51.129958 (+ 49531us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4956544}
I20260709 13:01:51.135830  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:51.037140  3773 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
W20260709 13:01:51.078352  3765 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20260709 13:01:51.110773  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.061244  1738 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:51.136932  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.137351  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:51.138594  1742 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:51.140285  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.142347  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.058995  1584 rpcz_store.cc:276] 0709 13:01:48.331375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.247969 (+916594us) service_pool.cc:224] Handling call
0709 13:01:49.248359 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.248717 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.248748 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.249246 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 13:01:50.178438 (+929192us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:50.178490 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.178569 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:51.042454 (+863885us) raft_consensus.cc:1685] finished
0709 13:01:51.042476 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.042624 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1257,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":913945,"threads_started":1,"wal-append.queue_time_us":1355984}
I20260709 13:01:51.032256  3761 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.046046  3774 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.144665  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.637236  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.589996  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.638325  1734 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:01:51.050472  3605 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.639698  1738 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:51.514138  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.598816  3794 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.641577  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.678717  1735 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:51.577133  2282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.008s	sys 0.019s
W20260709 13:01:51.680086  2282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.010s	sys 0.019s
I20260709 13:01:51.643774  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.682085  1742 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.644423  1592 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.644973  1665 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:51.185109  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:51.206185  1664 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:51.082334  3771 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:51.631314  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.689546  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:51.656522  1660 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.000s
W20260709 13:01:51.077006  3753 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.901s	user 0.003s	sys 0.003s
W20260709 13:01:51.658874  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.010s	sys 0.012s
W20260709 13:01:51.706578  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.010s	sys 0.012s
W20260709 13:01:51.707134  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 659538us
I20260709 13:01:51.322271  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:51.476897  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 339775us
I20260709 13:01:51.579515  3721 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260709 13:01:51.086900  3597 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:51.592458  3783 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.463s	user 0.000s	sys 0.000s
W20260709 13:01:51.591467  3779 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.540s	user 0.000s	sys 0.000s
W20260709 13:01:51.062281  3778 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.300s	user 0.001s	sys 0.004s
I20260709 13:01:51.084375  3759 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:51.079521  3699 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:51.056254  3583 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:51.742249  3746 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.743000  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:51.666839  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.022s	sys 0.022s
W20260709 13:01:51.745425  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.022s	sys 0.023s
I20260709 13:01:51.670140  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:51.749267  1584 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.707062  3585 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:51.136943  3670 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:51.761931  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.188515  3766 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:51.691128  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 506161us
W20260709 13:01:51.665287  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0971559840000396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260709 13:01:51.058348  3786 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.615s
I20260709 13:01:51.220465  3789 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:51.188688  3411 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:51.695261  1660 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:51.076790  3598 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.670562  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.322176  3792 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:51.487984  3795 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:51.710134  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:51.801095  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91095.3us
I20260709 13:01:51.802170  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:51.662040  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.013s	sys 0.009s
W20260709 13:01:51.803931  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.014s	sys 0.010s
W20260709 13:01:51.804309  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 757892us
I20260709 13:01:51.805246  1584 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:51.805735  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:51.806447  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:51.807205  1584 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.807780  3800 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:51.808636  3800 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:51.809283  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:51.810334  3800 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:51.811937  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:51.714259  1592 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:51.813328  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:51.712410  1708 connection.cc:256] RPC call timeout handler was delayed by -1.3095055999999659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260709 13:01:51.719151  1742 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:51.722532  1664 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:01:51.725569  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.819267  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.820142  1742 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 13:01:51.675484  3187 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.021s	sys 0.024s
W20260709 13:01:51.729328  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 407185us
I20260709 13:01:51.822495  3792 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:51.735939  1735 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:51.672115  3784 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:51.513614  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.658457  3774 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.754879  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:51.759495  2316 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:51.759893  3585 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:51.766939  3682 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.489s	user 0.014s	sys 0.001s
I20260709 13:01:51.748519  3347 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:51.743018  3790 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:51.733559  3578 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.781270  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:51.844189  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.763379  1489 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:01:51.783792  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:51.762228  3670 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.817126  3764 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.750558  3759 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.823764  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.825467  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:51.759212  3699 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.798664  3766 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:51.850396  3783 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:01:51.697299  3794 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:51.688723  3710 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.010s	sys 0.009s
I20260709 13:01:51.838640  1584 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:51.838860  3792 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:01:51.839390  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:51.832294  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:51.824705  3187 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.022s	sys 0.025s
W20260709 13:01:51.847229  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94353.3us
I20260709 13:01:51.892700  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:51.849262  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:01:51.894788  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:51.708472  1738 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.815604  3564 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.823812  3746 raft_consensus.cc:2764] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:01:51.860941  1666 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:51.850306  1489 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 2 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 3 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:51.833355  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.904541  1735 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.850154  3682 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:51.873471  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48128.7us
W20260709 13:01:51.874365  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 90730.8us
I20260709 13:01:51.914670  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:51.915927  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:51.846316  3585 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:51.815917  3411 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.845211  3790 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.811229  3800 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 13:01:51.829139  3625 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:51.853673  3584 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.856281  3670 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:51.889986  1663 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:51.837724  3726 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 13:01:51.887218  1584 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:51.987604  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1895) took 1086 ms. Trace:
I20260709 13:01:51.988170  1584 rpcz_store.cc:276] 0709 13:01:50.900901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.886139 (+985238us) service_pool.cc:224] Handling call
0709 13:01:51.886573 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.987269 (+100696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.987533 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:51.895452  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:51.876389  3773 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.876611  3797 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.829675  3793 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:01:51.875259  3405 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.838452  3642 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:51.882113  3598 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:51.919852  1735 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:51.852564  3347 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:51.883674  3710 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.011s	sys 0.010s
I20260709 13:01:51.883538  3814 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:51.799038  3789 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:51.925774  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93683.5us
I20260709 13:01:51.926806  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1893) took 1026 ms. Trace:
I20260709 13:01:51.922294  1489 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 2 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 3 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:52.031996  1489 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 4 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:01:51.932673  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1860) took 4937 ms. Trace:
I20260709 13:01:51.944552  1668 raft_consensus.cc:2368] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:52.034008  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:51.899354  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:51.925613  3658 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:51.842330  3784 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.922876  3782 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:51.921685  3811 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:51.950237  3766 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.956055  1666 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:51.896847  3718 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.876957  3759 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:51.970178  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:52.017571  3814 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.992359  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:51.993789  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0528374759999792s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15200000000000002s for a response
W20260709 13:01:52.043501  3690 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.427s	user 0.010s	sys 0.006s
W20260709 13:01:52.055768  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0474374759999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260709 13:01:52.055873  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63662.4us
W20260709 13:01:52.056468  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0438374759999647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15300000000000002s for a response
I20260709 13:01:52.057961  3690 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:52.059291  1708 leader_election.cc:336] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.520s (SENT)
I20260709 13:01:52.059790  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:51.934064  3790 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:52.060451  3690 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.061631  1734 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:52.017158  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.033604  1589 rpcz_store.cc:276] 0709 13:01:46.995303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.150997 (+1155694us) service_pool.cc:224] Handling call
0709 13:01:49.259384 (+1108387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.366605 (+107221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.932436 (+2565831us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54003b79d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:51.932609 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2530124544}
I20260709 13:01:52.034849  1739 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:52.035799  3761 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.065340  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.038738  1592 rpcz_store.cc:276] 0709 13:01:50.900276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.814230 (+913954us) service_pool.cc:224] Handling call
0709 13:01:51.814515 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.814738 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:51.858774 (+ 44036us) log.cc:844] Serialized 14 byte log entry
0709 13:01:51.926271 (+ 67497us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:51.926330 (+    59us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:51.926382 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:51.926452 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:51.926739 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83169,"raft.queue_time_us":41491,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":10447,"thread_start_us":74041,"threads_started":2,"wal-append.queue_time_us":157584}
I20260709 13:01:52.067111  1589 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:52.035586  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.068068  1584 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:51.974133  3731 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:51.972142  3794 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:51.979641  3746 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:51.977999  2315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 220) took 2797 ms (client timeout 2177 ms). Trace:
I20260709 13:01:51.973410  3698 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:51.958808  3699 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:51.970956  3781 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:51.951383  3411 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:51.971144  3682 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.013374  3586 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:51.945463  3807 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:52.047227  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 152708us
I20260709 13:01:52.103192  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.048864  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.011857  3642 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.959807  3774 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.012094  3598 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:51.990738  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:52.062667  2309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 219) took 2902 ms (client timeout 1627 ms). Trace:
W20260709 13:01:52.054138  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:52.107410  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.031669  3584 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:52.055259  1740 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.062791  3814 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:52.113026  1737 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:52.063397  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
W20260709 13:01:52.049980  3762 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.000s	sys 0.016s
I20260709 13:01:52.050879  3658 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:52.018069  3773 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.052446  3818 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:52.066969  3726 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:52.100485  1584 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:52.104198  1739 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:01:52.095510  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 175298us
I20260709 13:01:52.016507  3797 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.069371  1592 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:01:52.117070  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68357.3us
I20260709 13:01:52.149381  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:52.150647  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.117731  3586 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:52.118287  3795 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:52.106392  3642 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.122076  3731 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.122716  3761 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.123636  3800 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.126312  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.168352  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.081297  3690 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.140501  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.169071  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1912) took 1001 ms. Trace:
I20260709 13:01:52.169546  1737 rpcz_store.cc:276] 0709 13:01:51.167506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.147217 (+979711us) service_pool.cc:224] Handling call
0709 13:01:52.168529 (+ 21312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.168731 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.168759 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:52.168801 (+    42us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:52.168832 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.168887 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.169011 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:52.170177  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1898) took 3338 ms (client timeout 1910 ms). Trace:
W20260709 13:01:52.171005  1736 rpcz_store.cc:269] 0709 13:01:48.825065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.825268 (+   203us) service_pool.cc:224] Handling call
0709 13:01:52.163034 (+3337766us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000e2ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:52.163159 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406256000}
I20260709 13:01:52.052259  3405 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 13:01:52.188557  3405 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:52.190361  3820 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:52.190653  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 218) took 3381 ms. Trace:
I20260709 13:01:52.191075  2314 rpcz_store.cc:276] 0709 13:01:48.808803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:48.812047 (+  3244us) service_pool.cc:224] Handling call
0709 13:01:48.844346 (+ 32299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.984406 (+2140060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.190363 (+1205957us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.190562 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4920796416}
I20260709 13:01:52.101796  3699 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.152742  3805 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:52.183108  3814 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.115402  3762 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.000s	sys 0.017s
I20260709 13:01:52.156808  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.193032  3805 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.193634  1584 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:52.194849  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.195400  3670 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.118577  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59315.1us
I20260709 13:01:52.196259  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.161574  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.159932  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 276) took 3576 ms. Trace:
I20260709 13:01:52.110116  3584 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:52.165844  1742 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:52.166750  1740 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:52.172473  1736 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:52.079954  3826 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 13:01:52.162461  3795 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.173779  1737 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:01:52.173578  3761 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.159686  3793 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 13:01:52.162689  3821 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:52.174448  3800 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.175257  1589 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:52.159868  3710 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:52.158677  3596 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.180514  3759 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.181147  3658 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.182757  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:52.103322  3781 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.090824  3746 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.234766  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52069.6us
I20260709 13:01:52.236510  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:52.185396  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1927) took 1346 ms. Trace:
I20260709 13:01:52.237726  1668 rpcz_store.cc:276] 0709 13:01:50.839291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.841674 (+  2383us) service_pool.cc:224] Handling call
0709 13:01:52.185172 (+1343498us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001a6dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:52.185331 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2384892160}
I20260709 13:01:52.239708  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.240546  1668 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:52.186339  3347 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:52.083190  2315 rpcz_store.cc:269] 0709 13:01:49.180781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.241892 (+ 61111us) service_pool.cc:224] Handling call
0709 13:01:51.977765 (+2735873us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54006a0ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:51.977892 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":328396672}
W20260709 13:01:52.114271  2309 rpcz_store.cc:269] 0709 13:01:49.160238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.210128 (+ 49890us) service_pool.cc:224] Handling call
0709 13:01:51.891603 (+2681475us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:52.054740 (+163137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6556928}
I20260709 13:01:52.169697  3731 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.195782  3357 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:52.055646  3811 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.197698  1592 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:52.110005  3598 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.142779  3726 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.142457  3818 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.197672  1591 rpcz_store.cc:276] 0709 13:01:48.583103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.252519 (+669416us) service_pool.cc:224] Handling call
0709 13:01:49.252840 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:49.524556 (+271716us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:01:49.524592 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:49.524644 (+    52us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:49.524675 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.159661 (+2634986us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540031d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.159715 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.159868 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159089,"raft.queue_time_us":196445,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":28173,"spinlock_wait_cycles":257948928}
I20260709 13:01:52.197357  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.192914  3699 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.155670  3578 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:52.175804  3690 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.212947  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 267) took 1279 ms. Trace:
I20260709 13:01:52.179302  3773 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.215245  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.222762  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:01:52.189406  3405 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:52.183929  3555 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:52.270355  1589 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:52.211529  3795 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.151027  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:52.251979  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.255873  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.258419  1584 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:52.246493  3596 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:52.272811  1737 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:52.264348  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.274271  1592 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.252830  3357 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:52.278467  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:52.267899  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99668.9us
I20260709 13:01:52.266644  1664 rpcz_store.cc:276] 0709 13:01:50.933609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.832368 (+898759us) service_pool.cc:224] Handling call
0709 13:01:51.852041 (+ 19673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.852267 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:52.064076 (+211809us) log.cc:844] Serialized 14 byte log entry
0709 13:01:52.212452 (+148376us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:52.212535 (+    83us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:52.212608 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.212701 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.212865 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13455,"prepare.queue_time_us":309945,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":13082,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":5832,"thread_start_us":331164,"threads_started":2}
I20260709 13:01:52.253805  3759 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:01:52.285087  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 134175us
I20260709 13:01:52.279342  1591 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:52.273628  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.249897  3586 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.283773  3808 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:52.261915  3726 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.246887  3826 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
W20260709 13:01:52.268484  3819 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 13:01:52.285753  3339 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:52.321683  3339 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:52.250097  3710 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:52.287214  1589 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:52.322017  3726 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.322929  3814 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:52.288126  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.323011  3710 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.273036  3596 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.293076  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:52.284538  3789 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.297266  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.297868  1592 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:52.298739  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.299849  3784 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.300268  3699 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.261965  3818 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.305301  3811 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:52.308892  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:52.285013  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:52.315840  3585 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.296221  3808 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:52.317440  3761 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.318780  3731 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.320165  3690 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.253022  3584 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.323922  3405 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.323302  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.327301  1736 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.296099  1705 leader_election.cc:400] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:52.327862  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.330580  3596 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:52.331162  3797 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:52.375730  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:52.469291  3751 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.297s	user 0.011s	sys 0.025s
I20260709 13:01:52.452883  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 270) took 1518 ms. Trace:
I20260709 13:01:52.482095  1660 rpcz_store.cc:276] 0709 13:01:50.934494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.848674 (+914180us) service_pool.cc:224] Handling call
0709 13:01:51.849015 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:51.951625 (+102610us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:52.081325 (+129700us) log.cc:844] Serialized 14 byte log entry
0709 13:01:52.139055 (+ 57730us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:52.139165 (+   110us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:52.139216 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.430793 (+291577us) spinlock_profiling.cc:243] Waited 7.2 ms on lock 0x7b180017ea20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:01:52.452381 (+ 21588us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.452528 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.452764 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45317,"prepare.queue_time_us":221307,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":10157,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6765,"spinlock_wait_cycles":252028928,"thread_start_us":176442,"threads_started":2,"wal-append.queue_time_us":91846}
I20260709 13:01:52.388666  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.490321  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.306840  3793 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:01:52.434122  2280 connection.cc:256] RPC call timeout handler was delayed by -1.0165249819999644s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260709 13:01:52.472823  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 164059us
I20260709 13:01:52.425153  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1345) took 3198 ms. Trace:
I20260709 13:01:52.543118  1741 rpcz_store.cc:276] 0709 13:01:49.226999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.565320 (+338321us) service_pool.cc:224] Handling call
0709 13:01:49.625524 (+ 60204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.384915 (+759391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.424785 (+2039870us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b54003520d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.425033 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1714134912}
I20260709 13:01:52.473290  1705 leader_election.cc:403] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:01:52.444790  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:52.352255  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 185031us
I20260709 13:01:52.545902  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:52.546928  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.547833  1742 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:52.384246  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1886) took 3671 ms. Trace:
I20260709 13:01:52.477000  1584 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:01:52.478729  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.576488  1592 raft_consensus.cc:2368] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:52.376374  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1915) took 1179 ms. Trace:
I20260709 13:01:52.577230  1734 rpcz_store.cc:276] 0709 13:01:51.196658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.670337 (+473679us) service_pool.cc:224] Handling call
0709 13:01:52.376073 (+705736us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54003534d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:52.376291 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627452800}
I20260709 13:01:52.579452  1734 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:52.428319  3339 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.529938  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 249266us
I20260709 13:01:52.298743  3670 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.295089  3805 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:52.479465  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 282969us
W20260709 13:01:52.654901  1708 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.530s (SENT)
W20260709 13:01:52.632413  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 339283us
I20260709 13:01:52.635653  1742 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 13:01:52.394580  3583 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.650162  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:52.652460  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107104us
I20260709 13:01:52.549700  1590 rpcz_store.cc:276] 0709 13:01:48.712946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:50.260454 (+1547508us) service_pool.cc:224] Handling call
0709 13:01:50.260818 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:50.399480 (+138662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:50.399517 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:50.399564 (+    47us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:50.399603 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.383847 (+1984244us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.383925 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.384153 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":318314240}
I20260709 13:01:52.660177  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.681542  1592 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:01:52.662169  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:52.387334  3605 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.685s	user 0.009s	sys 0.005s
I20260709 13:01:52.310995  3759 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:52.671402  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:52.683985  1735 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.351691  3773 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:52.399681  3589 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:52.708298  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.687317  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.687582  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.709558  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.351812  3726 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:52.719161  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.719873  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.720048  3761 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:01:52.427628  3800 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:52.698621  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.721426  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.709920  1741 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:52.719002  1659 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:52.657596  3187 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:52.659324  3690 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:01:52.748315  2317 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:52.700279  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.688942  3583 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.762959  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.763828  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.764447  1584 raft_consensus.cc:2368] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:52.764849  1741 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:52.765614  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:52.711320  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49288.3us
I20260709 13:01:52.714069  2312 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:01:52.709942  1590 raft_consensus.cc:2368] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:52.718950  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.768282  2309 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:52.769078  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:52.770186  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.770315  1741 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:52.534701  3721 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:52.699721  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.394889  3781 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:52.686733  1668 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:52.737013  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:52.771637  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72042.7us
I20260709 13:01:52.712857  3584 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.745301  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 255) took 3740 ms. Trace:
I20260709 13:01:52.470691  3808 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.708153  3773 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:52.673036  3789 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:52.723320  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1345) took 3134 ms. Trace:
I20260709 13:01:52.759212  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:52.787050  1659 rpcz_store.cc:276] 0709 13:01:49.589209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.590710 (+  1501us) service_pool.cc:224] Handling call
0709 13:01:52.685576 (+3094866us) spinlock_profiling.cc:243] Waited 9.27 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:52.718760 (+ 33184us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.723029 (+  4269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.723235 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21330688}
I20260709 13:01:52.787117  1735 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:52.367090  3357 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.790006  3357 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.736634  3734 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:52.536805  3793 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.772452  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.773131  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.737605  2313 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:01:52.775594  3824 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:52.775777  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.753890  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1341) took 3491 ms. Trace:
I20260709 13:01:52.771299  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.809203  1667 rpcz_store.cc:276] 0709 13:01:49.262350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.390702 (+128352us) service_pool.cc:224] Handling call
0709 13:01:49.420934 (+ 30232us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:50.424180 (+1003246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:52.753469 (+2329289us) spinlock_profiling.cc:243] Waited 964 ms on lock 0x7b5400117e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:52.753812 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2218020224}
I20260709 13:01:52.703217  3805 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 13:01:52.780601  1736 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:52.782961  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.725800  3339 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.811501  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.789924  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:52.819037  1584 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:52.705024  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.820003  1592 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:52.448547  3792 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.822023  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1348) took 3231 ms (client timeout 1877 ms). Trace:
W20260709 13:01:52.822494  1666 rpcz_store.cc:269] 0709 13:01:49.590235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.590928 (+   693us) service_pool.cc:224] Handling call
0709 13:01:51.811754 (+2220826us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:52.821838 (+1010084us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:52.821946 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":116007040}
I20260709 13:01:52.823704  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.716231  3823 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.092s
I20260709 13:01:52.688558  3731 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 13:01:52.827359  3781 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:52.702800  3605 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.777724  3721 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:52.832372  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:52.838512  3721 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.839612  1591 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:52.840304  3768 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260709 13:01:52.777112  1733 rpcz_store.cc:276] 0709 13:01:49.004430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.004539 (+   109us) service_pool.cc:224] Handling call
0709 13:01:49.004848 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:49.005260 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.005296 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:49.005827 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 13:01:49.404129 (+398302us) spinlock_profiling.cc:243] Waited 7.34 ms on lock 0x7b18002faa60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:01:49.880426 (+476297us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:49.880474 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:49.880568 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:52.468470 (+2587902us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:52.744977 (+276507us) raft_consensus.cc:1685] finished
0709 13:01:52.745007 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:52.745213 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":219,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":10693,"spinlock_wait_cycles":98167936,"thread_start_us":476046,"threads_started":1,"wal-append.queue_time_us":1166397}
I20260709 13:01:52.810005  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.811589  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.761906  3187 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:52.786497  3754 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.612s	user 0.018s	sys 0.036s
I20260709 13:01:52.759990  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.797901  3761 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:01:52.795255  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58267us
I20260709 13:01:52.779036  3746 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:52.782208  3811 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:52.800251  3734 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.828150  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.850245  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.850950  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.809602  3793 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:52.798621  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:50.645234  1593 rpcz_store.cc:276] 0709 13:01:48.325411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.056613 (+731202us) service_pool.cc:224] Handling call
0709 13:01:49.057072 (+   459us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:49.101750 (+ 44678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:49.101791 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:49.108869 (+  7078us) log.cc:844] Serialized 52 byte log entry
0709 13:01:49.415402 (+306533us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:50.643320 (+1227918us) log.cc:844] Serialized 14 byte log entry
0709 13:01:50.643820 (+   500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:50.644105 (+   285us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:50.644172 (+    67us) raft_consensus.cc:1685] finished
0709 13:01:50.644194 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:50.644430 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35034,"prepare.queue_time_us":484315,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":6606,"raft.queue_time_us":216384,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":5024,"thread_start_us":1286051,"threads_started":2,"wal-append.queue_time_us":332463}
I20260709 13:01:52.878876  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.880046  1593 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:52.881033  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.858203  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:52.815515  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:52.882711  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:52.841532  3731 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
W20260709 13:01:52.887158  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 191933us
I20260709 13:01:52.813324  3796 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.871066  1741 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:52.871275  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.896183  1662 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:52.871884  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.834007  3808 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.862787  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:52.787058  3789 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.873346  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.907698  1588 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:52.904381  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:52.890475  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.868198  3746 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 13:01:52.912828  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.872196  3734 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:52.850004  3781 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:01:52.870667  3720 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:52.859223  3811 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:52.892167  3731 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:52.960536  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:52.961714  1588 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:52.925966  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:52.927274  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:52.933729  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:52.936738  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:52.969648  1590 raft_consensus.cc:2410] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:01:52.894666  3805 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:01:52.973436  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:52.940842  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:52.944654  1593 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:52.849584  3824 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:52.946936  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.924989  3821 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:52.952149  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:52.971359  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:52.896212  3761 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:52.976869  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:52.982957  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49321.1us
I20260709 13:01:52.832902  3759 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.021977  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:53.023160  1593 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:52.939957  3597 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:52.886255  3605 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.025955  1708 leader_election.cc:400] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:53.035092  1708 leader_election.cc:403] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 13:01:53.032380  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:52.900084  1659 raft_consensus.cc:2376] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4. Re-sending same reply.
I20260709 13:01:52.976497  3723 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260709 13:01:52.981951  3187 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.019546  3699 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:53.021438  3761 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:52.986676  3814 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:52.986629  3710 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 13:01:52.910929  3796 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:52.980894  3828 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:52.973302  3805 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:52.872874  3773 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:52.981578  3568 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.110s
I20260709 13:01:53.046638  1662 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:52.983662  3383 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 13:01:53.047312  1593 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:53.101547  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.119338  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1906) took 1185 ms. Trace:
I20260709 13:01:53.046824  1587 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:53.119798  1593 rpcz_store.cc:276] 0709 13:01:51.934159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.024128 (+1089969us) service_pool.cc:224] Handling call
0709 13:01:53.024474 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.118923 (+ 94449us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.119258 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.085527  2316 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 13:01:53.114427  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 231842us
I20260709 13:01:53.123056  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:53.108260  1588 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:01:53.125485  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:53.041074  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89012.8us
I20260709 13:01:53.052876  1590 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:53.127774  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1907) took 1156 ms. Trace:
I20260709 13:01:53.128242  1590 rpcz_store.cc:276] 0709 13:01:51.971588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.027320 (+1055732us) service_pool.cc:224] Handling call
0709 13:01:53.052411 (+ 25091us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.127396 (+ 74985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.127697 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.061326  1708 leader_election.cc:400] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
W20260709 13:01:53.129861  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68740.1us
I20260709 13:01:53.130915  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:53.132450  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:53.080672  1741 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:01:53.147910  1590 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:53.148860  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2067) took 1004 ms. Trace:
I20260709 13:01:53.150209  1590 rpcz_store.cc:276] 0709 13:01:52.144288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.129216 (+984928us) service_pool.cc:224] Handling call
0709 13:01:53.147660 (+ 18444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.148564 (+   904us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.148789 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.150652  1708 leader_election.cc:400] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:53.156068  1708 leader_election.cc:403] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:01:52.976536  3811 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:53.164340  1588 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 13:01:53.070310  3840 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 13:01:53.165444  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2065) took 1030 ms. Trace:
W20260709 13:01:53.100286  3600 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.504s	user 0.034s	sys 0.040s
I20260709 13:01:53.054064  3761 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:53.611842  3586 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.568348  1588 rpcz_store.cc:276] 0709 13:01:52.135476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.125016 (+989540us) service_pool.cc:224] Handling call
0709 13:01:53.163928 (+ 38912us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.165029 (+  1101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.165354 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.058154  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1355) took 1187 ms. Trace:
I20260709 13:01:53.621440  1659 rpcz_store.cc:276] 0709 13:01:51.871042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.789232 (+918190us) service_pool.cc:224] Handling call
0709 13:01:53.058071 (+268839us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:53.623096  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 1912) took 1350 ms (client timeout 1608 ms). Trace:
I20260709 13:01:53.624375  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:53.577787  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.001s	sys 0.011s
I20260709 13:01:53.179932  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:53.224985  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.662518  1662 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:53.663627  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1353) took 1793 ms (client timeout 2337 ms). Trace:
W20260709 13:01:53.664152  1662 rpcz_store.cc:269] 0709 13:01:51.870100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.179578 (+1309478us) service_pool.cc:224] Handling call
0709 13:01:53.663382 (+483804us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.184662  1708 leader_election.cc:400] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
W20260709 13:01:53.585382  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.222967  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 263) took 1356 ms. Trace:
W20260709 13:01:53.269310  2280 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.014s	sys 0.011s
W20260709 13:01:53.685881  2280 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.014s	sys 0.012s
I20260709 13:01:53.686138  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:53.686939  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 964041us
W20260709 13:01:53.193347  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56324.8us
I20260709 13:01:53.689872  1734 rpcz_store.cc:276] 0709 13:01:51.866743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.580426 (+713683us) service_pool.cc:224] Handling call
0709 13:01:52.580790 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.581083 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:52.859050 (+277967us) log.cc:844] Serialized 14 byte log entry
0709 13:01:53.222424 (+363374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:53.222517 (+    93us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:53.222586 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.222692 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.222871 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":377251,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":115,"raft.queue_time_us":3739,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":9436,"thread_start_us":557754,"threads_started":2,"wal-append.queue_time_us":477108}
I20260709 13:01:53.694217  3746 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:53.694940  3746 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:53.695379  3746 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:53.093794  2310 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 1.
I20260709 13:01:53.318459  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:53.458745  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:53.703405  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:53.588406  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.704639  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 1947) took 1393 ms (client timeout 1501 ms). Trace:
I20260709 13:01:53.321722  2313 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:53.706650  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.303563  3357 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.460201  3568 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:53.071316  3583 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.727331  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1923) took 1007 ms. Trace:
I20260709 13:01:53.113870  3585 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.076793  3596 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.739462  1592 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 13:01:53.739956  1592 pending_rounds.cc:85] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 4
I20260709 13:01:53.740276  1592 pending_rounds.cc:107] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:01:53.740624  1592 raft_consensus.cc:2889] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:53.436723  3598 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.170s	user 0.012s	sys 0.003s
W20260709 13:01:53.181537  3600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.585s	user 0.034s	sys 0.040s
I20260709 13:01:53.141796  3605 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:53.751434  3605 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:53.751561  1739 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:01:53.752082  3605 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:53.270170  3642 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.072s	user 0.014s	sys 0.002s
I20260709 13:01:53.754415  3642 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.053640  3658 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.146s
I20260709 13:01:53.202028  3699 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
W20260709 13:01:53.419229  3690 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.657s	user 0.017s	sys 0.001s
I20260709 13:01:53.058168  3720 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:53.080786  3726 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:53.759104  3690 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:53.080106  3731 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:53.500352  3771 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.003s	sys 0.011s
W20260709 13:01:53.761181  3771 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.937s	user 0.003s	sys 0.012s
I20260709 13:01:53.146236  3773 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:53.373389  3764 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:53.764866  3764 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:53.766139  3658 consensus_queue.cc:1243] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:01:53.767114  3764 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.768267  1666 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:53.769901  1665 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:53.363953  3774 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:53.771661  3774 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:53.148540  3781 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:53.744688  3598 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.775102  3726 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:53.219669  3790 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.159s	user 0.017s	sys 0.001s
I20260709 13:01:53.421195  3792 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:53.456910  3793 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.569s	user 0.005s	sys 0.008s
W20260709 13:01:53.345197  3795 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.060s	user 0.010s	sys 0.005s
I20260709 13:01:53.064383  3796 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.162238  3800 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:53.095038  3805 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:53.483896  3808 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.569s	user 0.008s	sys 0.006s
I20260709 13:01:53.509460  3811 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:53.372046  3818 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.902s	user 0.015s	sys 0.000s
I20260709 13:01:53.158305  3821 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:01:53.394347  3824 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:53.380570  3834 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:53.382688  3837 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.487s	user 0.002s	sys 0.000s
W20260709 13:01:53.379567  3838 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.547s	user 0.000s	sys 0.002s
W20260709 13:01:53.349373  3845 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
W20260709 13:01:53.362542  3847 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 13:01:53.600428  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.613652  3721 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.566835  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2070) took 1405 ms. Trace:
W20260709 13:01:53.501313  3734 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.523s	user 0.017s	sys 0.000s
I20260709 13:01:53.576411  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1914) took 1204 ms. Trace:
I20260709 13:01:53.627900  3815 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:01:53.627071  1588 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:53.628827  3849 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:53.629945  3586 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.792415  3793 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.809393  3734 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.810525  1664 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:53.811000  1660 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:01:53.811669  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1358) took 1939 ms (client timeout 1838 ms). Trace:
I20260709 13:01:53.812115  3734 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:53.812079  1660 rpcz_store.cc:269] 0709 13:01:51.872135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.599226 (+1727091us) service_pool.cc:224] Handling call
0709 13:01:53.811569 (+212343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.813151  3793 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.813957  1590 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:53.814936  3834 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:53.815290  3721 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.816558  3834 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.804492  1587 rpcz_store.cc:276] 0709 13:01:52.160771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.808846 (+648075us) service_pool.cc:224] Handling call
0709 13:01:53.565827 (+756981us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540031d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.566234 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462815104}
I20260709 13:01:53.818565  3811 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:53.819051  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1346) took 4229 ms. Trace:
I20260709 13:01:53.639920  3853 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:53.642900  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.633176  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:53.677696  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.001s	sys 0.011s
I20260709 13:01:53.665961  1708 leader_election.cc:403] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:01:53.682456  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:53.623991  1591 rpcz_store.cc:269] 0709 13:01:52.272777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.298625 (+ 25848us) service_pool.cc:224] Handling call
0709 13:01:53.622338 (+1323713us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b5400393fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.622803 (+   465us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843226880}
I20260709 13:01:53.285117  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:53.820559  3853 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.681300  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:53.824360  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 639938us
I20260709 13:01:53.824986  1662 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:53.825522  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1357) took 1953 ms. Trace:
W20260709 13:01:53.825754  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 848784us
I20260709 13:01:53.826179  1662 rpcz_store.cc:276] 0709 13:01:51.871858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.665148 (+1793290us) service_pool.cc:224] Handling call
0709 13:01:53.825433 (+160285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.828555  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:53.678978  1557 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.014s	sys 0.037s
I20260709 13:01:53.829164  3600 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 13:01:53.705039  1663 rpcz_store.cc:269] 0709 13:01:52.310899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.669088 (+358189us) service_pool.cc:224] Handling call
0709 13:01:53.702880 (+1033792us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:53.704578 (+  1698us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23394176}
I20260709 13:01:53.829806  1662 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:53.830652  3821 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:53.830920  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.831086  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 278) took 1165 ms. Trace:
I20260709 13:01:53.722314  3357 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.831913  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1354) took 1961 ms. Trace:
I20260709 13:01:53.832086  1737 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:53.832343  1663 rpcz_store.cc:276] 0709 13:01:51.870465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.830439 (+1959974us) service_pool.cc:224] Handling call
0709 13:01:53.831831 (+  1392us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.832813  1741 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:53.722008  1667 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:53.441016  3584 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.660s	user 0.013s	sys 0.003s
I20260709 13:01:53.833786  3357 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:53.726328  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1359) took 1846 ms (client timeout 1670 ms). Trace:
W20260709 13:01:53.737704  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.834656  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:53.834543  1668 rpcz_store.cc:269] 0709 13:01:51.872647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.255464 (+382817us) service_pool.cc:224] Handling call
0709 13:01:53.716074 (+1460610us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54001a9854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.719417 (+  3343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":953943296}
I20260709 13:01:53.835923  3600 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:53.836323  3600 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:53.298825  3789 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:53.780244  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:53.776566  3764 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:53.771301  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 579) took 1049 ms. Trace:
I20260709 13:01:53.838495  3834 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:53.785473  3774 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.839650  3774 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:53.833844  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1351) took 1965 ms (client timeout 2050 ms). Trace:
I20260709 13:01:53.838831  2310 rpcz_store.cc:276] 0709 13:01:52.721832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.753268 (+ 31436us) service_pool.cc:224] Handling call
0709 13:01:53.770749 (+1017481us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000c49d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.771041 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466798848}
W20260709 13:01:53.841989  1667 rpcz_store.cc:269] 0709 13:01:51.868774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.810772 (+941998us) service_pool.cc:224] Handling call
0709 13:01:53.705972 (+895200us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:53.833777 (+127805us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10428672}
I20260709 13:01:53.805115  1586 rpcz_store.cc:276] 0709 13:01:52.371941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.572111 (+1200170us) service_pool.cc:224] Handling call
0709 13:01:53.572941 (+   830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.574999 (+  2058us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.575042 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:53.575138 (+    96us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:53.575240 (+   102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.575634 (+   394us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.575839 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.807700  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 285) took 1054 ms. Trace:
I20260709 13:01:53.844326  1588 rpcz_store.cc:276] 0709 13:01:52.753260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.625480 (+872220us) service_pool.cc:224] Handling call
0709 13:01:53.626553 (+  1073us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.807262 (+180709us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.807605 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:53.744616  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45397.3us
I20260709 13:01:53.807551  3818 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.845876  3790 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.847564  2318 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:53.848101  3818 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:53.810889  3586 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.819746  1587 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 13:01:53.829764  1557 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.015s	sys 0.038s
I20260709 13:01:53.831128  1591 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:53.728540  1736 rpcz_store.cc:276] 0709 13:01:52.702246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.821658 (+119412us) service_pool.cc:224] Handling call
0709 13:01:52.821974 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.837383 (+ 15409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.837417 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:52.837474 (+    57us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:52.837534 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.700857 (+863323us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.700961 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.709032 (+  8071us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34923904}
I20260709 13:01:53.835129  1589 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:53.718294  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 280) took 1016 ms. Trace:
W20260709 13:01:53.856221  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 702155us
I20260709 13:01:53.856413  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.856637  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:53.857108  1557 connection.cc:256] RPC call timeout handler was delayed by -2.0202744070000449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2177s for a response
I20260709 13:01:53.856441  1659 rpcz_store.cc:276] 0709 13:01:52.698325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.623660 (+925335us) service_pool.cc:224] Handling call
0709 13:01:53.714954 (+ 91294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:53.858249  2310 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:01:53.841343  3853 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:53.638799  1740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1906) took 2499 ms (client timeout 1546 ms). Trace:
I20260709 13:01:53.858703  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:53.859130  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.859947  1663 raft_consensus.cc:2376] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4. Re-sending same reply.
I20260709 13:01:53.860661  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.861207  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:53.859148  1740 rpcz_store.cc:269] 0709 13:01:51.139197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.140056 (+   859us) service_pool.cc:224] Handling call
0709 13:01:53.638166 (+2498110us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.638431 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":977153792}
I20260709 13:01:53.849800  1662 rpcz_store.cc:276] 0709 13:01:52.666054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.828022 (+1161968us) service_pool.cc:224] Handling call
0709 13:01:53.831009 (+  2987us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:53.835166  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.862792  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.863706  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.864517  1740 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:53.829854  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.795012  3795 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.869529  1736 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:53.858453  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:53.864243  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.849015  3818 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:53.871583  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81154.9us
I20260709 13:01:53.835947  1741 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:53.778928  3642 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.794165  3800 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:53.880012  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57394us
I20260709 13:01:53.930397  3731 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:53.930584  3800 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.880839  3357 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:01:53.882901  3813 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:53.884063  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:53.939044  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 268) took 1988 ms. Trace:
W20260709 13:01:53.939653  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55775.7us
I20260709 13:01:53.939584  1739 rpcz_store.cc:276] 0709 13:01:51.950360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.697622 (+747262us) service_pool.cc:224] Handling call
0709 13:01:52.698003 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.752654 (+1054651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.938680 (+186026us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.938970 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2401235328}
I20260709 13:01:53.942068  3721 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.942468  3792 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:53.943039  3792 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:53.804083  3598 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.943437  3792 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:53.942508  3813 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 13:01:53.945245  3813 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:01:53.946476  3657 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:53.946875  3657 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:53.948130  3657 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.951632  3657 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:53.941256  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.954873  1739 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:53.956563  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:53.884668  1634 connection.cc:256] RPC call timeout handler was delayed by -1.4868738980000558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
I20260709 13:01:53.884778  3586 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.885529  3584 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.957980  3818 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:53.959105  3584 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.837853  3600 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.870038  3795 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:53.961745  3818 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:53.852141  3726 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.962124  3795 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:53.963032  3795 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:53.886814  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:53.966106  1662 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:53.966936  3790 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:53.967129  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.968055  1662 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:53.967976  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:53.890007  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.792860  3596 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:53.892269  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.893225  1586 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:01:53.893858  3834 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:53.894639  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:53.895658  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.897589  1738 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:53.899943  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.901850  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:53.902117  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1356) took 2030 ms (client timeout 2093 ms). Trace:
I20260709 13:01:53.910419  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:53.911326  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1930) took 3024 ms. Trace:
W20260709 13:01:53.915400  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1350) took 2047 ms (client timeout 1747 ms). Trace:
I20260709 13:01:53.916895  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2068) took 1772 ms. Trace:
I20260709 13:01:53.917016  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1929) took 1132 ms. Trace:
I20260709 13:01:53.917330  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:53.923930  3405 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:53.924471  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 267) took 2015 ms. Trace:
I20260709 13:01:54.023449  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:53.985004  1589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2049) took 1855 ms (client timeout 1805 ms). Trace:
I20260709 13:01:54.032394  1659 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:54.032052  1586 pending_rounds.cc:85] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:01:53.943471  3357 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 13:01:54.033134  1736 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:53.985992  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.035121  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.012234  1705 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:53.969344  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.042696  1662 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:54.044250  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.045229  1662 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:54.047590  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:53.819631  1661 rpcz_store.cc:276] 0709 13:01:49.589575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:49.616073 (+ 26498us) service_pool.cc:224] Handling call
0709 13:01:52.209253 (+2593180us) spinlock_profiling.cc:243] Waited 2.06 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:01:52.209592 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:52.231526 (+ 21934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.231560 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.580525 (+1348965us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:01:53.587093 (+  6568us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.673458 (+ 86365us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:53.676875 (+  3417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.683814 (+  6939us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:53.818539 (+134725us) raft_consensus.cc:1685] finished
0709 13:01:53.818568 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.818755 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15395,"raft.queue_time_us":16826,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":5355,"spinlock_wait_cycles":12383872,"thread_start_us":75104,"threads_started":1,"wal-append.queue_time_us":113937}
I20260709 13:01:54.053200  1662 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:54.055970  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.970023  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.036854  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:54.031798  3746 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.064803  1588 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:54.033182  1590 rpcz_store.cc:276] 0709 13:01:52.144533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.152056 (+1007523us) service_pool.cc:224] Handling call
0709 13:01:53.171966 (+ 19910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.814521 (+642555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.916575 (+102054us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b540031f454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.916788 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1475191040}
I20260709 13:01:54.067193  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.068819  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:54.070758  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.071895  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:54.031659  1742 rpcz_store.cc:276] 0709 13:01:51.908736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.660288 (+751552us) service_pool.cc:224] Handling call
0709 13:01:52.660795 (+   507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:52.959182 (+298387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:52.959221 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:52.959269 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:52.959298 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.923954 (+964656us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.924027 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.924269 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":685615872}
I20260709 13:01:54.074628  3746 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.074908  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:54.076835  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:54.031289  1665 rpcz_store.cc:276] 0709 13:01:50.887107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.697644 (+810537us) service_pool.cc:224] Handling call
0709 13:01:51.698023 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.770697 (+2072674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.910848 (+140151us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.911215 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4764053376}
I20260709 13:01:54.033250  1586 pending_rounds.cc:107] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:01:54.037961  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:54.071938  1588 pending_rounds.cc:85] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 4
I20260709 13:01:54.079845  1592 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.081652  1586 raft_consensus.cc:2889] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:54.079864  1588 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 13:01:54.083492  1588 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:54.032068  1666 rpcz_store.cc:269] 0709 13:01:51.871563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.823439 (+951876us) service_pool.cc:224] Handling call
0709 13:01:53.901752 (+1078313us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54003c8a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.902029 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2169864064}
I20260709 13:01:54.086928  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:54.088059  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.088865  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.039371  3834 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:54.040086  1634 connection.cc:256] RPC call timeout handler was delayed by -1.4697738980000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260709 13:01:54.074954  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.040810  1489 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 235) took 8401 ms. Trace:
I20260709 13:01:54.040899  1740 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:54.030678  1663 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 2.
I20260709 13:01:54.042095  1705 leader_election.cc:403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:01:53.974306  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:53.973490  3796 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:53.973099  3795 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:54.036825  3405 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:01:53.969712  3586 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.030434  1733 rpcz_store.cc:276] 0709 13:01:52.784810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.854597 (+ 69787us) service_pool.cc:224] Handling call
0709 13:01:52.854872 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.826059 (+971187us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:01:53.826109 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:53.826180 (+    71us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:53.826213 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.916575 (+ 90362us) spinlock_profiling.cc:243] Waited 966 ms on lock 0x7b5400435b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:53.916639 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.916783 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5665,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":279,"spinlock_wait_cycles":2221852544}
I20260709 13:01:53.970753  3584 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.024745  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:53.971143  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 542) took 1219 ms. Trace:
I20260709 13:01:54.016516  3761 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:54.032616  1589 rpcz_store.cc:269] 0709 13:01:52.129601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.322999 (+193398us) service_pool.cc:224] Handling call
0709 13:01:53.984605 (+1661606us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54003b1ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.984867 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3131427712}
I20260709 13:01:54.049839  1736 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:53.985966  3585 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:54.137097  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125165us
I20260709 13:01:54.117852  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:53.969260  2310 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:01:54.068819  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:54.079062  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:54.030757  1664 rpcz_store.cc:269] 0709 13:01:51.868432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.280497 (+412065us) service_pool.cc:224] Handling call
0709 13:01:53.915004 (+1634507us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540013d8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.915286 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3067716480}
I20260709 13:01:53.994617  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2069) took 1840 ms. Trace:
I20260709 13:01:54.037851  1734 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:53.970263  3583 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.089483  1662 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:54.090463  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.033721  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:54.091280  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 276) took 1643 ms. Trace:
I20260709 13:01:54.156585  1668 rpcz_store.cc:276] 0709 13:01:52.448143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.835793 (+1387650us) service_pool.cc:224] Handling call
0709 13:01:53.969722 (+133929us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:54.091211 (+121489us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34624896}
I20260709 13:01:53.988068  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.161849  1660 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:54.161927  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:54.162365  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 279) took 1474 ms (client timeout 1898 ms). Trace:
I20260709 13:01:54.162667  1668 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:54.162765  1660 rpcz_store.cc:269] 0709 13:01:52.688263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.813190 (+1124927us) service_pool.cc:224] Handling call
0709 13:01:53.987850 (+174660us) spinlock_profiling.cc:243] Waited 9.99 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:54.162273 (+174423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22967552}
I20260709 13:01:54.163161  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.163872  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.164007  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1371) took 1018 ms. Trace:
I20260709 13:01:54.164094  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.164305  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.164497  1740 rpcz_store.cc:276] 0709 13:01:53.146166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.119528 (+973362us) service_pool.cc:224] Handling call
0709 13:01:54.163946 (+ 44418us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":35456}
I20260709 13:01:54.165047  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:54.110771  1708 connection.cc:256] RPC call timeout handler was delayed by -1.1557462350000378s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1805s for a response
I20260709 13:01:54.112658  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.112931  1667 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:53.961572  3600 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.136652  3585 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:54.153918  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129376us
I20260709 13:01:54.089258  3805 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.164978  1739 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.121961  2318 rpcz_store.cc:276] 0709 13:01:52.758547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.758815 (+   268us) service_pool.cc:224] Handling call
0709 13:01:53.970810 (+1211995us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54006a0ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:53.971062 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2423720320}
I20260709 13:01:53.924254  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:54.109663  3796 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:54.159245  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.159570  1735 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:54.157932  3596 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.159845  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.156220  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1368) took 1011 ms. Trace:
I20260709 13:01:54.152486  1585 rpcz_store.cc:276] 0709 13:01:52.154170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.566630 (+1412460us) service_pool.cc:224] Handling call
0709 13:01:53.567867 (+  1237us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:53.568910 (+  1043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:53.569011 (+   101us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:53.579941 (+ 10930us) log.cc:844] Serialized 26 byte log entry
0709 13:01:53.618925 (+ 38984us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:53.618994 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:53.620258 (+  1264us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:53.994073 (+373815us) raft_consensus.cc:1685] finished
0709 13:01:53.994112 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:53.994354 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":38071,"threads_started":1,"wal-append.queue_time_us":141393}
I20260709 13:01:54.165402  1742 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:54.160683  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.184844  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.154685  1737 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:54.165468  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:54.226847  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 272) took 1868 ms (client timeout 1785 ms). Trace:
I20260709 13:01:54.198676  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.165608  1661 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:01:54.166525  1634 connection.cc:256] RPC call timeout handler was delayed by -1.1250738980000392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1948s for a response
I20260709 13:01:54.157843  1665 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:54.172631  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 287) took 1418 ms. Trace:
I20260709 13:01:54.228220  1734 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.165823  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.225574  1741 rpcz_store.cc:276] 0709 13:01:53.144778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.909886 (+765108us) service_pool.cc:224] Handling call
0709 13:01:54.156149 (+246263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.229055  1668 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:01:54.228977  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0197738980000395s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20650000000000002s for a response
I20260709 13:01:54.201267  1738 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:01:54.231321  1738 pending_rounds.cc:85] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:01:54.231666  1738 pending_rounds.cc:107] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.6
W20260709 13:01:54.231808  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95497.3us
I20260709 13:01:54.232017  1587 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:01:54.227393  1737 rpcz_store.cc:269] 0709 13:01:52.358560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.632232 (+1273672us) service_pool.cc:224] Handling call
0709 13:01:54.226767 (+594535us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.163166  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.231931  1738 raft_consensus.cc:2889] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:54.231161  1660 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.236840  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.236590  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:54.233319  3596 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.245581  1733 raft_consensus.cc:2376] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3. Re-sending same reply.
W20260709 13:01:54.257929  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.260723  3583 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.261395  3583 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.262099  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 259) took 3219 ms. Trace:
I20260709 13:01:54.262230  3596 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:54.262581  1738 rpcz_store.cc:276] 0709 13:01:51.042918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:51.044391 (+  1473us) service_pool.cc:224] Handling call
0709 13:01:51.061098 (+ 16707us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.636098 (+2575000us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b58007b14d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:01:54.261734 (+625636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.262012 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2139574016}
I20260709 13:01:54.263484  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.264055  3583 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:54.264813  3789 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.264887  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.265898  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.265611  3789 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.267390  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.268572  1589 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:54.235672  3405 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:54.273927  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.274829  3796 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:54.275736  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.273963  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:54.276527  3796 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:54.233655  3790 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:01:54.277213  1587 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:54.277817  3383 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.278256  3383 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:54.278424  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.278970  3383 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.279870  1587 raft_consensus.cc:2368] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:54.280992  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:54.282209  1587 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:01:54.282505  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1928) took 1504 ms (client timeout 1608 ms). Trace:
W20260709 13:01:54.283025  1735 rpcz_store.cc:269] 0709 13:01:52.778523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:52.789622 (+ 11099us) service_pool.cc:224] Handling call
0709 13:01:54.282157 (+1492535us) spinlock_profiling.cc:243] Waited 941 ms on lock 0x7b540012e3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:54.282393 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2164581120}
I20260709 13:01:54.283488  1587 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:54.284114  3818 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.284772  3818 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.241714  3731 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:54.179365  3834 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:54.186661  3761 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:54.228482  1584 rpcz_store.cc:276] 0709 13:01:52.753994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.705954 (+951960us) service_pool.cc:224] Handling call
0709 13:01:53.706601 (+   647us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.171015 (+464414us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:01:54.171063 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.171133 (+    70us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:54.171179 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.171983 (+   804us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:54.172125 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.172334 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":434,"raft.queue_time_us":21976,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":334,"spinlock_wait_cycles":998018816}
I20260709 13:01:54.291903  1741 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:01:54.303259  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.134338  3792 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.298822  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.196163  3586 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:54.290412  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 366399us
I20260709 13:01:54.304299  1737 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:54.285445  1587 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:01:54.293810  1585 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:01:54.309432  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.312244  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.304998  1590 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:54.312311  1740 raft_consensus.cc:2376] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5. Re-sending same reply.
I20260709 13:01:54.319629  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:54.320842  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74374us
I20260709 13:01:54.323750  1659 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:01:54.327534  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:54.313313  1587 pending_rounds.cc:85] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:01:54.332222  1587 pending_rounds.cc:107] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:01:54.332631  1587 raft_consensus.cc:2889] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:54.333907  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:54.334075  1738 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:01:54.334659  3818 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.324071  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:54.351042  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:54.352111  1592 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:01:54.356684  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.359302  1585 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:54.361192  1668 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:54.359270  1739 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:54.362502  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1930) took 1534 ms. Trace:
I20260709 13:01:54.363087  1738 rpcz_store.cc:276] 0709 13:01:52.807369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.333355 (+1525986us) service_pool.cc:224] Handling call
0709 13:01:54.333840 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.341108 (+  7268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.341407 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.288820  3584 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.367646  1591 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:54.308952  3849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.011s	sys 0.006s
W20260709 13:01:54.381027  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79423.5us
I20260709 13:01:54.386453  1586 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:01:54.387378  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1360) took 2483 ms. Trace:
W20260709 13:01:54.387662  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 156027us
I20260709 13:01:54.387993  1668 rpcz_store.cc:276] 0709 13:01:51.885470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.317158 (+2431688us) service_pool.cc:224] Handling call
0709 13:01:54.360854 (+ 43696us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.367851 (+  6997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.368155 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:54.392462  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70706.1us
I20260709 13:01:54.406765  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.279868  3789 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:54.378468  3849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.011s	sys 0.006s
I20260709 13:01:54.385385  1736 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:01:54.350086  3834 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:54.396893  1737 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:54.290946  3731 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.421351  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:54.314625  3583 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:54.425558  1662 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:54.431381  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:54.427708  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:01:54.418401  3605 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.666s	user 0.009s	sys 0.005s
I20260709 13:01:54.442719  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 272) took 2053 ms. Trace:
I20260709 13:01:54.446540  1662 rpcz_store.cc:276] 0709 13:01:52.383926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.339028 (+1955102us) service_pool.cc:224] Handling call
0709 13:01:54.425316 (+ 86288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.436742 (+ 11426us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.436993 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.349910  3790 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:01:54.414988  3357 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:54.451210  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.452109  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:54.294348  3585 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.458148  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1909) took 2406 ms. Trace:
I20260709 13:01:54.436296  3834 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:54.459525  3789 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 13:01:54.458235  3773 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:54.479590  1585 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:54.455260  3818 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.481640  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1934) took 1581 ms. Trace:
I20260709 13:01:54.485149  1740 rpcz_store.cc:276] 0709 13:01:52.893656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.377935 (+1484279us) service_pool.cc:224] Handling call
0709 13:01:54.454146 (+ 76211us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.474061 (+ 19915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.474110 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.474183 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:54.474239 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.474337 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.474531 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.108450  1489 rpcz_store.cc:276] 0709 13:01:45.708297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:45.715521 (+  7224us) service_pool.cc:224] Handling call
0709 13:01:47.350069 (+1634548us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf47090 00007fcdedfe5f3a 00007fcdf3d9c1cc 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc 00007fcdf235a392 00007fcdec31848c 00007fcdec3178c6
0709 13:01:54.040369 (+6690300us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:01:52.286833 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:01:52.300158 (+ 13325us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:01:52.300200 (+    42us) write_op.cc:435] Acquired schema lock
0709 13:01:52.300218 (+    18us) tablet.cc:662] Decoding operations
0709 13:01:52.304474 (+  4256us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:01:52.304568 (+    94us) write_op.cc:641] Partition lock acquired for write op
0709 13:01:52.304596 (+    28us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:01:52.305030 (+   434us) tablet.cc:701] Row locks acquired
0709 13:01:52.305051 (+    21us) write_op.cc:260] PREPARE: finished
0709 13:01:52.305213 (+   162us) write_op.cc:270] Start()
0709 13:01:52.305340 (+   127us) write_op.cc:276] Timestamp: P: 1783602112305160 usec, L: 0
0709 13:01:52.305367 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 13:01:52.422716 (+117349us) log.cc:844] Serialized 1764 byte log entry
0709 13:01:52.837543 (+414827us) op_driver.cc:464] REPLICATION: finished
0709 13:01:53.245847 (+408304us) write_op.cc:301] APPLY: starting
0709 13:01:53.246002 (+   155us) tablet.cc:1367] starting BulkCheckPresence
0709 13:01:53.246181 (+   179us) tablet.cc:1370] finished BulkCheckPresence
0709 13:01:53.255256 (+  9075us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:01:53.365521 (+110265us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:01:53.393770 (+ 28249us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0709 13:01:53.404682 (+ 10912us) write_op.cc:312] APPLY: finished
0709 13:01:53.416681 (+ 11999us) log.cc:844] Serialized 41 byte log entry
0709 13:01:53.420549 (+  3868us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:53.423734 (+  3185us) write_op.cc:454] Released schema lock
0709 13:01:53.425082 (+  1348us) write_op.cc:341] FINISH: Updating metrics
0709 13:01:53.430008 (+  4926us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:01:53.430740 (+   732us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":393,"reported_tablets":11,"spinlock_wait_cycles":1339466624,"child_traces":[["op",{"apply.queue_time_us":407637,"apply.run_cpu_time_us":7187,"apply.run_wall_time_us":182525,"num_ops":4,"prepare.queue_time_us":185773,"prepare.run_cpu_time_us":28670,"prepare.run_wall_time_us":379978,"raft.queue_time_us":187296,"raft.run_cpu_time_us":57846,"raft.run_wall_time_us":305297,"replication_time_us":531681,"thread_start_us":727743,"threads_started":4,"wal-append.queue_time_us":223546,"wal-append.run_cpu_time_us":30584,"wal-append.run_wall_time_us":1772783}]]}
W20260709 13:01:54.492808  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.493556  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.495909  1733 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:54.496253  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.473212  3834 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.433194  3584 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.500655  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:54.509713  1668 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:54.463243  1593 rpcz_store.cc:276] 0709 13:01:51.972321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:53.120866 (+1148545us) service_pool.cc:224] Handling call
0709 13:01:53.121327 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:53.122918 (+  1591us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:53.608552 (+485634us) log.cc:844] Serialized 14 byte log entry
0709 13:01:53.926732 (+318180us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b18002dd480. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:01:54.234726 (+307994us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.256620 (+ 21894us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:54.300973 (+ 44353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.301080 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.378570 (+ 77490us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":311588,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":65604,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":1266,"spinlock_wait_cycles":2424192,"thread_start_us":1036378,"threads_started":3,"wal-append.queue_time_us":172085}
I20260709 13:01:54.505558  1662 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:01:54.506294  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 85056.9us
I20260709 13:01:54.488829  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:54.499143  1742 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:54.515102  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1367) took 1370 ms. Trace:
I20260709 13:01:54.515725  1742 rpcz_store.cc:276] 0709 13:01:53.144440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.418229 (+1273789us) service_pool.cc:224] Handling call
0709 13:01:54.473872 (+ 55643us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.514735 (+ 40863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.515023 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:54.517932  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 464429us
I20260709 13:01:54.448298  3583 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:54.456785  3605 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.522260  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:54.488734  3818 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:54.521279  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:54.535914  1736 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 2
W20260709 13:01:54.537581  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.541114  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 273) took 1536 ms. Trace:
I20260709 13:01:54.541605  1738 rpcz_store.cc:276] 0709 13:01:52.989532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.394292 (+1404760us) service_pool.cc:224] Handling call
0709 13:01:54.454990 (+ 60698us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.517469 (+ 62479us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.517535 (+    66us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.525468 (+  7933us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:54.525554 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.525627 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.525776 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.505962  3585 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:54.548542  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 239199us
I20260709 13:01:54.507728  3805 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:54.549175  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 144985us
I20260709 13:01:54.558725  1662 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:54.536322  1736 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.3
I20260709 13:01:54.559513  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 274) took 2174 ms. Trace:
I20260709 13:01:54.567734  1662 rpcz_store.cc:276] 0709 13:01:52.384903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.519327 (+2134424us) service_pool.cc:224] Handling call
0709 13:01:54.550973 (+ 31646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.559200 (+  8227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.559448 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.539705  3818 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.534209  3596 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:01:54.577386  1736 raft_consensus.cc:2917] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted by new leader
W20260709 13:01:54.558327  3657 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.606s	user 0.006s	sys 0.009s
I20260709 13:01:54.582681  1742 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:54.586091  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1381) took 1373 ms. Trace:
W20260709 13:01:54.588014  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 143448us
I20260709 13:01:54.586506  1742 rpcz_store.cc:276] 0709 13:01:53.212707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.524522 (+1311815us) service_pool.cc:224] Handling call
0709 13:01:54.582399 (+ 57877us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.585687 (+  3288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.586014 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.598871  1740 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:01:54.599920  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1375) took 1420 ms. Trace:
I20260709 13:01:54.600380  1740 rpcz_store.cc:276] 0709 13:01:53.179405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.499915 (+1320510us) service_pool.cc:224] Handling call
0709 13:01:54.540936 (+ 41021us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.599596 (+ 58660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.599851 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.602265  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:54.586862  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:54.605753  3731 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:54.623255  3584 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.624519  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:54.626210  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:54.627368  3731 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:54.627658  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:54.627892  3731 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.628376  3731 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:54.628839  3805 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.595906  3792 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.630249  3792 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:54.631332  3583 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:54.631511  3657 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:01:54.632126  3583 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.632823  3583 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:54.634485  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86103.6us
I20260709 13:01:54.640293  1737 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:01:54.582048  3596 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:54.654280  1736 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:54.629317  3805 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.653311  3596 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:54.563238  3790 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 13:01:54.663630  1664 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:54.660955  3805 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.672118  1740 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:01:54.673269  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 296) took 1040 ms. Trace:
I20260709 13:01:54.673148  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:54.675544  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1385) took 1450 ms. Trace:
I20260709 13:01:54.676036  1740 rpcz_store.cc:276] 0709 13:01:53.224715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.646672 (+1421957us) service_pool.cc:224] Handling call
0709 13:01:54.670940 (+ 24268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.675164 (+  4224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.675453 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:54.676934  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89628.5us
I20260709 13:01:54.666131  3600 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:54.685431  1742 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
W20260709 13:01:54.688110  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.675225  1588 rpcz_store.cc:276] 0709 13:01:53.633243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.071292 (+438049us) service_pool.cc:224] Handling call
0709 13:01:54.071685 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.084626 (+ 12941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.084676 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.085311 (+   635us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.433213 (+347902us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:54.668671 (+235458us) log.cc:844] Serialized 14 byte log entry
0709 13:01:54.672921 (+  4250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.673006 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:54.673051 (+    45us) raft_consensus.cc:1685] finished
0709 13:01:54.673063 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.673207 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5437,"thread_start_us":546948,"threads_started":2,"wal-append.queue_time_us":483968}
I20260709 13:01:54.691371  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1384) took 1470 ms. Trace:
I20260709 13:01:54.691972  1742 rpcz_store.cc:276] 0709 13:01:53.221215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.597868 (+1376653us) service_pool.cc:224] Handling call
0709 13:01:54.667082 (+ 69214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.690872 (+ 23790us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.691284 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:54.677342  3789 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:01:54.708622  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 125252us
I20260709 13:01:54.715128  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:54.717893  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48649.5us
I20260709 13:01:54.698493  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:54.722389  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.705303  3761 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.707077  3789 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:54.706270  3792 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.733139  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:54.720186  3734 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:54.711330  3795 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.753942  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:54.724699  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:54.771647  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:54.786098  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 238438us
I20260709 13:01:54.731185  3800 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:54.790771  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.742822  3792 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:54.796873  1734 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.568s	user 0.013s	sys 0.002s
W20260709 13:01:54.797135  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50430.7us
I20260709 13:01:54.747115  3789 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:54.755334  3605 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.747860  3357 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:54.799275  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:54.787949  3771 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.754776  3734 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:54.769836  3815 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:54.829545  1734 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:54.840777  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 346903us
W20260709 13:01:54.841862  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97724.8us
W20260709 13:01:54.804673  3383 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.525s	user 0.006s	sys 0.010s
W20260709 13:01:54.846304  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56236.2us
I20260709 13:01:54.804105  3792 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.849175  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:54.860956  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:54.844805  3583 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:54.800179  3795 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.805229  3800 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:54.884425  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:54.846992  3824 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:54.855280  3657 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:01:54.898321  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49284.5us
I20260709 13:01:54.855319  3357 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:54.928877  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 475813us
I20260709 13:01:54.915649  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:54.850751  3383 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:54.884275  3605 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:54.926182  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:54.946684  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 139257us
W20260709 13:01:54.953900  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 245887us
I20260709 13:01:54.899999  3824 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:54.851029  3771 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:54.944357  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:54.926873  3657 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:54.975323  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 129787us
W20260709 13:01:54.998167  1591 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.621s	user 0.007s	sys 0.007s
I20260709 13:01:54.998694  1591 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
W20260709 13:01:54.998086  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51073.7us
W20260709 13:01:54.999739  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:55.007699  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.010082  2318 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:54.960718  3605 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:55.011426  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 295682us
I20260709 13:01:55.012214  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1362) took 2792 ms. Trace:
I20260709 13:01:55.015758  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:54.887715  3815 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:55.024725  1659 rpcz_store.cc:276] 0709 13:01:52.207806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.352002 (+2144196us) service_pool.cc:224] Handling call
0709 13:01:54.442013 (+ 90011us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.480233 (+ 38220us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:54.802861 (+322628us) log.cc:844] Serialized 14 byte log entry
0709 13:01:54.999465 (+196604us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:54.999530 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:54.999581 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.999652 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:54.999804 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5795,"prepare.queue_time_us":298623,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":3054,"thread_start_us":411904,"threads_started":2}
I20260709 13:01:54.953281  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:54.974323  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:55.038487  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44328.1us
I20260709 13:01:55.004813   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:01:54.994450  3583 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:01:55.040614  1660 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.738s	user 0.014s	sys 0.003s
I20260709 13:01:55.042253  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1942) took 3288 ms. Trace:
I20260709 13:01:55.060936  1660 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:55.062867  1659 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:01:55.063683  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 277) took 2592 ms. Trace:
I20260709 13:01:55.064150  1659 rpcz_store.cc:276] 0709 13:01:52.471726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.029393 (+2557667us) service_pool.cc:224] Handling call
0709 13:01:55.055130 (+ 25737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.063373 (+  8243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.063620 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.065447  1667 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:55.030014  3605 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:54.974447  3889 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:55.038707  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:55.049019  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.050076  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1931) took 2232 ms. Trace:
I20260709 13:01:55.081398  1735 rpcz_store.cc:276] 0709 13:01:52.818033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.340625 (+1522592us) service_pool.cc:224] Handling call
0709 13:01:54.394685 (+ 54060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.455666 (+ 60981us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:54.678066 (+222400us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.049451 (+371385us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.049542 (+    91us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:55.049607 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.049704 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.049996 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80389,"prepare.queue_time_us":280565,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"thread_start_us":402590,"threads_started":3}
I20260709 13:01:54.993028  3657 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:01:55.051029  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 122473us
W20260709 13:01:55.054385  3882 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 13:01:55.053660  3357 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.050172  3584 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 13:01:55.060827  1663 rpcz_store.cc:276] 0709 13:01:51.753723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.293109 (+2539386us) service_pool.cc:224] Handling call
0709 13:01:54.293737 (+   628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.294108 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:54.521759 (+227651us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.041765 (+520006us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.041831 (+    66us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:55.041886 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.041978 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.042161 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":388650,"prepare.run_cpu_time_us":272,"prepare.run_wall_time_us":7871,"thread_start_us":702767,"threads_started":3,"wal-append.queue_time_us":422022}
I20260709 13:01:54.974346  3600 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:55.050400  3774 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:55.084848  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 84611.4us
I20260709 13:01:55.078321  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:55.091346  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 248808us
I20260709 13:01:55.119179  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.088049  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1932) took 2256 ms. Trace:
I20260709 13:01:55.073094  3583 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:55.118255  1660 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
W20260709 13:01:55.122735  2314 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.002s
I20260709 13:01:55.152233  2314 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:55.128604  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:55.129594  3888 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 13:01:55.096752  3734 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.068466  3731 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.112486  3824 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.153127  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 281) took 2428 ms. Trace:
W20260709 13:01:55.166692  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58589.9us
W20260709 13:01:55.167246  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.281648  1660 rpcz_store.cc:276] 0709 13:01:52.724587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.062521 (+2337934us) service_pool.cc:224] Handling call
0709 13:01:55.102755 (+ 40234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.152222 (+ 49467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.153030 (+   808us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:55.164620  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51065.5us
W20260709 13:01:55.207878  3805 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.000s
I20260709 13:01:55.226193  3383 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.237354  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1372) took 2090 ms. Trace:
I20260709 13:01:55.268747  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1936) took 1401 ms. Trace:
W20260709 13:01:55.230515  3796 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.953s	user 0.018s	sys 0.000s
I20260709 13:01:55.219096  3795 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.227516  3815 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.125458  3642 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.106s
I20260709 13:01:55.275348  3584 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election won for term 3
W20260709 13:01:55.233026  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 300) took 1231 ms (client timeout 1638 ms). Trace:
I20260709 13:01:55.107357  3405 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.127507  1741 rpcz_store.cc:276] 0709 13:01:52.831474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.377735 (+1546261us) service_pool.cc:224] Handling call
0709 13:01:54.447394 (+ 69659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.488587 (+ 41193us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:01:54.806527 (+317940us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.087369 (+280842us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.087442 (+    73us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:55.087500 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.087581 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.087757 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47150,"prepare.queue_time_us":353502,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":433759,"threads_started":3,"wal-append.queue_time_us":254162}
I20260709 13:01:55.233805  3761 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:55.225942  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 251297us
I20260709 13:01:55.277063  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1943) took 3523 ms. Trace:
I20260709 13:01:55.287576  3824 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:55.235961  3800 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:55.199249  3878 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
W20260709 13:01:55.208567  1557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.036s (SENT) [suppressed 25 similar messages]
W20260709 13:01:55.235364  3880 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.347s	user 0.000s	sys 0.001s
W20260709 13:01:55.233659  3885 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 13:01:55.255898  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 301) took 1177 ms. Trace:
I20260709 13:01:55.232482  3899 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:55.136951  3881 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:55.169622  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1382) took 1953 ms. Trace:
I20260709 13:01:55.413590  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:55.413995  1738 rpcz_store.cc:276] 0709 13:01:53.216218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.572327 (+1356109us) service_pool.cc:224] Handling call
0709 13:01:54.662683 (+ 90356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.699721 (+ 37038us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:54.903033 (+203312us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.169129 (+266096us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.169209 (+    80us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:55.169270 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.169365 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.169542 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65966,"prepare.queue_time_us":141216,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3925,"raft.queue_time_us":241053,"spinlock_wait_cycles":22490496,"thread_start_us":275063,"threads_started":2,"wal-append.queue_time_us":374314}
I20260709 13:01:55.283174  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.279173  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:55.285262  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 229779us
W20260709 13:01:55.288564  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 279680us
I20260709 13:01:55.293092  1735 rpcz_store.cc:276] 0709 13:01:53.866958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.098979 (+1232021us) service_pool.cc:224] Handling call
0709 13:01:55.118951 (+ 19972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.207866 (+ 88915us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.207896 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.207951 (+    55us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:55.268359 (+ 60408us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.268469 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.268659 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:55.294060  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.293056  1736 rpcz_store.cc:276] 0709 13:01:53.146513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.436698 (+1290185us) service_pool.cc:224] Handling call
0709 13:01:54.485837 (+ 49139us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.787743 (+301906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.791546 (+  3803us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.808309 (+ 16763us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.978664 (+170355us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:54.978720 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.978840 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.237129 (+258289us) raft_consensus.cc:1685] finished
0709 13:01:55.237150 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.237295 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":90551,"threads_started":1,"wal-append.queue_time_us":377965}
I20260709 13:01:55.297191  1660 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:55.305706  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1378) took 1205 ms. Trace:
I20260709 13:01:55.358747  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.418301  1664 rpcz_store.cc:276] 0709 13:01:54.099849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.152288 (+ 52439us) service_pool.cc:224] Handling call
0709 13:01:55.305428 (+1153140us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:55.305618 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1007980288}
W20260709 13:01:55.418774  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 135688us
I20260709 13:01:55.359357  2313 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:01:55.291528  3889 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:55.291553  3805 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.300995  3734 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:55.425542  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:55.377761  1591 rpcz_store.cc:269] 0709 13:01:54.001789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.036444 (+ 34655us) service_pool.cc:224] Handling call
0709 13:01:55.232842 (+1196398us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:55.232963 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":708023552}
W20260709 13:01:55.312779  3818 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.677s	user 0.009s	sys 0.005s
I20260709 13:01:55.384282  3584 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:55.393854  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 276) took 1529 ms. Trace:
I20260709 13:01:55.358335  3907 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:55.394599  3600 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.395376  3731 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:55.396847  3596 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.726s	user 0.010s	sys 0.002s
I20260709 13:01:55.398011  3726 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:55.399300  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1937) took 4386 ms. Trace:
W20260709 13:01:55.401039  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.406319  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1363) took 3190 ms. Trace:
I20260709 13:01:55.406044  3583 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.380497  1665 rpcz_store.cc:276] 0709 13:01:51.754062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.315982 (+2561920us) service_pool.cc:224] Handling call
0709 13:01:54.421009 (+105027us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.451872 (+ 30863us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.451923 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.478452 (+ 26529us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.804956 (+326504us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:54.847367 (+ 42411us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.276348 (+428981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.276612 (+   264us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.276687 (+    75us) raft_consensus.cc:1685] finished
0709 13:01:55.276710 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.276974 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":49652,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":222839,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":13647,"thread_start_us":695118,"threads_started":2,"wal-append.queue_time_us":367916}
I20260709 13:01:55.412572  1592 rpcz_store.cc:276] 0709 13:01:54.078985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.394544 (+315559us) service_pool.cc:224] Handling call
0709 13:01:54.425088 (+ 30544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.482558 (+ 57470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.482591 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.483370 (+   779us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.690686 (+207316us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:54.690748 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.690864 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.255283 (+564419us) raft_consensus.cc:1685] finished
0709 13:01:55.255317 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.255766 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75171,"raft.queue_time_us":41351,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":14671,"thread_start_us":206806,"threads_started":1,"wal-append.queue_time_us":321329}
I20260709 13:01:55.377898  3815 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:55.448185  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:55.423310  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1958) took 1451 ms. Trace:
I20260709 13:01:55.449239  1660 rpcz_store.cc:276] 0709 13:01:53.966830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.282575 (+1315745us) service_pool.cc:224] Handling call
0709 13:01:55.296709 (+ 14134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.418228 (+121519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.418576 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.383267  3405 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:55.383481  3761 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:55.426863  1739 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.009s	sys 0.007s
I20260709 13:01:55.429723  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 279) took 1560 ms. Trace:
I20260709 13:01:55.422960  3849 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:01:55.439249  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80619.9us
I20260709 13:01:55.441164  1667 rpcz_store.cc:276] 0709 13:01:51.012640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.275395 (+3262755us) service_pool.cc:224] Handling call
0709 13:01:55.062062 (+786667us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.068488 (+  6426us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.399024 (+330536us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54002f3350. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:55.399185 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1808167680}
I20260709 13:01:55.454238  1664 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:55.463234  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1373) took 1391 ms. Trace:
I20260709 13:01:55.463701  1664 rpcz_store.cc:276] 0709 13:01:54.071760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.427366 (+1355606us) service_pool.cc:224] Handling call
0709 13:01:55.447606 (+ 20240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.462844 (+ 15238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.463160 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.431041  1740 rpcz_store.cc:276] 0709 13:01:53.864297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.677892 (+813595us) service_pool.cc:224] Handling call
0709 13:01:54.690576 (+ 12684us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.713947 (+ 23371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.713981 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.735974 (+ 21993us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.942031 (+206057us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:54.942084 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.942173 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.393581 (+451408us) raft_consensus.cc:1685] finished
0709 13:01:55.393608 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.393785 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9689,"raft.queue_time_us":320094,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":30238,"thread_start_us":133439,"threads_started":1,"wal-append.queue_time_us":427884}
I20260709 13:01:55.456766  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1935) took 2508 ms. Trace:
I20260709 13:01:55.456403  1738 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 13:01:55.447764  3834 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.884s	user 0.012s	sys 0.002s
I20260709 13:01:55.456086  1734 rpcz_store.cc:276] 0709 13:01:53.865270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.955374 (+1090104us) service_pool.cc:224] Handling call
0709 13:01:54.999271 (+ 43897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.056675 (+ 57404us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:55.125915 (+ 69240us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.425197 (+299282us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.425260 (+    63us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:55.425310 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.425383 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.425538 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17190,"prepare.queue_time_us":52183,"prepare.run_cpu_time_us":235,"prepare.run_wall_time_us":3605,"raft.queue_time_us":18934,"thread_start_us":207753,"threads_started":1}
W20260709 13:01:55.459440  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.309835  3900 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:55.468220  1739 rpcz_store.cc:276] 0709 13:01:52.948313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.409447 (+1461134us) service_pool.cc:224] Handling call
0709 13:01:54.452906 (+ 43459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.488282 (+ 35376us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:54.704126 (+215844us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.456089 (+751963us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.456169 (+    80us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:55.456217 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.456298 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.456634 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78063,"prepare.queue_time_us":272134,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":226659,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":53660,"thread_start_us":908480,"threads_started":3,"wal-append.queue_time_us":335403}
I20260709 13:01:55.485395  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1937) took 1603 ms. Trace:
I20260709 13:01:55.485958  1738 rpcz_store.cc:276] 0709 13:01:53.867273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.415616 (+1548343us) service_pool.cc:224] Handling call
0709 13:01:55.426671 (+ 11055us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.468731 (+ 42060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.471083 (+  2352us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:55.489496  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69167us
I20260709 13:01:55.440893  3726 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:55.501226  1665 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:01:55.501603  1740 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:55.446802  3605 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.489959  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:55.455029  3774 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:55.525569  3892 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:55.526149  1659 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:55.526511  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 289953us
I20260709 13:01:55.527405  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:55.457844  3889 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.472625  3899 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:55.515982  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92765.2us
I20260709 13:01:55.454444  3907 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:55.516672  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1380) took 1379 ms. Trace:
I20260709 13:01:55.541489  1665 rpcz_store.cc:276] 0709 13:01:54.122494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.470088 (+1347594us) service_pool.cc:224] Handling call
0709 13:01:55.488816 (+ 18728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.501806 (+ 12990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.502049 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.450258  3596 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.518956  2318 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:55.460561  3771 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.469802  3849 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.522362  3726 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.522513  3584 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.469480  3909 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:55.517716  3642 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:55.523717  3903 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:55.586508  3925 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:01:55.586707  3903 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.587924  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1380) took 2379 ms. Trace:
I20260709 13:01:55.589057  3903 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.524641  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.588332  1733 rpcz_store.cc:276] 0709 13:01:53.208643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.496758 (+1288115us) service_pool.cc:224] Handling call
0709 13:01:54.497188 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.842573 (+345385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.842608 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.880896 (+ 38288us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.429468 (+548572us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:55.431611 (+  2143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.432147 (+   536us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.587687 (+155540us) raft_consensus.cc:1685] finished
0709 13:01:55.587709 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.587858 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32978,"raft.queue_time_us":86556,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":10595,"thread_start_us":779694,"threads_started":2,"wal-append.queue_time_us":602688}
W20260709 13:01:55.589933  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65393.3us
I20260709 13:01:55.590276  3889 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:55.590668  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.513370  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:55.538195  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1941) took 1508 ms. Trace:
I20260709 13:01:55.525174  3910 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.596444  3910 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.597790  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:55.598829  2310 raft_consensus.cc:2368] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:01:55.599502  3910 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:55.600142  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:55.604034  3919 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 13:01:55.608871  3926 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:55.609107  3726 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:55.619652  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:55.429994  3912 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:55.472662  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 271) took 4489 ms. Trace:
I20260709 13:01:55.488824  3893 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.544967  3896 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:55.579715  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:55.553974  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141488us
W20260709 13:01:55.575183  1706 leader_election.cc:336] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.517s (SENT)
I20260709 13:01:55.574709  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 299) took 1631 ms. Trace:
I20260709 13:01:55.578615  1739 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 13:01:55.579247  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 370713us
I20260709 13:01:55.486441  3834 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.491078  3815 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.591935  1734 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:01:55.530613  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 298) took 1593 ms. Trace:
I20260709 13:01:55.436949  3818 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.591939  1740 rpcz_store.cc:276] 0709 13:01:54.029811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.468354 (+1438543us) service_pool.cc:224] Handling call
0709 13:01:55.501381 (+ 33027us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.502061 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.538081 (+ 36020us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.609531  3926 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:55.613387  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.713621  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:55.531263  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:55.553709  3915 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.552488  3904 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:55.644572  3792 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.744s	user 0.011s	sys 0.005s
I20260709 13:01:55.577939  3849 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:55.664177  1591 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:55.688035  2312 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:55.552425  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:55.714214  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:55.714707  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 283) took 1136 ms. Trace:
I20260709 13:01:55.714134  1584 rpcz_store.cc:276] 0709 13:01:53.943213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.349426 (+406213us) service_pool.cc:224] Handling call
0709 13:01:54.395852 (+ 46426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.464429 (+ 68577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.464463 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.471484 (+  7021us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.699912 (+228428us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:54.699964 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.700061 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.573899 (+873838us) raft_consensus.cc:1685] finished
0709 13:01:55.573929 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.574615 (+   686us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101040,"raft.queue_time_us":36544,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":10717,"thread_start_us":227986,"threads_started":1,"wal-append.queue_time_us":332971}
I20260709 13:01:55.619086  3731 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:55.692981  1666 rpcz_store.cc:276] 0709 13:01:50.983207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.088043 (+3104836us) service_pool.cc:224] Handling call
0709 13:01:54.088421 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.088869 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.088907 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.089482 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.582812 (+493330us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:54.582854 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.582928 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.472435 (+889507us) raft_consensus.cc:1685] finished
0709 13:01:55.472461 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.472600 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"thread_start_us":464192,"threads_started":1,"wal-append.queue_time_us":590171}
I20260709 13:01:55.726090  1739 rpcz_store.cc:276] 0709 13:01:54.578542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.495444 (+916902us) service_pool.cc:224] Handling call
0709 13:01:55.520211 (+ 24767us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.714309 (+194098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.714625 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:01:55.569931  3795 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.111s
I20260709 13:01:55.586731  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:55.737226  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.716020  1590 rpcz_store.cc:276] 0709 13:01:53.937330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.333400 (+396070us) service_pool.cc:224] Handling call
0709 13:01:54.388511 (+ 55111us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.479599 (+ 91088us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.479636 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.518684 (+ 39048us) log.cc:844] Serialized 26 byte log entry
0709 13:01:54.762155 (+243471us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:54.762238 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:54.762409 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:55.530366 (+767957us) raft_consensus.cc:1685] finished
0709 13:01:55.530390 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.530547 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71257,"raft.queue_time_us":36610,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8478,"thread_start_us":198434,"threads_started":1,"wal-append.queue_time_us":311868}
W20260709 13:01:55.713639  3922 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:01:55.526156  3892 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.718273  3896 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:55.754462  1736 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:55.747385  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81491.2us
I20260709 13:01:55.722189  3926 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:55.792430  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:55.788650  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.680186  3800 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
I20260709 13:01:55.773846  3896 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:55.718590  3910 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:55.712867  3771 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260709 13:01:55.739095  3889 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.774834  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:55.796973  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 269644us
I20260709 13:01:55.806614  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:55.831626  1665 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:01:55.736735  3585 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.182s	user 0.007s	sys 0.011s
W20260709 13:01:55.810235  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 235104us
W20260709 13:01:55.822314  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 329570us
I20260709 13:01:55.728921  3792 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.836624  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 1952) took 2193 ms. Trace:
I20260709 13:01:55.838440  1659 rpcz_store.cc:276] 0709 13:01:53.643075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.098398 (+1455323us) service_pool.cc:224] Handling call
0709 13:01:55.111829 (+ 13431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.526618 (+414789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.836371 (+309753us) spinlock_profiling.cc:243] Waited 414 ms on lock 0x7b54001a8bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:55.836560 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":952404224}
I20260709 13:01:55.839318  1557 leader_election.cc:400] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:01:55.839798  1557 leader_election.cc:403] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 13:01:55.841158  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:55.835625  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:55.849663  1668 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:01:55.852727  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:55.855892  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.858474  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:55.859405  1741 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:55.727962  3904 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.790141  2309 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:55.867134  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:55.875597  1659 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:55.732492  3915 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.842013  3584 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:55.877141  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:55.877867  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.880312  1665 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:55.881878  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:55.888669  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.850979  3900 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:55.890161  1593 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 13:01:55.859045  3910 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:55.894816  3910 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.855281  3805 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:55.766518  3893 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:55.904922  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 65764.6us
I20260709 13:01:55.908372  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:55.911351  1740 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:55.901185  1665 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:55.909265  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:55.902055  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:55.906430  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:55.918453  1741 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:55.919215  1659 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:55.831799  3881 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:01:55.842381  3889 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:55.897337  3910 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:55.831826  3899 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:55.940922  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92395us
I20260709 13:01:55.941372  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:55.946204  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:55.948941  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98862.3us
W20260709 13:01:55.949213  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 143173us
W20260709 13:01:55.952132  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.898666  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 586) took 1097 ms. Trace:
I20260709 13:01:55.954078  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:55.898396  3900 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:55.962625  1733 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:01:55.943516  3642 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:55.988489  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:55.957185  2309 rpcz_store.cc:276] 0709 13:01:54.795027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.857532 (+ 62505us) service_pool.cc:224] Handling call
0709 13:01:55.586490 (+728958us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:01:55.892186 (+305696us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38793088}
I20260709 13:01:55.966214  1740 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:55.908996  3584 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.006793  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:55.963479  3900 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:55.958941  3920 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:55.828425  3907 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:55.960685  3596 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.016887  1740 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.009711  3357 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:56.017729  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 117575us
W20260709 13:01:56.016175  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 106428us
I20260709 13:01:56.030660  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:01:56.025537  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72186.6us
I20260709 13:01:56.038058  1659 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:56.041023  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 289) took 1582 ms. Trace:
I20260709 13:01:56.041541  1659 rpcz_store.cc:276] 0709 13:01:54.458368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.039320 (+1580952us) service_pool.cc:224] Handling call
0709 13:01:56.040096 (+   776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.040440 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.040465 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:56.040549 (+    84us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:56.040594 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.040673 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.040847 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:01:56.028714  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 77734.3us
I20260709 13:01:56.046751  1585 pending_rounds.cc:85] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 4
I20260709 13:01:56.047171  1585 pending_rounds.cc:107] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 13:01:56.047473  1585 raft_consensus.cc:2889] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:56.051314  3935 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:01:56.001681  3931 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:01:56.018573  3818 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:56.013969  3937 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.059309  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:56.028949  3834 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:56.030148  3357 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:01:56.065366  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.067312  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:56.068629  3928 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.232s	user 0.002s	sys 0.000s
I20260709 13:01:56.069808  3568 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.071125  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1368) took 2425 ms. Trace:
W20260709 13:01:56.072973  3918 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.457s	user 0.000s	sys 0.002s
I20260709 13:01:56.073668  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 582) took 1384 ms. Trace:
I20260709 13:01:56.074079  2313 rpcz_store.cc:276] 0709 13:01:54.689572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.701769 (+ 12197us) service_pool.cc:224] Handling call
0709 13:01:56.073434 (+1371665us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:56.073605 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1474720000}
I20260709 13:01:56.074785  3936 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:56.075677  3936 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:56.071586  1663 rpcz_store.cc:276] 0709 13:01:53.645339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.108805 (+1463466us) service_pool.cc:224] Handling call
0709 13:01:55.135793 (+ 26988us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.269195 (+133402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.269243 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.309622 (+ 40379us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.576424 (+266802us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:01:55.576481 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.576562 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.070757 (+494195us) raft_consensus.cc:1685] finished
0709 13:01:56.070789 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.070983 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16859,"raft.queue_time_us":188541,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":34901,"thread_start_us":143195,"threads_started":1,"wal-append.queue_time_us":361253}
I20260709 13:01:56.077809  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 274) took 2214 ms. Trace:
I20260709 13:01:56.052026  3886 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.034967  3800 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.052311  3920 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:56.083935  1740 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:56.078235  1737 rpcz_store.cc:276] 0709 13:01:53.863631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.657681 (+794050us) service_pool.cc:224] Handling call
0709 13:01:54.671070 (+ 13389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.961722 (+290652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.973570 (+ 11848us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.015554 (+ 41984us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.285292 (+269738us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:55.285342 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.285422 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.077581 (+792159us) raft_consensus.cc:1685] finished
0709 13:01:56.077604 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.077743 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":163580,"raft.queue_time_us":370532,"raft.run_cpu_time_us":467,"raft.run_wall_time_us":19644,"thread_start_us":447310,"threads_started":2,"wal-append.queue_time_us":431043}
W20260709 13:01:56.091801  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57414.4us
W20260709 13:01:56.069377  3890 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.988s	user 0.003s	sys 0.003s
I20260709 13:01:56.048784  3910 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 13:01:56.013830  3731 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:56.102712  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79804.5us
I20260709 13:01:56.109808  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.110096  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:01:56.109802  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50614.2us
I20260709 13:01:56.111083  1740 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:56.111178  1737 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.112423  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:56.125226  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 234096us
I20260709 13:01:56.127841  3929 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:01:56.129703  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68607.6us
I20260709 13:01:56.129266  3920 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.131183  3940 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.131616  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:56.077343  3936 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.136039  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 284) took 1345 ms. Trace:
I20260709 13:01:56.136562  1708 leader_election.cc:400] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:56.136983  1708 leader_election.cc:403] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 13:01:56.133219  3931 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:56.141482  1741 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:56.103951  3792 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:56.136544  1738 rpcz_store.cc:276] 0709 13:01:54.790488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.486868 (+696380us) service_pool.cc:224] Handling call
0709 13:01:55.513598 (+ 26730us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.697947 (+184349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.697982 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.698529 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.918021 (+219492us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:55.918072 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.918172 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.135792 (+217620us) raft_consensus.cc:1685] finished
0709 13:01:56.135815 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.135973 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70270,"raft.queue_time_us":160643,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":56001,"thread_start_us":328743,"threads_started":2,"wal-append.queue_time_us":355397}
I20260709 13:01:56.106328  3585 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:56.164029  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.105583  3800 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.190058  1737 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:56.191859  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.192880  1737 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
W20260709 13:01:56.194088  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50393.9us
I20260709 13:01:56.137329  3834 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:56.098333  3886 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.196882  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.203105  3771 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:56.203969  3726 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.001s
I20260709 13:01:56.204650  3726 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.205807  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.207675  3934 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:56.208292  3934 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:56.170945  3916 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:01:56.209369  3934 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.157917  3818 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4.
I20260709 13:01:56.098469  3596 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260709 13:01:56.197188  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:01:56.219153  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44919.3us
I20260709 13:01:56.142352  3892 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:56.212652  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67663.3us
I20260709 13:01:56.144440  3931 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:56.179466  3585 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:56.174304  3600 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20260709 13:01:56.229740  3921 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.287s	user 0.000s	sys 0.002s
I20260709 13:01:56.230430  3815 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.148130  3357 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.231119  3941 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:56.232138  3357 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.233667  3924 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:56.230908  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:56.245584  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.247123  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 589) took 1287 ms. Trace:
I20260709 13:01:56.196557  3834 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.266743  1737 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:01:56.205184  3726 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.249907  2316 rpcz_store.cc:276] 0709 13:01:54.947675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.947826 (+   151us) service_pool.cc:224] Handling call
0709 13:01:55.002518 (+ 54692us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:01:55.050823 (+ 48305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.050858 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:01:55.089126 (+ 38268us) log.cc:844] Serialized 52 byte log entry
0709 13:01:55.413801 (+324675us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:55.413843 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.413915 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.235028 (+821113us) raft_consensus.cc:1685] finished
0709 13:01:56.235051 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.235235 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18751,"raft.queue_time_us":61874,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":14363,"thread_start_us":246902,"threads_started":1,"wal-append.queue_time_us":727129}
I20260709 13:01:56.238983  3585 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:56.268908  1738 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:56.230908  3904 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:56.196878  3792 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 13:01:56.240723  3893 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.268597  3834 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:56.273938  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71902.6us
I20260709 13:01:56.276175  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:56.228788  3931 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:56.288533  3929 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:01:56.274103  3731 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.288842  3584 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.298409  3956 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:56.298887  3956 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:56.299317  3956 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:56.274358  3761 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.790s	user 0.013s	sys 0.002s
I20260709 13:01:56.273768  3947 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:56.276939  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.307319  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.148294  3642 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.308611  1737 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:56.286803  2315 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:01:56.303812  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 282) took 2045 ms. Trace:
I20260709 13:01:56.308977  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:56.312569  3734 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.883s	user 0.011s	sys 0.003s
W20260709 13:01:56.314010  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 126845us
I20260709 13:01:56.374696  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.374818  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1386) took 2418 ms. Trace:
I20260709 13:01:56.314710  3726 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:56.316797  3909 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:56.317343  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74830.3us
W20260709 13:01:56.318383  3824 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.909s	user 0.012s	sys 0.001s
W20260709 13:01:56.359396  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.280747  3596 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:56.323134  1734 rpcz_store.cc:276] 0709 13:01:54.257998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.471125 (+1213127us) service_pool.cc:224] Handling call
0709 13:01:55.471450 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.715462 (+244012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.303556 (+588094us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:56.303744 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276633344}
W20260709 13:01:56.364387  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:56.371191  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95117.9us
W20260709 13:01:56.372601  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63698.3us
W20260709 13:01:56.314599  3932 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.281s	user 0.001s	sys 0.001s
I20260709 13:01:56.375419  1735 rpcz_store.cc:276] 0709 13:01:53.956672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.428302 (+1471630us) service_pool.cc:224] Handling call
0709 13:01:55.451373 (+ 23071us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:55.550406 (+ 99033us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:55.550460 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:55.581716 (+ 31256us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.084819 (+503103us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:56.352528 (+267709us) log.cc:844] Serialized 14 byte log entry
0709 13:01:56.374336 (+ 21808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.374464 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.374526 (+    62us) raft_consensus.cc:1685] finished
0709 13:01:56.374545 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.374743 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103300,"raft.queue_time_us":90550,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":12561,"thread_start_us":602215,"threads_started":2,"wal-append.queue_time_us":460973}
W20260709 13:01:56.378664  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 298694us
I20260709 13:01:56.383176  1734 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:56.384529  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:56.390098  1659 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:56.393045  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.382863  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:56.394428  1706 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.941s (SENT)
I20260709 13:01:56.395022  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:56.395479  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.396723  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:56.405038  3824 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:56.405658  3824 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.406030  3824 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:56.407534  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.408298  2313 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:56.409575  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.409746  3947 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:01:56.411072  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:56.411515  3914 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:56.411727  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:01:56.412081  3914 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:56.416616  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:56.418454  3949 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 13:01:56.380201  3909 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:55.442713  1661 rpcz_store.cc:276] 0709 13:01:52.216230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.349201 (+2132971us) service_pool.cc:224] Handling call
0709 13:01:54.435836 (+ 86635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:54.460060 (+ 24224us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:01:54.979932 (+519872us) log.cc:844] Serialized 14 byte log entry
0709 13:01:55.405869 (+425937us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:55.405934 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:55.405998 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.406085 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:55.406247 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100143,"prepare.queue_time_us":268806,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":115,"raft.queue_time_us":108920,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":9989,"thread_start_us":726945,"threads_started":2,"wal-append.queue_time_us":429451}
I20260709 13:01:56.424909  1661 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:56.425344  1661 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:56.426434  2310 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:01:56.426319  1661 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.427631  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.321924  3761 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.430730  3734 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:56.379133  3792 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:56.427692  1661 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:56.440284  1659 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
W20260709 13:01:56.441043  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44341.7us
I20260709 13:01:56.436184  1737 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:01:56.443281  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:56.431270  3761 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:56.462332  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 71006.7us
I20260709 13:01:56.463855  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:56.469833  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.431421  3605 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.478318  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:56.479907  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 84478.6us
I20260709 13:01:56.478350  1590 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:56.439107  3893 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:56.438941  3792 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:56.477319  3946 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:56.479982  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87071.6us
W20260709 13:01:56.461333  3789 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.605s	user 0.007s	sys 0.010s
I20260709 13:01:56.487068  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:56.488416  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.475464  3596 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:56.470212  3773 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:01:56.473278  3657 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.365s	user 0.016s	sys 0.000s
I20260709 13:01:56.499859  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.474618  3734 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:56.513213  1590 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:56.487279  3937 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:56.457306  3947 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
W20260709 13:01:56.519388  2280 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.155s (SENT)
W20260709 13:01:56.608395  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89038.6us
I20260709 13:01:56.612478  3889 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:56.491425  3792 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.544366  3944 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
W20260709 13:01:56.569062  3849 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.789s	user 0.014s	sys 0.000s
W20260709 13:01:56.554949  3405 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.066s	user 0.014s	sys 0.002s
I20260709 13:01:56.584823  1708 leader_election.cc:400] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 2. Current term is 3.
I20260709 13:01:56.545397  3963 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:56.569681  3945 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 13:01:56.519942  3893 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.556509  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:56.473667  3900 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.570459  3937 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:56.545919  3955 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:01:56.524987  3931 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.520211  3657 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:56.556337  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:56.564163  3914 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:56.501778  3596 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:56.621748  3795 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.602571  3734 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.511577  3886 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:56.606801  3909 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.619694  1661 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20260709 13:01:56.660132  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103727us
I20260709 13:01:56.661095  1661 pending_rounds.cc:85] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:01:56.661410  1661 pending_rounds.cc:107] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 13:01:56.661816  1661 raft_consensus.cc:2889] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:56.498759  3789 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.640460  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:56.616348  3907 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.661684  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:01:56.625454  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 144541us
I20260709 13:01:56.651445  3941 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.653124  3596 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:56.676319  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:56.639130  3824 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.649451  3963 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:56.663523  3899 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:56.651038  3585 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:56.682770  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 157312us
I20260709 13:01:56.642151  3405 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.636730  3849 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.669736  3795 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.750795  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:56.686794  3963 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.771018  3907 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:56.772442  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:56.773249  3907 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:56.773942  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:56.778146  1662 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:01:56.778602  1662 pending_rounds.cc:85] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:01:56.778842  1662 pending_rounds.cc:107] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:01:56.779189  1662 raft_consensus.cc:2889] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:56.762050  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.780009  2316 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:56.708745  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1939) took 2802 ms. Trace:
I20260709 13:01:56.782482  1736 rpcz_store.cc:276] 0709 13:01:53.905889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.418234 (+1512345us) service_pool.cc:224] Handling call
0709 13:01:55.426112 (+  7878us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:55.789417 (+363305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.708441 (+919024us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54000e3654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:56.708670 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":754191616}
W20260709 13:01:56.786480  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73329.6us
I20260709 13:01:56.711403  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:56.704603  3966 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.003s
I20260709 13:01:56.751101  3947 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:01:56.752712  3657 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:56.811892  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 286) took 2381 ms (client timeout 2155 ms). Trace:
I20260709 13:01:56.687278  3909 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:56.678651  3886 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:01:56.677668  3583 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:01:56.689000  3824 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:56.690965  3926 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.861s	user 0.014s	sys 0.000s
I20260709 13:01:56.868180  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:01:56.754778  3896 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.900s	user 0.011s	sys 0.005s
I20260709 13:01:56.695982  3931 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:56.807864  2314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 229) took 2711 ms (client timeout 1704 ms). Trace:
I20260709 13:01:56.697638  3900 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:57.007776  2314 rpcz_store.cc:269] 0709 13:01:54.073811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.074548 (+   737us) service_pool.cc:224] Handling call
0709 13:01:56.784671 (+2710123us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b54005c0154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:56.784799 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1198008576}
I20260709 13:01:56.753250  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:56.767648  3893 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:01:56.995338  1668 rpcz_store.cc:269] 0709 13:01:54.406103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.520975 (+114872us) service_pool.cc:224] Handling call
0709 13:01:56.777556 (+2256581us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:56.787552 (+  9996us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3022557440}
I20260709 13:01:56.992136  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:56.757217  3834 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:56.871735  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:56.987581  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 287) took 2534 ms. Trace:
W20260709 13:01:57.001819  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 229488us
I20260709 13:01:56.773700  3907 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:01:56.960726  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60786.7us
I20260709 13:01:57.003010  2313 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 13:01:56.973318  3948 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:01:56.993803  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:56.805105  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 126165us
W20260709 13:01:56.970184  3969 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.416s	user 0.000s	sys 0.003s
I20260709 13:01:57.032042  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.032773  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:56.808424  3904 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.210s
I20260709 13:01:56.767043  3956 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:56.971419  3973 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.403s	user 0.001s	sys 0.000s
I20260709 13:01:56.758621  3789 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.015443  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.016736  1734 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:56.976212  3917 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:56.695739  3937 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.027578  3907 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.020396  1665 rpcz_store.cc:276] 0709 13:01:54.453059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.919623 (+1466564us) service_pool.cc:224] Handling call
0709 13:01:55.920565 (+   942us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:56.271812 (+351247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.271856 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:56.302376 (+ 30520us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.663800 (+361424us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:56.663839 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.663919 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:56.987093 (+323174us) raft_consensus.cc:1685] finished
0709 13:01:56.987127 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:56.987381 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118645,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":9995,"thread_start_us":553837,"threads_started":2,"wal-append.queue_time_us":399313}
I20260709 13:01:57.022387  1737 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:57.022903  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:56.679941  3914 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.330s
I20260709 13:01:56.969641  2310 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:57.062793  3911 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 13:01:57.063680  3914 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:57.063895  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.064185  3914 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:57.064586  1735 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.006011  3910 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:56.681095  3941 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:56.807943  3600 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.210s
I20260709 13:01:57.031338  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.021032  3893 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.075941  3893 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:57.076231  3642 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:01:57.076954  3834 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.033692  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.033253  3596 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.043888  3917 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:57.073487  3911 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20260709 13:01:57.078615  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57741.7us
I20260709 13:01:57.078850  3911 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:57.079438  3911 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:57.079170  3917 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.079491  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:57.080416  3917 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:57.081758  3931 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:57.080895  3911 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.083343  3910 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.044086  3657 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 13:01:57.087954  3959 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:01:57.088404  3959 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:57.089318  3959 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.089956  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.090550  3959 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:01:57.011613  3926 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.015240  3896 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:57.091544  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:57.040021  3933 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.142s	user 0.004s	sys 0.000s
I20260709 13:01:57.040788  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:01:57.041321  3976 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 13:01:56.901396  3943 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.044890  1590 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:01:57.026332  3948 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:57.032683  3585 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.052057  3792 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.069403  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.054296  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.074090  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.082609  3931 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:01:57.013445  3583 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:57.110523  1591 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:57.113896  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.142997  1737 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:57.121397  1592 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:01:57.122073  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.083359  3911 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.141405  1734 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:57.097812  3834 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.146759  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:57.065210  3914 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.151443  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.169077  1734 raft_consensus.cc:2376] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2. Re-sending same reply.
I20260709 13:01:57.170188  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.170414  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.171192  1734 raft_consensus.cc:2376] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4. Re-sending same reply.
I20260709 13:01:57.171582  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:57.171932  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.172638  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.169701  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:57.147307  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.154441  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.115634  3789 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.112613  3795 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.172856  1592 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:57.160596  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.157008  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:57.180711  1591 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.157728  1733 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:57.161885  3889 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:57.167042  2314 raft_consensus.cc:1217] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Deduplicated request from leader. Original: 2.5->[3.6-3.6]   Dedup: 3.6->[]
I20260709 13:01:57.139437  3909 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:57.103852  3910 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.177680  1586 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:57.181001  2313 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 3: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.182308  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.183228  2314 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:01:57.079622  3596 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.114475  3642 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 13:01:57.193966  1737 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:01:57.194908  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:57.179114  1632 connection.cc:256] RPC call timeout handler was delayed by -2.04016113199998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2665s for a response
W20260709 13:01:57.197687  1632 connection.cc:256] RPC call timeout handler was delayed by -1.366582104000031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
I20260709 13:01:57.197712  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.200573  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.200016  1737 raft_consensus.cc:2376] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4. Re-sending same reply.
I20260709 13:01:57.201380  1590 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:57.203125  1584 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.204294  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.161221  3930 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.165660  3834 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.181565  3886 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:57.215915  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1406) took 1271 ms. Trace:
I20260709 13:01:57.218523  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.173537  3956 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:57.181996  3849 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.225903  1738 rpcz_store.cc:276] 0709 13:01:55.944969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.182861 (+237892us) service_pool.cc:224] Handling call
0709 13:01:57.215638 (+1032777us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:57.215825 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112474880}
I20260709 13:01:57.230346  1735 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:57.231827  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.232666  1735 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:57.238144  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.240136  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.240378  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.240684  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.241142  1737 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:57.241422  1738 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:57.240453  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.242173  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:57.202071  3909 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.246600  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.247577  1737 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:57.248456  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.249205  1737 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.253881  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.254630  1584 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.228587  3930 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.259972  3930 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.259940  1737 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:01:57.260708  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.261408  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:01:57.262179  3981 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
W20260709 13:01:57.264974  3970 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.702s	user 0.003s	sys 0.005s
I20260709 13:01:57.262351  1737 raft_consensus.cc:2410] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:01:57.200825  3910 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.269258  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.273432  1590 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.160128  3943 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.274219  1593 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:57.275705  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.186233  3789 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.286579  1738 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:01:57.290216  3800 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.292112  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.306113  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.220189  3596 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:57.201311  3889 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:57.334780  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.345641  1588 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:57.337836  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.286418  3914 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.346961  1735 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:01:57.333839  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.234261  3963 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:57.342729  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.352731  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:57.351050  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.356257  1592 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:01:57.351114  1737 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:01:57.360421  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 160893us
I20260709 13:01:57.282191  3585 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 13:01:57.338011  3930 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:57.371318  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 182502us
I20260709 13:01:57.333822  3909 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:57.372115  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:57.373232  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.374074  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.376403  1588 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:57.378413  1588 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 13:01:57.380460  1588 pending_rounds.cc:85] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:01:57.380738  1588 pending_rounds.cc:107] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:01:57.381083  1588 raft_consensus.cc:2889] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:57.350288  3963 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:57.227593  3948 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.390007  1737 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:01:57.394747  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.396073  1737 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:01:57.398157  3948 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.401899  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.402838  1590 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:01:57.403486  3991 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:01:57.405288  3789 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.381834  3886 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.417830  3989 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:01:57.344203  3910 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.347677  3596 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.292750  3583 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:57.418392  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52143.5us
I20260709 13:01:57.498842  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.543141  1735 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:01:57.491489  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69217.7us
I20260709 13:01:57.545665  1584 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:57.545264  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.548161  1593 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:01:57.533010  3990 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
I20260709 13:01:57.537101  3926 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:57.549638  3988 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 13:01:57.551285  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.539915  3889 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:01:57.554044  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:57.535277  3984 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.356s	user 0.000s	sys 0.001s
W20260709 13:01:57.533730  3987 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 13:01:57.558775  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:57.565526  1735 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:57.540879  3915 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.563985  3954 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:57.418661  3642 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:57.538611  3943 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.534169  3941 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.418764  3792 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.570660  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.391646  3963 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.576156  1584 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:01:57.546845  3989 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:57.421147  3930 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:57.574517  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 198610us
I20260709 13:01:57.421386  3896 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.576766  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.571637  3954 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:57.576329  3915 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:01:57.579808  1556 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.838s (SENT)
I20260709 13:01:57.568354  1737 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:57.598322  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:57.590368  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.590167  3954 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.591403  3989 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.594139  3930 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.598593  1737 raft_consensus.cc:606] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Received request to transfer leadership to TS ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:57.600069  1737 raft_consensus.cc:614] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:57.601847  1735 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:57.603636  1735 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:57.604387  3954 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:57.606361  3596 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.611358  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.616863  3896 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:01:57.617581  3893 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.003s
I20260709 13:01:57.617982  3896 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:01:57.618119  3893 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.621263  1736 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:01:57.624910  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45108.4us
I20260709 13:01:57.637576  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:57.629966  3824 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:57.649055  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.630854  3989 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:01:57.650594  1586 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.658453  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:57.665143  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:01:57.647547  3907 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
I20260709 13:01:57.669214  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.666256  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.666394  3945 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.650454  3893 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:57.668298  3642 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:57.674655  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.673566  3907 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.673041  3999 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:57.683298  1734 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:57.672834  3911 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:57.684010  3911 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:57.684950  3911 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.690202  3910 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.692556  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.693465  1590 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.700488  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:57.674957  3945 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.694837  3997 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:57.697711  3642 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:57.717526  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 275) took 5332 ms. Trace:
I20260709 13:01:57.720682  3997 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.724229  1662 rpcz_store.cc:276] 0709 13:01:52.385157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.568997 (+2183840us) service_pool.cc:224] Handling call
0709 13:01:56.777709 (+2208712us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:56.779723 (+  2014us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.717279 (+937556us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b5400014fd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:57.717432 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5019706752}
I20260709 13:01:57.726415  1667 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:01:57.733605  3642 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.736829  3405 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:57.741695  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:57.742408  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.731774  3997 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:01:57.744185  3959 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.744621  1740 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:57.744700  3959 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.745362  3997 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.746258  1740 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 13:01:57.746654  1740 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 1
I20260709 13:01:57.746877  1740 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:01:57.746979  3959 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.747124  1740 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:57.748639  1740 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:57.751327  1741 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 13:01:57.754930  1735 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:57.756455  1735 pending_rounds.cc:85] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 1
I20260709 13:01:57.756757  1735 pending_rounds.cc:107] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:01:57.756857  3963 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:57.762807  3805 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:01:57.763526  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.757040  1735 raft_consensus.cc:2889] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:01:57.766965  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.776018  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:57.793257  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:57.793725  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.800611  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.800892  1740 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:57.803709  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.801612  1663 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:57.805729  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:57.815323  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.816293  1665 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:57.817898  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.820806  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:57.826720  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.792035  3948 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.793228  3849 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:57.841960  3849 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:57.842427  3849 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:57.843315  3805 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.825572  3872 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:57.834532  3945 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:01:57.846093  3872 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:57.792150  3959 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:57.829820  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.831247  3948 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.834540  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.854682  1592 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:57.837007  1661 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:01:57.834774  1590 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:57.837806  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.840021  3963 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.794956  3896 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:01:57.801864  3726 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:01:57.848423  1662 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:57.793872  3997 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:01:57.859697  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.860850  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.862138  1586 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:57.864830  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.871335  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.877223  1664 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:57.871516  1740 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:01:57.886423  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.890743  1667 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:01:57.891322  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:57.891988  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97040us
I20260709 13:01:57.893532  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.896844  1665 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:57.892359  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:01:57.895459  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.899595  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.900118  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.900632  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.902281  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.900827  1660 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:57.881580  3896 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.873381  3959 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:57.912189  3959 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:57.902944  1662 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:01:57.926748  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.929900  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.934653  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.932602  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.947386  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:57.954556  3964 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.398s	user 0.007s	sys 0.015s
I20260709 13:01:57.963557  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:57.962466  3964 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.407s	user 0.008s	sys 0.016s
I20260709 13:01:57.964357  1662 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:01:57.970753  3948 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:57.972400  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.972710  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.972028  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.973752  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:57.974647  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:57.973858  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:57.986253  1668 raft_consensus.cc:2368] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:01:57.986555  1593 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:57.974000  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.014557  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.017073  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:58.017782  1737 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:58.019284  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.016685  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:58.026561  1734 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:58.034539  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.037899  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:58.038739  2313 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:58.041857  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.042766  1659 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:58.046815  1668 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:58.047051  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:58.048300  1659 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:01:58.049350  2312 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEARNER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260709 13:01:58.050777  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.041857  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:58.052217  1735 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:58.056553  1664 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2.
I20260709 13:01:58.057700  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:57.957113  3872 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.033118  3405 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:58.066092  3405 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:58.064662  1737 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:58.073390  3939 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:58.074024  3939 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.075440  3939 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:58.076632  3405 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:58.076838  3939 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:58.079509  1659 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.079878  2313 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:58.080780  3907 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:58.084980  3975 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.528s	user 0.000s	sys 0.003s
W20260709 13:01:58.085775  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1384) took 2388 ms (client timeout 1838 ms). Trace:
W20260709 13:01:58.087016  3923 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.366s	user 0.006s	sys 0.005s
I20260709 13:01:58.089092  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 286) took 1088 ms. Trace:
I20260709 13:01:58.068629  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:58.166249  1736 rpcz_store.cc:276] 0709 13:01:57.001079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.009979 (+  8900us) service_pool.cc:224] Handling call
0709 13:01:58.088828 (+1078849us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b5400350cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:58.089016 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1341139840}
W20260709 13:01:58.167106  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98600.6us
I20260709 13:01:58.168177  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.058609  1734 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:01:58.090111  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 277) took 4225 ms. Trace:
I20260709 13:01:58.099607  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 307) took 2222 ms. Trace:
W20260709 13:01:58.110062  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:58.093829  3980 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.411s	user 0.003s	sys 0.000s
I20260709 13:01:58.110750  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:58.095320  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:58.177158  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:58.178212  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67566.6us
W20260709 13:01:58.178541  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83372.2us
I20260709 13:01:58.179378  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:58.049871  3849 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.174240  1585 rpcz_store.cc:276] 0709 13:01:55.877271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.877530 (+   259us) service_pool.cc:224] Handling call
0709 13:01:55.915340 (+ 37810us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:56.050680 (+135340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:56.050729 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:56.051523 (+   794us) log.cc:844] Serialized 26 byte log entry
0709 13:01:56.302154 (+250631us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:01:56.302207 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:56.703769 (+401562us) spinlock_profiling.cc:243] Waited 47.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:58.099239 (+1395470us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:58.099294 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.099338 (+    44us) raft_consensus.cc:1685] finished
0709 13:01:58.099350 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.099518 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":409930368,"thread_start_us":250098,"threads_started":1,"wal-append.queue_time_us":421462,"wal-append.run_cpu_time_us":14088,"wal-append.run_wall_time_us":1422803}
I20260709 13:01:58.170707  1742 rpcz_store.cc:276] 0709 13:01:53.864651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:54.703336 (+838685us) service_pool.cc:224] Handling call
0709 13:01:54.730424 (+ 27088us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:54.930787 (+200363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:54.930838 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:54.977056 (+ 46218us) log.cc:844] Serialized 26 byte log entry
0709 13:01:55.439757 (+462701us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:55.439804 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:55.439883 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.086595 (+2646712us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b24000c3788. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2f09d 00007fcdedb6110e 00007fcdedb7888c 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221
0709 13:01:58.089686 (+  3091us) raft_consensus.cc:1685] finished
0709 13:01:58.089714 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.090037 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":158274,"raft.queue_time_us":313718,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":14828,"spinlock_wait_cycles":51081600,"thread_start_us":504595,"threads_started":2,"wal-append.queue_time_us":477879}
I20260709 13:01:58.094861  3792 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:01:58.119477  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:58.122316  3904 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:58.123692  1740 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:58.125733  1664 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:58.126107  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.196022  1661 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:01:58.126639  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.127748  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.197140  1660 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:58.069805  3943 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.060s
I20260709 13:01:58.069378  3805 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20260709 13:01:58.131878  4017 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:01:58.132596  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:58.134436  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.135304  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.206822  1667 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:01:58.136039  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.206236  1737 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:58.140499  3896 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:58.142444  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.208487  1659 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:58.104239  3945 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:58.155215  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:58.125138  3405 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:58.086539  1666 rpcz_store.cc:269] 0709 13:01:55.696874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:55.771856 (+ 74982us) service_pool.cc:224] Handling call
0709 13:01:58.082624 (+2310768us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:58.082829 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2805099776}
I20260709 13:01:58.184126  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:58.222466  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67728.9us
I20260709 13:01:58.185765  1734 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.223347  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:58.224131  1585 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:58.181258  3944 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:58.224447  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:58.224778  1734 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:58.186075  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:58.226023  1736 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:58.187896  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.196605  1662 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:58.209172  1737 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.215876  1742 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.183012  3824 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:58.186995  1739 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:01:58.183219  3907 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:58.225219  1740 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:01:58.189116  3815 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:01:58.188726  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:58.184998  3963 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:58.221088  1735 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:01:58.238387  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52664.9us
I20260709 13:01:58.247804  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:58.239393  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.245333  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.242496  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.244515  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.193410  3904 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:58.211956  3914 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:01:58.218523  3945 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:58.268491  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:58.270912  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.246153  3959 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:58.236665  4012 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:58.251331  3947 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:01:58.238315  3907 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.281340  1735 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:58.251185  3963 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.282915  1585 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:58.264187  3886 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:01:58.294480  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.267056  3944 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:58.265853  3815 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.246382  3964 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
W20260709 13:01:58.295452  3920 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.021s	user 0.018s	sys 0.029s
I20260709 13:01:58.268029  3899 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:01:58.284282  4012 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.263634  3896 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 13:01:58.327488  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.289485  3914 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:58.265671  3600 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:01:58.269825  3916 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 13:01:58.290328  3963 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:58.279953  3915 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 13:01:58.339329  1663 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:58.341574  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1953) took 1145 ms. Trace:
I20260709 13:01:58.349920  1741 rpcz_store.cc:276] 0709 13:01:57.196254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.752225 (+555971us) service_pool.cc:224] Handling call
0709 13:01:57.752587 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.753189 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.753223 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.813917 (+ 60694us) log.cc:844] Serialized 26 byte log entry
0709 13:01:57.937950 (+124033us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:57.938018 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:57.938137 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.341189 (+403052us) raft_consensus.cc:1685] finished
0709 13:01:58.341226 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.341414 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":60760,"threads_started":1,"wal-append.queue_time_us":125098}
I20260709 13:01:58.301333  3997 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.300063  3886 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:01:58.309856  3920 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.041s	user 0.018s	sys 0.030s
I20260709 13:01:58.289789  3907 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:58.316884  3911 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.009s	sys 0.021s
I20260709 13:01:58.353212  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.358268  1659 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:01:58.345317  3914 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.342435  4012 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:01:58.361310  3911 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.009s	sys 0.021s
I20260709 13:01:58.375321  1741 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:58.360249  3886 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:58.360142  3936 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.400907  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:58.426575  3849 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:58.427281  1734 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:58.401753  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 245) took 1913 ms. Trace:
W20260709 13:01:58.406612  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:58.429970  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.429387  2310 rpcz_store.cc:276] 0709 13:01:56.488033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.491443 (+  3410us) service_pool.cc:224] Handling call
0709 13:01:56.804741 (+313298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.062663 (+257922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.401432 (+1338769us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54005c0150. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:58.401668 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522465152}
W20260709 13:01:58.430791  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.363655  3405 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:58.446746  1589 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:58.442113  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 191103us
I20260709 13:01:58.421612  3834 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:58.422639  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:58.374980  3914 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.429077  2318 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:01:58.436569  1587 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:01:58.438081  1588 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:01:58.452945  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:58.440683  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1416) took 1395 ms. Trace:
I20260709 13:01:58.457333  1591 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.457204  1738 rpcz_store.cc:276] 0709 13:01:57.045544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.660520 (+614976us) service_pool.cc:224] Handling call
0709 13:01:57.686120 (+ 25600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:57.686394 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:01:57.702415 (+ 16021us) log.cc:844] Serialized 14 byte log entry
0709 13:01:58.110948 (+408533us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x7b180043c3c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:01:58.325807 (+214859us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:58.325866 (+    59us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:01:58.325917 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.326945 (+  1028us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:58.440436 (+113491us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400434c50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:58.440599 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15825,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":620,"spinlock_wait_cycles":60961792,"thread_start_us":214628,"threads_started":1}
I20260709 13:01:58.418793  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.459501  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:01:58.460671  1735 raft_consensus.cc:606] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:01:58.461164  1735 raft_consensus.cc:614] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:01:58.462141  1741 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:58.318480  3904 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:58.477365  1593 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260709 13:01:58.485121  1742 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 13:01:58.485839  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63305us
I20260709 13:01:58.473465  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:58.483991  1584 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:01:58.473018  1591 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 13:01:58.473780  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:58.474629  1739 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:01:58.325677  3945 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.490043  1591 pending_rounds.cc:85] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 2
I20260709 13:01:58.491489  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:58.493516  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:58.491492  1591 pending_rounds.cc:107] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 13:01:58.495456  1591 raft_consensus.cc:2889] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:01:58.476256  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:01:58.498045  1589 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.498961  1585 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:58.503966  1590 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.441785  3824 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.362443  3997 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.452373  3911 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:01:58.527624  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:58.528705  1587 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:58.500367  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:58.520902  1660 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:58.536772  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:58.526079  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:01:58.539696  1588 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:01:58.475150  3959 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:58.549460  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:58.551753  1592 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:01:58.563210  1659 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:01:58.501307  3903 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:58.568708  1590 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:01:58.574364  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:58.535709  3948 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:58.536361  3904 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.595330  1588 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:01:58.584550  1587 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:58.453671  3936 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.586452  1592 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:01:58.554072  3920 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:58.548450  3945 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.592007  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.456357  3815 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:58.592680  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 404847us
I20260709 13:01:58.502624  3952 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260709 13:01:58.572094  3964 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:58.572001  4011 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:01:58.620059  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.568360  3903 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.592007  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:58.555437  3911 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.585536  3642 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:58.602782  3920 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.605547  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:01:58.629053  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 445019us
I20260709 13:01:58.656574  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:58.665406  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.632099  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:58.655107  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.620045  4011 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.626245  3903 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.672550  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:58.673926  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.675290  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:58.677434  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:58.678925  4034 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:58.680310  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.679476  3405 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:58.679657  4034 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.683790  3849 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:58.684397  3849 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.683024  3834 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:58.647835  3642 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:58.656306  3920 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:58.687403  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.688437  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.690832  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:58.693367  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:01:58.698483  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.676112  4011 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:58.714035  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.685318  4034 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:58.717368  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.727924  1668 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:01:58.720165  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:58.687836  3834 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:58.692363  3872 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:58.625735  3824 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:58.667650  3959 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:58.736092  4021 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260709 13:01:58.736966  4024 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
W20260709 13:01:58.742061  4026 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 13:01:58.742291  4006 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.031s	user 0.007s	sys 0.000s
I20260709 13:01:58.747416  3915 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:01:58.749099  3834 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.716786  3944 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:58.749912  4033 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:58.733237  3792 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:58.751614  4043 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 13:01:58.752698  4025 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 13:01:58.713201  4022 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:01:58.710565  3405 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.707471  4032 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:58.770190  1589 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:58.774709  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.762916  3872 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:58.779794  1741 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:01:58.727701  3943 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:01:58.787478  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:58.789736  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 246959us
I20260709 13:01:58.779273  3942 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.751646  3964 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:58.685428  3849 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.764843  3805 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 13:01:58.807559  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.768615  4034 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 13:01:58.799119  3899 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:58.821812  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:01:58.791142  3915 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:58.804425  3997 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:58.827507  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 211708us
I20260709 13:01:58.803179  4033 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:58.834781  1663 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:01:58.836675  3405 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.822058  3944 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 13:01:58.838426  3964 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:58.839865  4020 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.546s	user 0.004s	sys 0.000s
I20260709 13:01:58.833002  3943 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:58.841018  4039 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:01:58.841889  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:58.811668  4042 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.815189  4037 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:01:58.817071  4043 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:01:58.822166  3872 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:58.816118  4032 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:01:58.844004  3939 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.767s	user 0.017s	sys 0.001s
W20260709 13:01:58.888795  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 315843us
I20260709 13:01:58.904850  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:58.901126  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 269134us
I20260709 13:01:58.883188  3936 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:01:58.928861  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 234008us
W20260709 13:01:58.933063  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 202138us
I20260709 13:01:58.833212  3942 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:58.904525  3963 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.542s	user 0.009s	sys 0.003s
I20260709 13:01:58.945923  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:01:58.949361  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:58.905941  3911 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:01:58.946704  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.821337  3642 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:58.951397  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.895566  4044 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:58.957589  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:01:58.960306  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:58.882632  4012 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.934900  4037 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:58.968703  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 146989us
I20260709 13:01:58.991501  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:58.944036  4032 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:58.953023  3997 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:01:58.961390  3947 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.013s	sys 0.009s
I20260709 13:01:59.036337  2316 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:01:59.000501  4044 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 13:01:58.973296  3792 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:01:58.960376  4029 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:01:58.999001  3951 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:58.952688  3903 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:58.973902  4037 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:01:59.123077  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 246) took 1332 ms. Trace:
I20260709 13:01:58.961808  3963 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:58.959848  4045 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.145s
I20260709 13:01:59.123826  2314 rpcz_store.cc:276] 0709 13:01:57.790907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:57.791049 (+   142us) service_pool.cc:224] Handling call
0709 13:01:57.805335 (+ 14286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:57.926747 (+121412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:57.926785 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:57.957367 (+ 30582us) log.cc:844] Serialized 26 byte log entry
0709 13:01:58.152819 (+195452us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:58.152878 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.152967 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:01:58.826798 (+673831us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:01:59.113428 (+286630us) raft_consensus.cc:1685] finished
0709 13:01:59.113454 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:59.122986 (+  9532us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38914,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"spinlock_wait_cycles":10614016,"thread_start_us":256837,"threads_started":2,"wal-append.queue_time_us":264489}
I20260709 13:01:58.944340  4042 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:59.141464  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 200563us
W20260709 13:01:59.123029  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 165580us
I20260709 13:01:59.143134  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:01:58.969601  4043 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:59.040256  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:59.095468  3997 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:59.172701  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 220499us
I20260709 13:01:58.960705  4050 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:01:59.130821  3914 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.637s	user 0.008s	sys 0.005s
I20260709 13:01:59.187103  1665 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:01:59.002192  4040 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.135s
I20260709 13:01:59.091074  3904 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
W20260709 13:01:59.037268  3947 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.014s	sys 0.009s
I20260709 13:01:59.184882  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.200856  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:59.218719  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 241902us
I20260709 13:01:59.175117  4058 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:59.112684  4054 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260709 13:01:59.185562  4029 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 13:01:59.235350  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92503us
I20260709 13:01:59.128808  3899 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260709 13:01:59.190647  4050 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:01:59.041471  3939 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.182s
I20260709 13:01:59.242345  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:01:59.243723  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49494.2us
I20260709 13:01:59.265756  1664 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:01:59.275490  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:59.290132  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:59.290692  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.001659  4033 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:58.973034  3915 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:59.247308  3948 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.647s	user 0.011s	sys 0.002s
I20260709 13:01:59.126063  3951 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:59.224252  3947 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:01:59.199108  3914 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.240151  4029 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.338842  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:01:59.289739  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 244335us
I20260709 13:01:59.349532  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.359158  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.360286  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.350292  1589 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:01:59.395673  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.176329  3943 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.218375  4062 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.145s
I20260709 13:01:59.403218  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.411715  1664 raft_consensus.cc:2376] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 2. Re-sending same reply.
I20260709 13:01:59.400861  4072 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:01:59.357772  3907 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.977s	user 0.015s	sys 0.009s
I20260709 13:01:59.290167  4040 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260709 13:01:59.420641  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:59.423393  1738 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:01:59.351743  3948 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:01:59.356407  4050 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:01:59.430578  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:01:59.443584  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.386413  4029 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:59.475286  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 298014us
I20260709 13:01:59.480309  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:01:59.255892  3792 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.504208  1738 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 13:01:59.512223  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.014s	sys 0.006s
I20260709 13:01:59.422821  4072 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:01:59.436885  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 197719us
I20260709 13:01:59.425577  3915 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:01:59.512869  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.014s	sys 0.006s
I20260709 13:01:59.517491  1733 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:01:59.257225  4058 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:01:59.520175  1663 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.629s	user 0.009s	sys 0.004s
I20260709 13:01:59.535022  1663 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:01:59.536324  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.537688  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:59.538820  4058 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:59.538985  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:01:59.539536  4058 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:01:59.539893  4058 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.540254  4058 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:01:59.318370  3997 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.414821  3959 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
W20260709 13:01:59.554904  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 159353us
I20260709 13:01:59.449987  4033 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:01:59.565165  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.497983  4056 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:01:59.593634  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:01:59.589627  1663 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:01:59.547626  4071 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:01:59.505195  3942 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.556561  4012 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.433456  3907 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:59.553293  4072 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:01:59.617969  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:01:59.576233  3939 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
W20260709 13:01:59.624322  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95356us
I20260709 13:01:59.593334  4061 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:01:59.621145  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:59.446861  4063 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.167s
I20260709 13:01:59.592222  4069 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:01:59.615640  3997 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:01:59.642710  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.646677  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:01:59.647557  2310 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:01:59.638028  3964 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.634872  4070 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 13:01:59.654242  3907 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:01:59.615325  4071 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:01:59.656639  3886 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.204s	user 0.013s	sys 0.001s
W20260709 13:01:59.657421  3920 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.943s	user 0.006s	sys 0.009s
I20260709 13:01:59.659616  1736 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:01:59.663959  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:01:59.617203  3963 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.669844  4088 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.620711  4077 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.623971  3959 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:01:59.596971  3936 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 13:01:59.632822  3989 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.596954  3947 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.658222  3920 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:59.624665  4044 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.081s
I20260709 13:01:59.763592  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 603) took 1194 ms. Trace:
W20260709 13:01:59.948700  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 361772us
I20260709 13:01:59.742398  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 1969) took 1506 ms. Trace:
W20260709 13:01:59.714896  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:01:59.952304  1705 leader_election.cc:336] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.519s (SENT)
I20260709 13:01:59.636350  4061 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:01:59.688218  1663 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
W20260709 13:01:59.947973  2282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.003s	sys 0.006s
I20260709 13:01:59.729756  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:01:59.640782  3849 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:01:59.740123  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 602) took 1190 ms. Trace:
I20260709 13:01:59.956002  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:00.001905  1662 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.029s	user 0.006s	sys 0.005s
I20260709 13:02:00.008059  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:01:59.993443  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:00.032310  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80054.2us
W20260709 13:02:00.002220  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.008s	sys 0.002s
W20260709 13:02:00.034265  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.009s	sys 0.003s
I20260709 13:01:59.950471  2316 rpcz_store.cc:276] 0709 13:01:58.569735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.569909 (+   174us) service_pool.cc:224] Handling call
0709 13:01:58.586284 (+ 16375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:59.037397 (+451113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.763227 (+725830us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:59.763500 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034587904}
W20260709 13:02:00.034902  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 554703us
I20260709 13:01:59.953353  1736 rpcz_store.cc:276] 0709 13:01:58.236213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.436158 (+199945us) service_pool.cc:224] Handling call
0709 13:01:59.742089 (+1305931us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400404a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:01:59.742293 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2764690816}
I20260709 13:02:00.032904  1737 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:01:59.993918  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:00.032089  1662 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.059s	user 0.006s	sys 0.005s
I20260709 13:01:59.634591  4083 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:01:59.704458  4011 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.983s	user 0.013s	sys 0.000s
I20260709 13:01:59.671684  3907 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.671824  3939 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:01:59.733316  4053 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 13:01:59.767419  4037 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.592s	user 0.007s	sys 0.003s
I20260709 13:01:59.667841  3886 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:01:59.672230  3600 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.794s	user 0.022s	sys 0.009s
W20260709 13:01:59.716665  4059 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
W20260709 13:01:59.903914  3834 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.098s	user 0.009s	sys 0.003s
W20260709 13:02:00.083045  1705 leader_election.cc:336] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.789s (SENT)
I20260709 13:02:00.083562  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:00.105901  3899 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:01:59.770651  4029 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.121313  3899 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.123514  3899 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.123780  3834 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.087667  4080 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:00.086607  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.769873  4072 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.131569  4080 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:00.132453  3834 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.980262  3989 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.134262  4080 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:00.135027  4080 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:00.086524  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 304) took 1254 ms. Trace:
I20260709 13:01:59.731355  4068 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:00.033224  2314 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Advancing to term 6
W20260709 13:02:00.121261  3915 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.002s
I20260709 13:01:59.667258  4040 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 13:01:59.773609  4065 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.455s	user 0.000s	sys 0.002s
I20260709 13:01:59.955324  3959 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:00.036916  3945 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.429s	user 0.010s	sys 0.006s
I20260709 13:01:59.718458  4084 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.189949  4084 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.194370  3964 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:00.196341  3964 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:00.197031  3964 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:01:59.672586  4054 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:01:59.973325  4032 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.934s	user 0.007s	sys 0.006s
I20260709 13:02:00.200980  4032 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.100216  1736 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:44038
I20260709 13:02:00.205295  1736 raft_consensus.cc:606] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:00.205683  1736 raft_consensus.cc:614] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:00.207283   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:02:00.208520  3939 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:00.033562  2282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.902s	user 0.003s	sys 0.008s
I20260709 13:02:00.213536  4090 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:00.214326  4090 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.217068  4090 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:01:59.887329  3914 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.037380  2312 rpcz_store.cc:276] 0709 13:01:58.550192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.550311 (+   119us) service_pool.cc:224] Handling call
0709 13:01:58.550647 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:58.665633 (+114986us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:58.665675 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:01:58.665732 (+    57us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:01:58.665773 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.739812 (+1074039us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:01:59.739882 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:01:59.740047 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263368448}
I20260709 13:01:59.968431  3815 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:00.098115  4033 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.097426  3726 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:02:00.111107  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1404) took 1973 ms. Trace:
I20260709 13:01:59.973901  4043 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:00.239390  1667 rpcz_store.cc:276] 0709 13:01:58.137782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.230895 (+ 93113us) service_pool.cc:224] Handling call
0709 13:01:58.257079 (+ 26184us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:58.556227 (+299148us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:58.556259 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:58.573742 (+ 17483us) log.cc:844] Serialized 26 byte log entry
0709 13:01:58.961150 (+387408us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:01:58.961193 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:58.961274 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:00.110834 (+1149560us) raft_consensus.cc:1685] finished
0709 13:02:00.110856 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.111037 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38614,"raft.queue_time_us":117171,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4803,"spinlock_wait_cycles":35072,"thread_start_us":560841,"threads_started":2,"wal-append.queue_time_us":882614}
I20260709 13:02:00.247164  4043 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:00.248198  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:01:59.763542  3951 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:01:59.954231  4056 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:00.113909  4037 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.644737  4050 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:59.642772  4069 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.969867  4089 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:01:59.670512  4088 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:01:59.741242  4067 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.446s
I20260709 13:01:59.773142  4045 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.333s
I20260709 13:02:00.119246  3886 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.119338  4034 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:00.114310  3600 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.239s	user 0.036s	sys 0.031s
I20260709 13:02:00.120409  1666 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:00.124706  4029 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.105669  3963 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 13:02:00.290553  3903 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.064721  1664 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:00.071214  4061 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:00.087157  1735 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:00.135275  3989 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.089196  4082 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:01:59.983243  4077 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:00.125440  4073 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.512s	user 0.000s	sys 0.002s
I20260709 13:02:00.163659  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:00.156802  1662 rpcz_store.cc:276] 0709 13:01:58.832326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.832444 (+   118us) service_pool.cc:224] Handling call
0709 13:01:58.832809 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:58.833067 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:58.836506 (+  3439us) log.cc:844] Serialized 14 byte log entry
0709 13:02:00.086035 (+1249529us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.086113 (+    78us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:00.086181 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.086275 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.086449 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28136,"prepare.queue_time_us":3349,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":185716,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":20711,"thread_start_us":1227961,"threads_started":2,"wal-append.queue_time_us":657763}
W20260709 13:02:00.120558  4058 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 13:02:00.054373  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:00.172298  4040 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:00.345371  3914 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:00.346360  3914 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:00.346899  4032 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.347982  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:00.107998  3907 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.351603  3907 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:00.179945  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.018s	sys 0.043s
W20260709 13:02:00.353884  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.689s	user 0.019s	sys 0.044s
W20260709 13:02:00.354785  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 690944us
I20260709 13:02:00.356690  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:00.181054  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77484.3us
I20260709 13:02:00.358703  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:00.209329   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
W20260709 13:02:00.218319  3872 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.269s	user 0.014s	sys 0.000s
I20260709 13:02:00.218752  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:00.221450  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 305) took 1387 ms. Trace:
I20260709 13:02:00.221331  3964 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:00.373023  3964 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:00.223584  4084 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.384586  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:00.385404  1738 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:00.224587  4072 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:00.225739  1556 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.204s (SENT) [suppressed 39 similar messages]
I20260709 13:02:00.387449  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:00.388409  2316 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:00.390837  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.391531  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.391027  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:00.367151  3872 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:01:59.990267  3900 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.649s	user 0.005s	sys 0.011s
I20260709 13:02:00.225984  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 306) took 1348 ms. Trace:
W20260709 13:02:00.397405  3900 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.076s	user 0.009s	sys 0.013s
W20260709 13:02:00.398324  3383 raft_consensus.cc:670] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } [suppressed 3 similar messages]
W20260709 13:02:00.227113  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.399705  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.400205  3642 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:00.363910  3963 consensus_queue.cc:1243] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:00.401667  3886 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.402213  3886 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.403760  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:00.404399  3886 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.246340  4054 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:00.404724  1739 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:02:00.404580  4071 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:01:59.973592  4081 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.443s
I20260709 13:02:00.405663  4054 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.256577  4033 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:00.407375  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:00.408183  2318 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:00.366688  4029 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.409010  4033 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:00.261925  4042 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.263520  3920 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.104298  4011 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:00.413585  1733 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:00.415385  4011 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:00.290297  2314 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEARNER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:00.415980  3872 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.305006  4034 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:00.417285  4034 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.418566  4034 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:00.420322  3952 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:00.163080  4068 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:00.337529  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.339679  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 176156us
I20260709 13:02:00.424191  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:00.425920  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:00.340468  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 286175us
W20260709 13:02:00.332779  4064 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.087s	user 0.001s	sys 0.003s
I20260709 13:02:00.366503  1661 rpcz_store.cc:276] 0709 13:01:58.834067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.834195 (+   128us) service_pool.cc:224] Handling call
0709 13:01:58.873165 (+ 38970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:58.976218 (+103053us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:58.976256 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:58.976830 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 13:01:59.312676 (+335846us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:01:59.312743 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.312858 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:00.221163 (+908305us) raft_consensus.cc:1685] finished
0709 13:02:00.221194 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.221372 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21808,"raft.queue_time_us":75116,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":11580,"thread_start_us":295687,"threads_started":2,"wal-append.queue_time_us":1015794}
I20260709 13:02:00.368155  4082 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:00.433588  4082 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:00.433751  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:00.434217  4082 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:00.371030  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:00.099246  4083 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:00.377198  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.436802  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1429) took 1216 ms. Trace:
I20260709 13:02:00.378132  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:00.239249  3815 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:00.258533  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.363189  4058 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:00.442714  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72930.6us
I20260709 13:02:00.397468  1668 rpcz_store.cc:276] 0709 13:01:58.877243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.877431 (+   188us) service_pool.cc:224] Handling call
0709 13:01:58.877948 (+   517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:58.948037 (+ 70089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:58.948076 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:58.961560 (+ 13484us) log.cc:844] Serialized 26 byte log entry
0709 13:01:59.446277 (+484717us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:01:59.510481 (+ 64204us) log.cc:844] Serialized 14 byte log entry
0709 13:01:59.661948 (+151467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:01:59.662044 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:00.225761 (+563717us) raft_consensus.cc:1685] finished
0709 13:02:00.225788 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.225924 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":228897,"prepare.queue_time_us":64389,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":310586,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":11754,"thread_start_us":356243,"threads_started":1,"wal-append.queue_time_us":1184977}
W20260709 13:02:00.292392  1705 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.894s (SENT)
I20260709 13:02:00.443774  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:00.429073  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:00.432868  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:00.445029  1587 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 1.
I20260709 13:02:00.437287  1664 rpcz_store.cc:276] 0709 13:01:59.220294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.412673 (+192379us) service_pool.cc:224] Handling call
0709 13:02:00.427588 (+1014915us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b540013e2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:00.436731 (+  9143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1394382208}
I20260709 13:02:00.444125  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.445794  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:00.447495  1668 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:00.451754  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 303) took 1624 ms. Trace:
I20260709 13:02:00.460832  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:00.461859  1661 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:02:00.463944  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:00.452771  1659 rpcz_store.cc:276] 0709 13:01:58.827090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.841602 (+ 14512us) service_pool.cc:224] Handling call
0709 13:01:58.841950 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:01:58.842186 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:01:58.886795 (+ 44609us) log.cc:844] Serialized 14 byte log entry
0709 13:02:00.451010 (+1564215us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b180007bd60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:00.451176 (+   166us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:00.451259 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:00.451333 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.451447 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:00.451645 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":163990,"prepare.queue_time_us":48244,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":315566,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13042,"spinlock_wait_cycles":5857152,"thread_start_us":513859,"threads_started":2,"wal-append.queue_time_us":316351}
I20260709 13:02:00.467963  1735 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:02:00.374071  3951 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:00.301949  4089 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:00.455888  3405 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.018s	sys 0.042s
I20260709 13:02:00.544626  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:00.461182  3943 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.002s	sys 0.010s
W20260709 13:02:00.467690  4073 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
W20260709 13:02:00.506901  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61187.3us
I20260709 13:02:00.548326  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:00.548655  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.548332  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.548972  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:00.506486  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:00.542927  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 317205us
W20260709 13:02:00.551222  1632 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.011s	sys 0.020s
I20260709 13:02:00.504662  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:00.507454  1586 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:00.326722  3959 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:00.561061  1659 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:00.562000  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:00.563006  1632 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.011s	sys 0.020s
W20260709 13:02:00.569720  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 679473us
W20260709 13:02:00.558554  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 266219us
I20260709 13:02:00.567385  1667 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:00.573302  1735 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:02:00.574146  1740 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 13:02:00.562515  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56122.8us
I20260709 13:02:00.501441  3952 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:00.558012  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:00.555686  3943 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s	user 0.008s	sys 0.016s
I20260709 13:02:00.544844  4080 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:00.505980  4006 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.685s	user 0.010s	sys 0.000s
I20260709 13:02:00.557281  1738 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:00.459779  3914 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.577188  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.411409  4077 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.433691  4068 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:00.438794  4084 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.117s
W20260709 13:02:00.555771  3405 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.027s	sys 0.046s
I20260709 13:02:00.564330  1588 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:00.576460  4011 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:02:00.568595  4071 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:00.585789  4042 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.594223  3959 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:00.658263  4042 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.651929  4089 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.660043  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:00.409479  4029 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:00.579278  1556 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.500s (SENT)
I20260709 13:02:00.502298  3951 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:00.459811  3900 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:00.548589  4033 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:00.465916  4081 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.112s
I20260709 13:02:00.598837  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:00.420153  4088 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:00.466744  3989 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20260709 13:02:00.559280  4105 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 13:02:00.466893  3886 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.118s
I20260709 13:02:00.675853  1661 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:00.602120  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:00.601169  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:00.661469  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:00.677174  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:00.692589  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 113359us
I20260709 13:02:00.620818  4100 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 13:02:00.588150  4102 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:00.578660  4090 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:00.681280  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:00.682912  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:00.686151  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:00.701792  1588 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:00.660856  4042 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:00.697474  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98757.1us
I20260709 13:02:00.592432  3952 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:00.702214  1589 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:00.715950  1590 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:00.703428  1588 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:00.456722  4056 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.680261  3951 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:00.698474  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:00.703464  1556 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.689s (SENT)
I20260709 13:02:00.721909  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:00.711974  1740 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:00.606174  4080 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.716382  1589 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:00.648202  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1405) took 2509 ms. Trace:
I20260709 13:02:00.662328  4011 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:00.717103  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 277941us
I20260709 13:02:00.726179  1665 rpcz_store.cc:276] 0709 13:01:58.138508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.249616 (+111108us) service_pool.cc:224] Handling call
0709 13:01:58.250072 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:59.188545 (+938473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.647755 (+1459210us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b5400012a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:00.648106 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2154455936}
I20260709 13:02:00.735756  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:00.714391  3849 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.005s	sys 0.020s
I20260709 13:02:00.697223  4084 consensus_queue.cc:1243] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:00.696642  3899 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.739926  1584 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:00.771096  1738 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:00.700775  4037 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:00.748771  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.769608  4042 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:00.728819  3947 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.032s	sys 0.044s
I20260709 13:02:00.701221  4033 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:00.788648  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.687559  3914 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:00.791770  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.796566  1735 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260709 13:02:00.751714  3849 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.006s	sys 0.020s
I20260709 13:02:00.715757  3963 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:00.720696  3952 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:00.718676  4069 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:00.791739  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:00.804919  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:00.819830  4113 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:02:00.819682  3896 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:00.724213  4068 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:00.799243  4096 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.229s
I20260709 13:02:01.007139  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:00.899042  4037 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:00.881939  4083 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:01.017411  4037 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.825668  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:00.959511  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:00.831108  3963 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:01.008661  3943 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 13:02:01.020327  3963 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:00.821262  3600 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:01.022372  1738 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:01.022121  3600 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:01.006608  1556 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.731s (SENT)
I20260709 13:02:01.023646  4045 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:01.024387  1556 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.732s (SENT)
I20260709 13:02:01.024156  4045 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.024521  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:00.769615  3405 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:01.025200  1737 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:01.025313  1556 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.738s (SENT)
I20260709 13:02:01.025974  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:01.026423  4045 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:00.801707  3899 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:00.821975  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.028247  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.027171  4045 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.029772  4045 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.030421  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 238838us
I20260709 13:02:01.031730  4045 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.734483  4011 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:01.018033  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.033571  3896 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.035532  1737 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:00.826905  4029 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:00.796165  3947 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.032s	sys 0.044s
I20260709 13:02:01.038057  3896 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.820273  4035 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:00.768710  4081 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.271s
I20260709 13:02:01.039515  4035 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:01.012315  1705 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.980s (SENT)
I20260709 13:02:00.831410  3849 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:01.042289  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:01.043992  1705 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.063s (SENT)
I20260709 13:02:01.044601  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:01.043798  3849 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.015662  4058 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:00.805017  4042 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.054445  1556 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.787s (SENT)
W20260709 13:02:01.056061  1556 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.788s (SENT)
I20260709 13:02:01.048851  1665 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:01.004352  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:01.061131  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.062229  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:01.064316  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.091261  1740 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:01.066455  1667 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:01.068030  4043 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.820s	user 0.014s	sys 0.000s
I20260709 13:02:01.068152  4040 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.024573  3600 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:01.070788  3900 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:01.048070  4029 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:01.029214  3899 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:01.037385  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.131322  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 1.
I20260709 13:02:01.132356  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.133033  1737 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:01.081869  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 256341us
I20260709 13:02:01.136168  1737 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:01.066337  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1402) took 2929 ms. Trace:
I20260709 13:02:01.137727  1666 rpcz_store.cc:276] 0709 13:01:58.137109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.229925 (+ 92816us) service_pool.cc:224] Handling call
0709 13:01:58.257121 (+ 27196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.309723 (+2052602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.066035 (+756312us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540013d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:01.066273 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4284925184}
I20260709 13:02:01.140584  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:01.141018  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 85008.3us
I20260709 13:02:01.100106  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.156034  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.157215  2314 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:01.158039  1740 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:01.020453  4037 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:01.075299  4042 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:01.029011  1738 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:01.184756  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.080071  1661 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
W20260709 13:02:00.599079  3903 consensus_queue.cc:1175] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "316f3bf18ed94e32b238130a6bfedf1a" responder_term: 5 status { last_received { term: 4 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:00.710165  4102 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:01.194052  1556 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.821s (SENT)
W20260709 13:02:01.145202  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101264us
I20260709 13:02:01.147804  4011 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.150102  3914 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:01.150238  4033 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.107916  3405 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:01.160632  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.105922  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.082571  4035 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:01.179502  4062 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.163640  4100 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:01.180493  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.166319  4031 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:01.164142  3989 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.261530  3900 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:01.262254  3900 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:01.262878  3900 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:01.262377  3989 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.167503  4098 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.113s
I20260709 13:02:01.074730  4107 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20260709 13:02:01.265962  4070 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:01.266435  4070 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:01.266711  4031 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:01.267354  4070 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.164773  3600 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.172909  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.275637  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.276610  1735 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:01.183331  1668 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.735s	user 0.013s	sys 0.000s
I20260709 13:02:01.277753  1668 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:01.278908  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.279855  1668 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:01.180820  3997 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.522s	user 0.013s	sys 0.001s
I20260709 13:02:01.186498  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.119926  4043 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.190497  1666 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260709 13:02:01.254637  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74263.7us
I20260709 13:02:01.287608  1666 pending_rounds.cc:85] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 1
I20260709 13:02:01.287901  1666 pending_rounds.cc:107] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:02:01.288075  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.288331  2316 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 1. Current term is 2. Ops: []
I20260709 13:02:01.192885  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:01.272321  3642 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.857s	user 0.014s	sys 0.003s
I20260709 13:02:01.305759  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.307830  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.310720  4102 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:01.256585  4034 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.838s	user 0.011s	sys 0.003s
I20260709 13:02:01.288142  1666 raft_consensus.cc:2889] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:01.332885  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1443) took 1043 ms. Trace:
I20260709 13:02:01.333385  1666 rpcz_store.cc:276] 0709 13:02:00.290005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.152967 (+862962us) service_pool.cc:224] Handling call
0709 13:02:01.190006 (+ 37039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.332527 (+142521us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.332807 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.335045  1661 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 13:02:01.289942  4061 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.944s	user 0.008s	sys 0.003s
I20260709 13:02:01.290688  4055 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:01.296643  4035 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:01.289178  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.281046  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:01.268785  4070 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:01.387923  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98818us
I20260709 13:02:01.389014  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.268096  4031 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.394034  4034 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:01.394634  4034 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.395411  4034 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:01.395840  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.398236  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.419829  4077 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:01.420536  4077 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:01.422382  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1439) took 1447 ms. Trace:
I20260709 13:02:01.422933  1665 rpcz_store.cc:276] 0709 13:01:59.975065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.035089 (+1060024us) service_pool.cc:224] Handling call
0709 13:02:01.035495 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.058806 (+ 23311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.421976 (+363170us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b54002535d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:01.422296 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30201600}
I20260709 13:02:01.424475  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.428149  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.429267  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.431455  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.432749  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.437548  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.439730  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:01.446022  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64908.9us
I20260709 13:02:01.449029  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:01.433372  1661 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:01.274072  4100 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.464180  4096 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:02:01.469640  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.329427  3989 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:01.497960  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:01.506901  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:01.507930  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:01.508580  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 312152us
I20260709 13:02:01.476127  4035 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:01.509472  1556 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.880s (SENT)
I20260709 13:02:01.513339  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:01.520099  1556 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.974s (SENT)
W20260709 13:02:01.521379  1556 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.003s (SENT)
W20260709 13:02:01.526424  1556 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.013s (SENT)
W20260709 13:02:01.337018  4087 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.571s	user 0.015s	sys 0.000s
I20260709 13:02:01.327984  4080 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:01.337183  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.289165  3600 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.340487  1663 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:01.269402  4062 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.349885  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1976) took 1941 ms. Trace:
I20260709 13:02:01.546396  1741 rpcz_store.cc:276] 0709 13:01:59.408741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.411100 (+  2359us) service_pool.cc:224] Handling call
0709 13:01:59.429648 (+ 18548us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:01:59.617277 (+187629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:01:59.617355 (+    78us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:01:59.618262 (+   907us) log.cc:844] Serialized 26 byte log entry
0709 13:02:00.385231 (+766969us) spinlock_profiling.cc:243] Waited 4.69 ms on lock 0x7b180007b340. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:00.579438 (+194207us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:00.579488 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.579564 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:01.349391 (+769827us) raft_consensus.cc:1685] finished
0709 13:02:01.349424 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.349795 (+   371us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":191309,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":26601,"spinlock_wait_cycles":10787584,"thread_start_us":370208,"threads_started":2,"wal-append.queue_time_us":1498610}
I20260709 13:02:01.352324  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1437) took 1397 ms. Trace:
I20260709 13:02:01.549424  1667 rpcz_store.cc:276] 0709 13:01:59.954948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.599179 (+644231us) service_pool.cc:224] Handling call
0709 13:02:01.066220 (+467041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.096212 (+ 29992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.351984 (+255772us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b540013d3d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:01.352229 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073343360}
I20260709 13:02:01.358834  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:01.551301  1664 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:01.363826  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.552382  2314 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:01.348562  3815 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.483033  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.486049  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.486209  4088 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:01.528219  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:01.560561  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.561061  1665 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:01.530258  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:01.562088  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:01.511199  4071 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:01.192070  4069 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:01.222738  4040 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:01.456107  3642 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:01.536810  1666 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:01.456399  4043 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:01.542724  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93815us
I20260709 13:02:01.633860  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.634829  1589 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:01.541596  4044 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.638864  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.372411  1489 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:01.558429  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1436) took 1679 ms (client timeout 1821 ms). Trace:
W20260709 13:02:01.647265  1663 rpcz_store.cc:269] 0709 13:01:59.879360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.085828 (+206468us) service_pool.cc:224] Handling call
0709 13:02:01.558117 (+1472289us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:01.558329 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2781954944}
I20260709 13:02:01.532740  1662 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:01.574959  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:01.581769  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:01.650720  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69087.1us
W20260709 13:02:01.583889  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57383.4us
I20260709 13:02:01.567771  4088 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:01.538290  4077 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.653517  1556 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.267s (SENT)
I20260709 13:02:01.624365  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.629209  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.657635  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.657698  1665 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:01.658438  1666 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:02:01.659683  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.089s	user 0.042s	sys 0.057s
I20260709 13:02:01.344453  4061 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.651952  4080 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:01.671108  1733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.114s	user 0.061s	sys 0.090s
W20260709 13:02:01.672950  1733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.124s	user 0.061s	sys 0.090s
W20260709 13:02:01.676838  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:01.680339  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.680608  1661 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
W20260709 13:02:01.662015  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.042s	sys 0.057s
I20260709 13:02:01.580392  4071 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:01.680179  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:01.578655  4082 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.143s	user 0.004s	sys 0.011s
I20260709 13:02:01.591697  3815 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:01.687506  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:01.683759  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:01.688359  1734 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.114s	user 0.023s	sys 0.068s
W20260709 13:02:01.747658  1734 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.173s	user 0.023s	sys 0.068s
I20260709 13:02:01.681558  1489 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:01.689278  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1440) took 1629 ms (client timeout 2013 ms). Trace:
I20260709 13:02:01.633627  3642 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:01.595682  3959 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.937s	user 0.009s	sys 0.004s
I20260709 13:02:01.616539  3790 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:01.663277  3952 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.637s	user 0.011s	sys 0.002s
I20260709 13:02:01.684934  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 311) took 1227 ms. Trace:
I20260709 13:02:01.768283  1663 rpcz_store.cc:276] 0709 13:02:00.464636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.679091 (+1214455us) service_pool.cc:224] Handling call
0709 13:02:01.684230 (+  5139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.684459 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.684500 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:01.684564 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:01.684608 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.684683 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:01.684857 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.734511  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.770558  2318 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:01.627094  3405 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:01.737074  1593 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.115s	user 0.039s	sys 0.125s
I20260709 13:02:01.737816  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:01.739833  1589 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:01.633317  3989 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:01.744860  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:01.749162  1662 rpcz_store.cc:269] 0709 13:02:00.060400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.362732 (+302332us) service_pool.cc:224] Handling call
0709 13:02:01.160424 (+797692us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:01.689205 (+528781us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28010240}
I20260709 13:02:01.767777  3903 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.809471  1662 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:01.810169  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1450) took 1222 ms. Trace:
I20260709 13:02:01.810493  1662 rpcz_store.cc:276] 0709 13:02:00.587906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.808930 (+1221024us) service_pool.cc:224] Handling call
0709 13:02:01.809281 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.809923 (+   642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.810114 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.753188  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:01.756456  1584 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:02:01.670783  4045 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:01.672493  4080 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.771570  4048 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.735110  4088 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.772809  1590 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 13:02:01.772771  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85397.8us
I20260709 13:02:01.515374  4035 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:01.759052  3642 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:01.750118  4077 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:01.746587  4063 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:01.752338  3914 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.662473  4100 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:01.755599  4043 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.869316  4043 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.803298  1593 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.181s	user 0.040s	sys 0.125s
I20260709 13:02:01.804909  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:01.750100  1489 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:01.813213  1662 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:02:01.782466  3886 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:01.826342  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:01.750988  4082 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:01.818523  4045 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.825598  4055 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:01.822751  4080 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:01.834299  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:01.772564  3790 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.779731  3952 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.842279  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:01.848476  4088 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:01.848958  4038 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:01.839661  4105 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 13:02:01.832952  4048 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.940997  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:01.862635  3914 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:01.868177  4037 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:01.874708  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1453) took 1248 ms. Trace:
I20260709 13:02:01.824147  3903 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:01.879820  1489 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:01.808874  4071 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:01.884294  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.886591  1590 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:01.878369  4045 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:01.998397  4062 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:01.998778  3790 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.015305  3952 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.051831  3600 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:02.030666  3963 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.010s	user 0.011s	sys 0.004s
I20260709 13:02:01.922091  1665 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 13:02:01.928699  1668 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:02:01.931427  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.062315  1665 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 2
I20260709 13:02:02.062744  1665 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:02.063066  1665 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:01.948405  1662 rpcz_store.cc:276] 0709 13:02:00.626507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.811232 (+1184725us) service_pool.cc:224] Handling call
0709 13:02:01.811507 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.874395 (+ 62888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.874640 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:01.664373  4044 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:02.063450  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.955853  1660 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:01.961704  4086 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.243s	user 0.045s	sys 0.135s
I20260709 13:02:01.957828  1489 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:02:01.875111  3886 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.957732  4038 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:01.886106  4063 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.068054  1489 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:01.981398  4102 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:01.938998  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.953941  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.073134  1489 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:02.074788  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.074913  1588 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:02.063474  1665 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 5.4
W20260709 13:02:01.904443  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.975220  4037 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:02.083842  4086 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.461s	user 0.045s	sys 0.136s
I20260709 13:02:02.045610  4071 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:02.076799  1489 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.104089  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.107467  4044 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.072297  3886 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:01.939428  4055 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.092012  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1444) took 1783 ms. Trace:
I20260709 13:02:02.118623  1660 rpcz_store.cc:276] 0709 13:02:00.308408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.349302 (+ 40894us) service_pool.cc:224] Handling call
0709 13:02:02.091774 (+1742472us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b54001a8bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:02.091941 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3491141888}
I20260709 13:02:02.072527  4038 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:02.077031  4011 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.132844  1590 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:02.107475  1489 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 4 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:02.136564  1589 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:02.143213  4082 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.149597  4028 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:02.122087  3951 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.367s	user 0.013s	sys 0.002s
W20260709 13:02:02.152042  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.152880  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:02.095826  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.164047  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:01.881628  1664 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:02:02.165853  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 313) took 1654 ms (client timeout 2113 ms). Trace:
W20260709 13:02:02.166424  1664 rpcz_store.cc:269] 0709 13:02:00.511792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.580208 (+ 68416us) service_pool.cc:224] Handling call
0709 13:02:01.358621 (+778413us) spinlock_profiling.cc:243] Waited 59.5 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:02.165760 (+807139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136745344}
I20260709 13:02:02.150749  4028 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.174989  1584 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:02.135530  1489 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 5, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 5 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.180598  1489 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.183445  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:02.187553  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.089738  4102 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.085544  1665 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:02.188445  1589 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:02.160849  4086 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:02.096019  3963 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.207204  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.216974  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:02.218132  1590 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:02.210803  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:02.209232  1489 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.235421  4055 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.214634  4086 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:02.242302  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.236212  1489 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.304867  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.306103  1589 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:02.305327  1489 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.242785  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:02.302224  4034 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.906s	user 0.013s	sys 0.005s
I20260709 13:02:02.239961  4037 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.323433  3963 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.236721  4061 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.368361  4034 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.302178  1590 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:02.373108  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.376619  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.379573  1590 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:02.381848  4043 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.383846  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.384233  4037 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.384845  4037 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:02.334477  1556 connection.cc:256] RPC call timeout handler was delayed by -1.2537025840000524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28340000000000004s for a response
I20260709 13:02:02.237010  4069 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260709 13:02:02.300549  3907 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:02.367131  4028 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:02.301936  3900 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.038s	user 0.015s	sys 0.000s
I20260709 13:02:02.353649  1489 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:02.178812  3951 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.250448  4048 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.270007  3914 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:02.267836  4063 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.269093  4086 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.372538  4034 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.387727  4037 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:02.394876  3907 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.449872  4045 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.453518  4045 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.395048  4069 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:02.454279  4069 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:02.455443  4069 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.457135  4069 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:02.395215  1556 connection.cc:256] RPC call timeout handler was delayed by -1.2537025840000524s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28340000000000004s for a response
I20260709 13:02:02.385457  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.467144  1584 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:02.399209  1489 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:02.484278  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.483353  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.488828  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:02.492774  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.534343  1590 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:02.494355  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.408219  4102 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:02.487339  4040 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.820s	user 0.011s	sys 0.002s
W20260709 13:02:02.503988  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 469826us
I20260709 13:02:02.396395  3903 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:02.420270  4086 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:02.458244  3907 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.411810  3951 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:02.484833  3904 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.228s	user 0.055s	sys 0.110s
I20260709 13:02:02.490144  4061 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.506937   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:02.565153  4061 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:02.565933  3904 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.320s	user 0.055s	sys 0.111s
W20260709 13:02:02.510669  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.514261  4082 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.487715  3886 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.483778  4028 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:02.520417  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.505775  3642 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.660s	user 0.008s	sys 0.004s
I20260709 13:02:02.508952  4037 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:02.431209  4070 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.996s	user 0.007s	sys 0.005s
I20260709 13:02:02.397125  3900 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.556702  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:02.563369  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 729982us
I20260709 13:02:02.564572  3989 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:02.415832  4080 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:02.649482  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.570137  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:02.570746  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:02.672888  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:02.643551  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:02.587261  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.642776  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:02.576601  1588 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:02.678617  1590 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 13:02:02.680491  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.666s	user 0.052s	sys 0.124s
W20260709 13:02:02.681128  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.666s	user 0.052s	sys 0.124s
W20260709 13:02:02.683379  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 126808us
W20260709 13:02:02.683820  2283 connection.cc:256] RPC call timeout handler was delayed by -1.4424875079999993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1755s for a response
I20260709 13:02:02.685078  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:02.688871  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.692216  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132102us
I20260709 13:02:02.703158  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 312) took 2200 ms. Trace:
I20260709 13:02:02.575333  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.703703  1661 rpcz_store.cc:276] 0709 13:02:00.502902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.685283 (+1182381us) service_pool.cc:224] Handling call
0709 13:02:01.775389 (+ 90106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.681990 (+906601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:02.682025 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:02.682069 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:02.682100 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.702826 (+ 20726us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:02.702890 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.703070 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75681,"raft.queue_time_us":76260,"spinlock_wait_cycles":1908921728}
I20260709 13:02:02.705921  1589 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:02.707459  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:02.698974  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:02.560767  3815 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:02.715781  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:02.716065  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:02.719383  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:02.701488  4071 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:02.600240  4061 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.696135  3903 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.605000  4062 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:02:02.617348  3872 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.158s
I20260709 13:02:02.725265  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:02.599080  3907 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:02.606984  4076 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:02.721529  1589 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:02.696635  4081 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:02.767338  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.744393  1668 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:02.734714  3951 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:02.764772  2316 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:02.765882  2314 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:02:02.748616  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.088s	user 0.055s	sys 0.068s
I20260709 13:02:02.699818  4037 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.706903  4045 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 13:02:02.711830  3904 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
W20260709 13:02:02.777946  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.117s	user 0.056s	sys 0.068s
I20260709 13:02:02.772774  1660 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:02.705888  3642 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.617173  4038 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.696250  4082 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.735291  4040 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:02.723675  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:02.715600  4034 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 13:02:02.760887  4070 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:02.720860  4063 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:02:02.751075  4062 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.794237  2313 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:02.795048  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71490.7us
I20260709 13:02:02.706079  4028 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.725463  3600 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 13:02:02.812693  1667 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:02.818701  4028 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.781649  3951 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:02.821448  3951 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:02.805133  2316 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:02.766283  4076 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.820786  4028 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.752228  3815 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.755684  3903 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.807384  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:02.786841  4081 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.771114  3907 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:02.716163  3849 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.591s	user 0.041s	sys 0.054s
I20260709 13:02:02.845356  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:02.814000  4045 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:02.914881  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107799us
I20260709 13:02:02.916538  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:02.810253  4070 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:02.911103  3849 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.786s	user 0.041s	sys 0.054s
I20260709 13:02:02.911590  1589 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:02:02.915376  4045 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.919620  3904 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:02.920174  3989 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:02.920660  3600 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:02.921917  4048 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:02.921918  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 315) took 2554 ms. Trace:
I20260709 13:02:02.913211  2310 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:02.858494  4028 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.927701  4037 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:02.930261  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2122) took 1069 ms. Trace:
I20260709 13:02:02.964543  1588 rpcz_store.cc:276] 0709 13:02:01.861122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.471283 (+610161us) service_pool.cc:224] Handling call
0709 13:02:02.930009 (+458726us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b540025bcd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:02.930179 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93545856}
I20260709 13:02:02.930508  3815 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:02.931463  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.932119  3903 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.932934  4034 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:02.972024  4034 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.972395  3600 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:02.972954  3600 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.973299  4034 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.974146  3600 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.975112  4034 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:02.975826  3600 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:02.932709  4076 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.933405  4118 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:02.911312  4069 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:02.923341  4044 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:02.989140  4044 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:02.990865  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 318) took 2567 ms. Trace:
I20260709 13:02:02.990662  4044 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:02.991397  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:02.991294  1733 rpcz_store.cc:276] 0709 13:02:00.423597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.423845 (+   248us) service_pool.cc:224] Handling call
0709 13:02:00.441881 (+ 18036us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.458384 (+ 16503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.458443 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:00.459063 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 13:02:01.468408 (+1009345us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b18003297a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0557 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:01.675559 (+207151us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:01.675627 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:01.675738 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:02.990509 (+1314771us) raft_consensus.cc:1685] finished
0709 13:02:02.990528 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.990797 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15921,"raft.queue_time_us":24118,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":41651,"spinlock_wait_cycles":16224000,"thread_start_us":1126821,"threads_started":1,"wal-append.queue_time_us":2492461}
I20260709 13:02:02.992504  3903 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:02.993595  3903 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:02.959748  1736 rpcz_store.cc:276] 0709 13:02:00.367554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.367738 (+   184us) service_pool.cc:224] Handling call
0709 13:02:00.374764 (+  7026us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.375693 (+   929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.375722 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:00.377396 (+  1674us) log.cc:844] Serialized 26 byte log entry
0709 13:02:00.738996 (+361600us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:00.739042 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:00.739120 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:02.921586 (+2182466us) raft_consensus.cc:1685] finished
0709 13:02:02.921619 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:02.921834 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14280,"raft.queue_time_us":1995,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":53871,"thread_start_us":237393,"threads_started":1,"wal-append.queue_time_us":2377013}
I20260709 13:02:02.967288  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2006) took 1792 ms. Trace:
I20260709 13:02:02.965356  3900 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:02.987274  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 1960) took 2422 ms (client timeout 1679 ms). Trace:
W20260709 13:02:02.999212  1591 rpcz_store.cc:269] 0709 13:02:00.561289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.561524 (+   235us) service_pool.cc:224] Handling call
0709 13:02:02.983459 (+2421935us) spinlock_profiling.cc:243] Waited 2.39 s on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:02.983655 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5489937280}
I20260709 13:02:02.995040  1667 rpcz_store.cc:276] 0709 13:02:01.174524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.628826 (+454302us) service_pool.cc:224] Handling call
0709 13:02:02.967064 (+1338238us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540044e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:02.967220 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2654445824}
I20260709 13:02:02.972079  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:02.990656  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:02.917528  4062 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:02.992492  4044 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:02.969197  3886 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.025624  1588 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:03.000655  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.031482  1591 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:03.032823  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:03.034098  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:03.035111  1591 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.036445  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.004s	user 0.023s	sys 0.019s
W20260709 13:02:03.036964  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.004s	user 0.023s	sys 0.019s
I20260709 13:02:03.038563  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:03.080663  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.081810  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:02.994145  3903 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.090471  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.092108  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.089777  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:03.094658  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.094918  1736 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:03.095506  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2009) took 1354 ms. Trace:
I20260709 13:02:03.095896  1736 rpcz_store.cc:276] 0709 13:02:01.740772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.030295 (+1289523us) service_pool.cc:224] Handling call
0709 13:02:03.095430 (+ 65135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.098454  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:03.099892  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:02.997391  4069 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.022051  4037 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:03.021678  3907 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:03.000540  3815 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:03.023412  4028 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:03.008615  4080 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:03.098711  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.343s	user 0.018s	sys 0.075s
W20260709 13:02:03.114722  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.359s	user 0.018s	sys 0.075s
I20260709 13:02:03.115401  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.115692  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 329) took 2690 ms. Trace:
I20260709 13:02:02.994599  4118 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:03.116317  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.158807  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:03.159099  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.129338  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.160612  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:02.860692  4054 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.139s	user 0.024s	sys 0.071s
W20260709 13:02:03.163606  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.137s	user 0.042s	sys 0.044s
W20260709 13:02:03.164286  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.138s	user 0.042s	sys 0.044s
I20260709 13:02:03.158115  1592 rpcz_store.cc:276] 0709 13:02:00.425369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.429203 (+  3834us) service_pool.cc:224] Handling call
0709 13:02:00.430464 (+  1261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.430751 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:00.754610 (+323859us) log.cc:844] Serialized 14 byte log entry
0709 13:02:03.115290 (+2360680us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:03.115357 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:03.115412 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.115494 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.115633 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":257546,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":122,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":2652230,"threads_started":2}
I20260709 13:02:03.165979  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.168385  1592 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:03.192665  4062 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:03.161595  3815 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:03.170619  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.969s	user 0.032s	sys 0.071s
I20260709 13:02:03.172631  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2008) took 1487 ms. Trace:
W20260709 13:02:03.209710  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.008s	user 0.032s	sys 0.071s
I20260709 13:02:03.209787  1736 rpcz_store.cc:276] 0709 13:02:01.685745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.096839 (+1411094us) service_pool.cc:224] Handling call
0709 13:02:03.172559 (+ 75720us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.173290  1589 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:03.180374  1705 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.978s	user 0.008s	sys 0.018s
W20260709 13:02:03.212163  1705 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.010s	user 0.008s	sys 0.018s
I20260709 13:02:03.102761  3642 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:03.161935  4122 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 13:02:03.106168  4038 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:02.997602  4045 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.160187  4105 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.159981  4048 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.125749  4069 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:03.127347  4082 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:03.160344  4108 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:03.210486  4119 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:03.211189  3886 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.208274  4070 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.114212  4037 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.202880  1591 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:03.202880  1587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.710s	user 0.041s	sys 0.098s
W20260709 13:02:03.212591  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 2.01159e+06us
I20260709 13:02:03.217371  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.218446  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:03.241830  4086 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.677s	user 0.013s	sys 0.000s
I20260709 13:02:03.259647  1736 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:03.173832  4054 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.458s	user 0.024s	sys 0.072s
I20260709 13:02:03.243288  1667 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:03.243316  1664 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:03.262285  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.244516  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:03.245590  4119 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.245759  4108 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.268816  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:03.215193  3805 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.058s	user 0.023s	sys 0.027s
W20260709 13:02:03.272212  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.215440  4050 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.525s	user 0.014s	sys 0.024s
W20260709 13:02:03.260730  1587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.042s	sys 0.098s
W20260709 13:02:03.261303  1705 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.095s (SENT)
I20260709 13:02:03.249918  4081 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:03.236621  4066 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.732s	user 0.000s	sys 0.021s
W20260709 13:02:03.273111  1586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.607s	user 0.021s	sys 0.023s
I20260709 13:02:03.273219  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.259905  4132 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:03.274499  1742 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 13:02:03.276443  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.285849  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.307031  1589 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:03.306988  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.261015  1592 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.277526  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.277442  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:03.294170  1586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.629s	user 0.021s	sys 0.023s
I20260709 13:02:03.302270  4045 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.303045  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1436) took 2730 ms. Trace:
I20260709 13:02:03.311606  1739 rpcz_store.cc:276] 0709 13:02:00.572677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.576162 (+  3485us) service_pool.cc:224] Handling call
0709 13:02:00.598962 (+ 22800us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.659955 (+ 60993us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:00.659993 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:00.660566 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 13:02:02.788869 (+2128303us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:02.788914 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:02.992785 (+203871us) spinlock_profiling.cc:243] Waited 60.5 ms on lock 0x7b540014ebd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:02.992846 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:03.302686 (+309840us) raft_consensus.cc:1685] finished
0709 13:02:03.302710 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.302962 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":869,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":82,"spinlock_wait_cycles":139195520,"thread_start_us":2127719,"threads_started":1,"wal-append.queue_time_us":2587734}
W20260709 13:02:03.274950  4050 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.589s	user 0.015s	sys 0.024s
I20260709 13:02:03.309989  1591 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.323307  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.324056  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:03.275451  3805 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.118s	user 0.024s	sys 0.027s
I20260709 13:02:03.308368  1733 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:03.246913  4105 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.246937  4048 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.260056  3886 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.337807  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.338157  3886 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:03.338464  1739 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.338943  4105 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:03.262975  3951 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.321745  4040 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:03.342793  4040 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:03.343883  4048 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.343629  4040 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.345346  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:03.346436  1588 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
I20260709 13:02:03.347623  4119 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:03.348800  4118 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:03.293856  4066 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.806s	user 0.000s	sys 0.022s
I20260709 13:02:03.327558  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:03.351636  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.352563  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:03.276067  3405 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:03.355069  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.355784  1588 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.357802  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:03.264109  3939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 3.012s	user 0.044s	sys 0.115s
I20260709 13:02:03.333763  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1983) took 3342 ms. Trace:
W20260709 13:02:03.333953  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.335369  3815 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.310895  4132 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.358459  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.370102  1733 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:03.326833  4071 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:03.362890  1741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.566s	user 0.005s	sys 0.023s
I20260709 13:02:03.365787  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:03.367107  1740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.114s	user 0.007s	sys 0.017s
W20260709 13:02:03.373203  1740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.120s	user 0.007s	sys 0.017s
W20260709 13:02:03.373687  1741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.585s	user 0.006s	sys 0.023s
I20260709 13:02:03.373422  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:03.374198  1589 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.367166  1737 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.175s	user 0.002s	sys 0.016s
I20260709 13:02:03.366519  1742 rpcz_store.cc:276] 0709 13:01:59.991595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:59.991842 (+   247us) service_pool.cc:224] Handling call
0709 13:02:00.058552 (+ 66710us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.305572 (+3247020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.321603 (+ 16031us) spinlock_profiling.cc:243] Waited 3.22 s on lock 0x7b540039e4d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:03.333669 (+ 12066us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7395889920}
I20260709 13:02:03.371179  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.311220  4069 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:03.375133  1737 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.183s	user 0.002s	sys 0.016s
I20260709 13:02:03.375687  1733 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:02:03.343638  3951 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:03.356628  4063 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:03.373248  1736 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:03.382439  1735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.826s	user 0.001s	sys 0.017s
W20260709 13:02:03.386919  1735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.831s	user 0.002s	sys 0.017s
I20260709 13:02:03.363502  4048 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:03.393388  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:03.364619  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:03.365202  4105 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:03.385277  1592 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:03.366822  3886 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:03.383534  1739 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.400035  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.309893  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:03.366457  3939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.126s	user 0.045s	sys 0.115s
I20260709 13:02:03.396517  2312 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:03.397830  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:03.402199  1592 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.372911  4071 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:03.412477  1591 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.404618  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.405728  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:03.413777  1588 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:03.404654  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.415597  1736 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:03.417290  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.418140  1736 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.405812  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.372036  4132 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.413965  1733 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 13:02:03.420670  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.421267  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:03.407819  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91687.3us
I20260709 13:02:03.422518  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.423321  1585 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:03.425138  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:03.421806  1666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.588s	user 0.017s	sys 0.020s
W20260709 13:02:03.427235  1666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.598s	user 0.017s	sys 0.020s
I20260709 13:02:03.432792  1589 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.419979  4071 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.423803  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.435446  1742 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:03.436251  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2130) took 1532 ms. Trace:
I20260709 13:02:03.390640  4048 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:03.364719  3405 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.401361  2310 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.399390  3886 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:03.451318  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.438709  1588 rpcz_store.cc:276] 0709 13:02:01.909750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.434251 (+1524501us) service_pool.cc:224] Handling call
0709 13:02:03.434575 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.435932 (+  1357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.435959 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:03.435999 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:03.436028 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.436083 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.436195 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":88}
I20260709 13:02:03.443238  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1976) took 1779 ms. Trace:
I20260709 13:02:03.383402  4069 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.448829  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:03.459019  1739 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:03.440373  4105 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:03.462868  4033 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.457s	user 0.018s	sys 0.017s
I20260709 13:02:03.452826  1588 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:03.481436  1588 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:03.482553  1593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.603s	user 0.018s	sys 0.018s
W20260709 13:02:03.483073  1593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.610s	user 0.019s	sys 0.019s
I20260709 13:02:03.452339  1585 rpcz_store.cc:276] 0709 13:02:01.663909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.664092 (+   183us) service_pool.cc:224] Handling call
0709 13:02:01.730053 (+ 65961us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.435832 (+1705779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.442977 (+  7145us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b5400036bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:03.443175 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3894060288}
I20260709 13:02:03.451711  3886 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.452061  1733 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.491716  1592 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:03.457811  4076 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.495676  1591 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:03.419914  4119 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:03.438613  4038 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:03.435189  4050 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:03.451292  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.444242  4048 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.438261  4080 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:03.486765  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1962) took 2921 ms. Trace:
I20260709 13:02:03.512908  1593 rpcz_store.cc:276] 0709 13:02:00.565280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.565486 (+   206us) service_pool.cc:224] Handling call
0709 13:02:00.586373 (+ 20887us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:00.586646 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:01.872739 (+1286093us) log.cc:844] Serialized 14 byte log entry
0709 13:02:03.485970 (+1613231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:03.486347 (+   377us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:03.486410 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.486507 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.486693 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":101948,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":622,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":8304,"thread_start_us":2860911,"threads_started":2}
I20260709 13:02:03.459144  4040 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:03.493275  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.519390  1733 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.438961  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.483124  1588 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.520921  1588 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.497972  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:03.521807  1742 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.485297  4033 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.482s	user 0.018s	sys 0.017s
I20260709 13:02:03.501536  4038 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:03.502552  1734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.689s	user 0.013s	sys 0.027s
W20260709 13:02:03.502717  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51505.1us
W20260709 13:02:03.526964  1734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.713s	user 0.013s	sys 0.027s
I20260709 13:02:03.475541  4105 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:03.504433  2280 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.464s	user 0.007s	sys 0.031s
W20260709 13:02:03.532195  2280 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.492s	user 0.008s	sys 0.031s
I20260709 13:02:03.512176  3907 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:03.507454  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:03.475698  4096 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.298s	user 0.011s	sys 0.026s
I20260709 13:02:03.524108  4037 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:03.533653  4096 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.435s	user 0.012s	sys 0.026s
W20260709 13:02:03.511443  4031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.509s	user 0.011s	sys 0.016s
W20260709 13:02:03.536252  4031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.534s	user 0.011s	sys 0.016s
I20260709 13:02:03.538306  4108 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:03.526664  1659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.704s	user 0.009s	sys 0.018s
W20260709 13:02:03.539407  1659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.717s	user 0.009s	sys 0.018s
I20260709 13:02:03.540162  3405 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.540851  4043 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:03.501402  4119 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:03.543268  4119 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.546362  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 2.
W20260709 13:02:03.548755  4129 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:02:03.554957  4128 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
W20260709 13:02:03.530352  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91489.3us
I20260709 13:02:03.558157  1632 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.561749  4043 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:03.562026  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:03.562382  4043 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.562906  4043 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.564467  4123 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:03.564762  3903 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:03.565615  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.529150  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.566269  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.566864  4069 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:03.567397  4069 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:03.567632  4033 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:03.568058  4033 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.568465  4033 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.532584  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.50998e+06us
W20260709 13:02:03.573696  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.573783  2280 connection.cc:256] RPC call timeout handler was delayed by -1.4070711800000026s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21130000000000002s for a response
W20260709 13:02:03.574468  2280 connection.cc:256] RPC call timeout handler was delayed by -1.3683537759999922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1986s for a response
I20260709 13:02:03.575688  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:03.577172  2315 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:03.462298  4035 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.583s	user 0.007s	sys 0.016s
W20260709 13:02:03.577819  4035 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.700s	user 0.008s	sys 0.017s
I20260709 13:02:03.527997  4038 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:03.587124  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.588184  4062 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:03.588685  4062 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:03.589221  4062 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:03.593677  4130 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:03.594336  4130 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:03.595420  4130 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.567214  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:03.599452  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:03.601394  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:03.602833  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:03.603833  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:03.603703  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.605648  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.601785  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:03.606840  1667 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:03.606870  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.608464  1661 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
W20260709 13:02:03.512176  4056 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.746s	user 0.000s	sys 0.041s
I20260709 13:02:03.526497  3951 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:03.597823  4105 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:03.618930  3600 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.641s	user 0.007s	sys 0.007s
I20260709 13:02:03.613691  3907 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:03.615705  4069 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:03.625890  1663 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.008s	sys 0.020s
W20260709 13:02:03.639631  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.617769  4119 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:03.629570  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.641637  4069 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:03.638163  1584 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.963s	user 0.017s	sys 0.025s
I20260709 13:02:03.619815  3951 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:03.616209  4056 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.857s	user 0.000s	sys 0.041s
I20260709 13:02:03.622583  4131 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:03.641625  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.920s	user 0.000s	sys 0.034s
W20260709 13:02:03.667627  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.946s	user 0.000s	sys 0.034s
W20260709 13:02:03.644011  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.936s	user 0.015s	sys 0.045s
W20260709 13:02:03.668166  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.961s	user 0.016s	sys 0.046s
I20260709 13:02:03.650298  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:03.668534  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 961970us
I20260709 13:02:03.672842  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.679579  1664 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:03.676970  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.686813  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.691469  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:03.697741  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.650360  4037 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:03.629588  4033 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:03.654361  2318 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.008s	sys 0.017s
I20260709 13:02:03.638429  3907 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:03.666551  3951 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:03.643215  1590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s	user 0.004s	sys 0.023s
W20260709 13:02:03.734716  1590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.005s	sys 0.023s
I20260709 13:02:03.700232  1742 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:03.700649  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.703931  1736 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:03.736531  1662 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:03.737908  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2001) took 2480 ms. Trace:
I20260709 13:02:03.738459  1736 rpcz_store.cc:276] 0709 13:02:01.257567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.736770 (+2479203us) service_pool.cc:224] Handling call
0709 13:02:03.737207 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.737472 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.737530 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:03.737593 (+    63us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:03.737640 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.737715 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.737839 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.709745  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.710382  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:03.713915  2318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.922s	user 0.008s	sys 0.017s
I20260709 13:02:03.661682  4069 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:03.653254  4119 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:03.718273  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.750182  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:03.722239  1667 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:02:03.658814  1584 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.984s	user 0.017s	sys 0.025s
W20260709 13:02:03.639281  1663 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.131s	user 0.008s	sys 0.020s
I20260709 13:02:03.737972  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.708328  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1445) took 2882 ms. Trace:
I20260709 13:02:03.828459  1738 rpcz_store.cc:276] 0709 13:02:00.826163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.188286 (+362123us) service_pool.cc:224] Handling call
0709 13:02:01.188617 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.189542 (+   925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.189594 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.194952 (+  5358us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.210264 (+2015312us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:03.210308 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.210395 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:03.708058 (+497663us) raft_consensus.cc:1685] finished
0709 13:02:03.708081 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.708241 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3541,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":6507,"thread_start_us":2008758,"threads_started":1,"wal-append.queue_time_us":2378647}
I20260709 13:02:03.831271  1738 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:03.832058  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2003) took 2164 ms. Trace:
I20260709 13:02:03.708947  4131 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:03.834528  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2000) took 2588 ms. Trace:
I20260709 13:02:03.835081  1742 rpcz_store.cc:276] 0709 13:02:01.246626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.736069 (+2489443us) service_pool.cc:224] Handling call
0709 13:02:03.833898 (+ 97829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.834117 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.834154 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:03.834210 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:03.834255 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.834330 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:03.834447 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.724432  3939 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:03.678045  4080 raft_consensus.cc:2764] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 13:02:03.767342  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.938901  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:03.940009  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.751022  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.940848  1733 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:03.941282  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 190397us
I20260709 13:02:03.743345  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.832444  1738 rpcz_store.cc:276] 0709 13:02:01.667693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.830598 (+2162905us) service_pool.cc:224] Handling call
0709 13:02:03.831020 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.831757 (+   737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.831992 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.735808  3907 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.953156  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:03.894658  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.010s	sys 0.020s
I20260709 13:02:03.698395  3405 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.128s
W20260709 13:02:03.899044  1489 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.009s	sys 0.026s
I20260709 13:02:03.770558  4119 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:03.743029  4037 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:03.902400  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.840s	user 0.006s	sys 0.025s
I20260709 13:02:03.935213  1742 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 13:02:03.971972  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2004) took 2304 ms. Trace:
I20260709 13:02:03.972509  1742 rpcz_store.cc:276] 0709 13:02:01.668045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.905862 (+2237817us) service_pool.cc:224] Handling call
0709 13:02:03.934841 (+ 28979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.971542 (+ 36701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.971888 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:03.940862  1667 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:03.974642  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.976192  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:03.976897  1667 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:03.945003  1733 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:03.978068  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2006) took 2309 ms. Trace:
I20260709 13:02:03.978533  1733 rpcz_store.cc:276] 0709 13:02:01.668693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.944173 (+2275480us) service_pool.cc:224] Handling call
0709 13:02:03.944532 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:03.977697 (+ 33165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.977975 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:03.948472  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 213534us
I20260709 13:02:03.980602  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:03.949155  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:03.982084  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.006s	sys 0.026s
I20260709 13:02:03.753786  4048 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:03.999275  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:03.747282  4056 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.154s
I20260709 13:02:03.984719  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:04.010859  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:03.956451  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.877s	user 0.011s	sys 0.020s
W20260709 13:02:04.003247  1634 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.010s	sys 0.027s
I20260709 13:02:03.988191  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:04.011834  1634 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.010s	sys 0.027s
W20260709 13:02:04.012180  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 973751us
W20260709 13:02:04.012164  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 851019us
I20260709 13:02:04.019999  2315 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:04.027360  4123 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:02:03.972316  4071 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.008s
I20260709 13:02:04.030105  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:04.033175  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:04.036252  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:03.894367  4062 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.083269  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2007) took 2414 ms. Trace:
I20260709 13:02:04.083715  1742 rpcz_store.cc:276] 0709 13:02:01.669102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.023817 (+2354715us) service_pool.cc:224] Handling call
0709 13:02:04.082698 (+ 58881us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.082918 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.082952 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.083000 (+    48us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:04.083029 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.083087 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.083211 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:04.039017  1737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.006s	sys 0.040s
I20260709 13:02:03.971719  3600 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:04.074013  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:04.072134  1741 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.000s	sys 0.039s
I20260709 13:02:04.075469  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:04.092621  1741 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.000s	sys 0.040s
I20260709 13:02:04.093065  2315 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:03.960846  4045 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:04.078722  2312 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
W20260709 13:02:04.088634  1737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.007s	sys 0.040s
I20260709 13:02:04.033666  3939 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.113073  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:04.095813  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:04.112401  1733 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.123915  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2010) took 2362 ms. Trace:
I20260709 13:02:04.124325  1733 rpcz_store.cc:276] 0709 13:02:01.762078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.111842 (+2349764us) service_pool.cc:224] Handling call
0709 13:02:04.112199 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.123588 (+ 11389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.123844 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:04.130918  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:04.016031  4035 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
W20260709 13:02:04.097476  4066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.009s	sys 0.032s
W20260709 13:02:04.139909  3805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.019s	sys 0.025s
I20260709 13:02:04.104693  4118 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:04.156416  4035 consensus_queue.cc:1243] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:02:04.190052  4035 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:04.191452  4035 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:04.191907  4035 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:04.195399  4043 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.000s
W20260709 13:02:04.198326  4146 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:02:04.200886  4124 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.355s	user 0.002s	sys 0.000s
W20260709 13:02:04.201642  4144 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:02:04.227461  4043 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.204137  4037 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.230350  4043 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:04.152195  4066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.009s	sys 0.032s
I20260709 13:02:04.217191  4160 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:04.232550  4160 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:04.233878  4160 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.217437  4062 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.219103  4048 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:04.222565  4040 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.697s	user 0.008s	sys 0.003s
W20260709 13:02:04.224725  4038 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.645s	user 0.009s	sys 0.003s
I20260709 13:02:04.237903  4038 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.240620  4038 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:04.245144  4138 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.300s	user 0.000s	sys 0.000s
W20260709 13:02:04.247045  4136 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.495s	user 0.001s	sys 0.008s
I20260709 13:02:04.193050  4035 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.254743  4035 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.266822  4119 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.094067  4131 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:04.305927  4141 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.127s	user 0.002s	sys 0.000s
I20260709 13:02:03.945823  4132 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.309682  4132 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:03.957000  1489 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.009s	sys 0.026s
I20260709 13:02:04.347751  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 334) took 3887 ms. Trace:
W20260709 13:02:04.348899  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 368281us
I20260709 13:02:04.348835  1587 rpcz_store.cc:276] 0709 13:02:00.460456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.460648 (+   192us) service_pool.cc:224] Handling call
0709 13:02:00.543484 (+ 82836us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:00.543733 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:00.543755 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.408717 (+864962us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b2c00032118. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:01.409371 (+   654us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.275298 (+1865927us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:03.275343 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.275416 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.346350 (+1070934us) raft_consensus.cc:1685] finished
0709 13:02:04.346444 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.347265 (+   821us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4470528,"thread_start_us":1782647,"threads_started":1,"wal-append.queue_time_us":2248802}
I20260709 13:02:04.350958  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1972) took 2834 ms. Trace:
W20260709 13:02:04.352527  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.353351  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:04.354256  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.354830  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:04.351634  1586 rpcz_store.cc:276] 0709 13:02:01.516364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.516555 (+   191us) service_pool.cc:224] Handling call
0709 13:02:01.517013 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.545570 (+ 28557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.545625 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.568154 (+ 22529us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.309579 (+1741425us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:03.309625 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.309693 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.350247 (+1040554us) raft_consensus.cc:1685] finished
0709 13:02:04.350294 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.350662 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5529,"raft.queue_time_us":10106,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":23783,"thread_start_us":1643923,"threads_started":1,"wal-append.queue_time_us":2092470}
I20260709 13:02:04.358872  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1980) took 2568 ms. Trace:
W20260709 13:02:04.353412  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.359357  1590 rpcz_store.cc:276] 0709 13:02:01.790508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.687402 (+896894us) service_pool.cc:224] Handling call
0709 13:02:02.687883 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:02.698022 (+ 10139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:02.698068 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:02.701697 (+  3629us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.740793 (+1039096us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:03.740853 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.741748 (+   895us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.358276 (+616528us) raft_consensus.cc:1685] finished
0709 13:02:04.358325 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.358640 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9612,"raft.queue_time_us":34565,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":20396,"thread_start_us":981635,"threads_started":1,"wal-append.queue_time_us":1417633}
I20260709 13:02:04.356915  1588 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:02:04.375864  1664 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:04.375866  1662 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:04.376019  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 315) took 3798 ms. Trace:
I20260709 13:02:04.385527  1659 rpcz_store.cc:276] 0709 13:02:00.581785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.772467 (+1190682us) service_pool.cc:224] Handling call
0709 13:02:01.798370 (+ 25903us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.798729 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.798779 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.821948 (+ 23169us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.540621 (+1718673us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:03.540662 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.543585 (+  2923us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.293398 (+749813us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:04.375067 (+ 81669us) raft_consensus.cc:1685] finished
0709 13:02:04.375127 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.375658 (+   531us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10262912,"thread_start_us":1718298,"threads_started":1,"wal-append.queue_time_us":2281422}
I20260709 13:02:04.390381  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:04.392405  1666 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.947s	user 0.016s	sys 0.034s
W20260709 13:02:04.393345  1666 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.948s	user 0.016s	sys 0.034s
I20260709 13:02:04.400238  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 307) took 3999 ms. Trace:
W20260709 13:02:04.402333  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.400835  1666 rpcz_store.cc:276] 0709 13:02:00.400943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.670685 (+1269742us) service_pool.cc:224] Handling call
0709 13:02:01.754071 (+ 83386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.779233 (+ 25162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.779281 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.779912 (+   631us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.427720 (+1647808us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:04.398699 (+970979us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.399407 (+   708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.399601 (+   194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.399863 (+   262us) raft_consensus.cc:1685] finished
0709 13:02:04.399880 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.400149 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35779,"prepare.queue_time_us":599872,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":47,"raft.queue_time_us":25665,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":11460,"thread_start_us":2551010,"threads_started":2,"wal-append.queue_time_us":2347900}
I20260709 13:02:04.405395  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:04.406805  1667 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:04.408658  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:04.409881  1667 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:04.413821  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:04.419297  1666 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:04.417820  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:04.428500  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1437) took 3855 ms. Trace:
I20260709 13:02:04.428524  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1444) took 3602 ms. Trace:
I20260709 13:02:04.429294  1740 rpcz_store.cc:276] 0709 13:02:00.825792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.161608 (+335816us) service_pool.cc:224] Handling call
0709 13:02:01.161949 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.162582 (+   633us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.162609 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.163172 (+   563us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.373751 (+2210579us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:03.373796 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.373940 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.427739 (+1053799us) raft_consensus.cc:1685] finished
0709 13:02:04.427793 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.428168 (+   375us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1550,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"thread_start_us":2120215,"threads_started":1,"wal-append.queue_time_us":2758080}
I20260709 13:02:04.429231  1734 rpcz_store.cc:276] 0709 13:02:00.572979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:00.573477 (+   498us) service_pool.cc:224] Handling call
0709 13:02:00.573987 (+   510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.748096 (+1174109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.748133 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.766281 (+ 18148us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.527346 (+1761065us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:03.527388 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.527459 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.427750 (+900291us) raft_consensus.cc:1685] finished
0709 13:02:04.427793 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.428179 (+   386us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86098,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"thread_start_us":2887037,"threads_started":2,"wal-append.queue_time_us":2487030}
I20260709 13:02:04.433947  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2024) took 1099 ms. Trace:
I20260709 13:02:04.435106  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1993) took 3292 ms. Trace:
I20260709 13:02:04.435956  1735 rpcz_store.cc:276] 0709 13:02:01.142601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.383051 (+240450us) service_pool.cc:224] Handling call
0709 13:02:01.383534 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:01.450715 (+ 67181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.450782 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:01.479653 (+ 28871us) log.cc:844] Serialized 26 byte log entry
0709 13:02:03.414615 (+1934962us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:03.414714 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.414814 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.434558 (+1019744us) raft_consensus.cc:1685] finished
0709 13:02:04.434617 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.434864 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":368,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":87883,"thread_start_us":1831436,"threads_started":1,"wal-append.queue_time_us":2748284}
I20260709 13:02:04.434664  1740 rpcz_store.cc:276] 0709 13:02:03.334715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.431646 (+1096931us) service_pool.cc:224] Handling call
0709 13:02:04.432351 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.433206 (+   855us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:04.433259 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.433349 (+    90us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.433407 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.433491 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.433844 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:04.436636  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2030) took 1034 ms. Trace:
I20260709 13:02:04.451058  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1994) took 3298 ms. Trace:
I20260709 13:02:04.480101  1741 rpcz_store.cc:276] 0709 13:02:01.152579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.581818 (+429239us) service_pool.cc:224] Handling call
0709 13:02:01.653390 (+ 71572us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:01.653993 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:01.654041 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:01.690415 (+ 36374us) log.cc:844] Serialized 52 byte log entry
0709 13:02:03.374111 (+1683696us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:04.098694 (+724583us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.098964 (+   270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.099275 (+   311us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.438689 (+339414us) raft_consensus.cc:1685] finished
0709 13:02:04.438737 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.450845 (+ 12108us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":746047,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":5816,"raft.queue_time_us":52318,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":95345,"spinlock_wait_cycles":11008256,"thread_start_us":2263546,"threads_started":2,"wal-append.queue_time_us":2507986}
I20260709 13:02:04.443125  1738 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:04.462409  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 251) took 1739 ms. Trace:
I20260709 13:02:04.483918  2318 rpcz_store.cc:276] 0709 13:02:02.722990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.723139 (+   149us) service_pool.cc:224] Handling call
0709 13:02:02.754414 (+ 31275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:02.766237 (+ 11823us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:02.766268 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:02.791643 (+ 25375us) log.cc:844] Serialized 564 byte log entry
0709 13:02:03.740774 (+949131us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:03.740824 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:03.740905 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.461727 (+720822us) raft_consensus.cc:1685] finished
0709 13:02:04.461767 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.462125 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":948397,"threads_started":1,"wal-append.queue_time_us":1450689}
W20260709 13:02:04.464846  2310 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.001s	user 0.010s	sys 0.006s
I20260709 13:02:04.486805  2310 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:04.487318  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2005) took 2819 ms. Trace:
I20260709 13:02:04.487615  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 570) took 1176 ms. Trace:
I20260709 13:02:04.488066  2310 rpcz_store.cc:276] 0709 13:02:03.311039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.321809 (+ 10770us) service_pool.cc:224] Handling call
0709 13:02:04.487465 (+1165656us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:04.487850  1738 rpcz_store.cc:276] 0709 13:02:01.668379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.942819 (+2274440us) service_pool.cc:224] Handling call
0709 13:02:03.943161 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.484028 (+540867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.486927 (+  2899us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b5400338954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:04.487239 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1135151360}
I20260709 13:02:04.112944  3600 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:04.156782  3805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.019s	sys 0.025s
I20260709 13:02:04.490914  3405 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:04.466460  2311 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:04.489589  1734 rpcz_store.cc:276] 0709 13:02:03.402197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.432548 (+1030351us) service_pool.cc:224] Handling call
0709 13:02:04.433101 (+   553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.436238 (+  3137us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.436274 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.436346 (+    72us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.436384 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.436454 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.436564 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:04.417120  1588 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:04.294504  4069 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.133424  3939 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.586298  4080 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:04.587214  4080 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:04.288120  4105 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.003s
I20260709 13:02:04.138551  3903 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:04.662573  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 300253us
I20260709 13:02:04.579210  3405 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:04.094745  4071 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:04.686086  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.004s	sys 0.011s
I20260709 13:02:04.492094  3600 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:04.699977  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.007s	sys 0.012s
W20260709 13:02:04.700536  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.007s	sys 0.012s
W20260709 13:02:04.697486  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.004s	sys 0.012s
I20260709 13:02:04.690982  4080 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:04.717664  1589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.191s	user 0.018s	sys 0.052s
W20260709 13:02:04.717530  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 704439us
I20260709 13:02:04.725355  4105 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.729044  4105 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.729869  4033 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.730170  1668 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:04.720918  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.851s	user 0.015s	sys 0.015s
I20260709 13:02:04.710345  3939 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.707314  4061 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:04.730548  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.068s	user 0.015s	sys 0.046s
W20260709 13:02:04.730783  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.132s	user 0.016s	sys 0.030s
W20260709 13:02:04.737432  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.015s	sys 0.016s
W20260709 13:02:04.733744  1589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.019s	sys 0.052s
W20260709 13:02:04.722123  4031 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.125s	user 0.017s	sys 0.031s
W20260709 13:02:04.748664  4031 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.211s	user 0.017s	sys 0.032s
W20260709 13:02:04.738755  4130 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.020s	sys 0.036s
W20260709 13:02:04.749756  4130 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.020s	sys 0.037s
W20260709 13:02:04.750128  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.004s	sys 0.015s
W20260709 13:02:04.750679  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.004s	sys 0.015s
W20260709 13:02:04.751237  3886 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.233s	user 0.018s	sys 0.001s
I20260709 13:02:04.751921  3886 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.738776  4119 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:04.745751  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.015s	sys 0.000s
W20260709 13:02:04.746351  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.141s	user 0.015s	sys 0.046s
I20260709 13:02:04.747864  4069 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:04.708272  4118 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:04.722294  4096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.128s	user 0.030s	sys 0.037s
I20260709 13:02:04.741540  1665 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:04.745834  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:04.753780  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.215s	user 0.017s	sys 0.030s
W20260709 13:02:04.762633  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.016s	sys 0.001s
I20260709 13:02:04.730316  4155 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.082s
W20260709 13:02:04.766247  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.006s	sys 0.010s
I20260709 13:02:04.774995  4118 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.778264  4185 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.779062  4185 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.740185  4156 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
W20260709 13:02:04.779055  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:04.784178  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.007s	sys 0.011s
W20260709 13:02:04.772819  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.005s	sys 0.015s
I20260709 13:02:04.723639  3405 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:04.782285  4096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.248s	user 0.031s	sys 0.038s
I20260709 13:02:04.790411  1660 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:04.791237  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:04.788772  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.006s	sys 0.015s
I20260709 13:02:04.753962  4063 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:04.793052  3886 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:04.789386  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2011) took 2023 ms. Trace:
I20260709 13:02:04.732219  4154 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260709 13:02:04.721124  4056 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.136s
I20260709 13:02:04.793704  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:04.799784  1584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.000s	sys 0.022s
W20260709 13:02:04.804822  1736 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.966s	user 0.013s	sys 0.047s
W20260709 13:02:04.811861  1736 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.063s	user 0.014s	sys 0.048s
W20260709 13:02:04.799409  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 669707us
W20260709 13:02:04.799476  4045 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.043s	sys 0.025s
W20260709 13:02:04.815088  1584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.001s	sys 0.023s
I20260709 13:02:04.740701  3815 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
W20260709 13:02:04.814023  1557 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.976s	user 0.008s	sys 0.020s
I20260709 13:02:04.818742  4156 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:04.819141  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2004) took 1690 ms. Trace:
W20260709 13:02:04.811859  1737 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.013s	sys 0.012s
I20260709 13:02:04.799683  1742 rpcz_store.cc:276] 0709 13:02:02.681134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:04.116927 (+1435793us) service_pool.cc:224] Handling call
0709 13:02:04.436691 (+319764us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.437516 (+   825us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:04.438133 (+   617us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.702576 (+264443us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.702660 (+    84us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.702731 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.702834 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.704732 (+  1898us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":91329,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":12895,"raft.queue_time_us":48636,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":142,"thread_start_us":258118,"threads_started":1}
W20260709 13:02:04.822825  4123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.009s	sys 0.016s
W20260709 13:02:04.821910  4163 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.091s	user 0.002s	sys 0.000s
W20260709 13:02:04.826496  1557 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.079s	user 0.008s	sys 0.020s
W20260709 13:02:04.827734  1737 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.013s	sys 0.013s
I20260709 13:02:04.826988  4069 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.824011  1589 rpcz_store.cc:276] 0709 13:02:03.128565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.459750 (+331185us) service_pool.cc:224] Handling call
0709 13:02:03.460109 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.460611 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.460786 (+   175us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:03.461379 (+   593us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.783014 (+1321635us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:04.797372 (+ 14358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.810564 (+ 13192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.813843 (+  3279us) raft_consensus.cc:1685] finished
0709 13:02:04.813871 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.819072 (+  5201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":676,"raft.queue_time_us":327,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":12550,"thread_start_us":1293233,"threads_started":1,"wal-append.queue_time_us":1288367}
W20260709 13:02:04.829839  4045 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.044s	sys 0.026s
I20260709 13:02:04.831074  4045 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:04.832340  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.22861e+06us
I20260709 13:02:04.832571  4096 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.833407  4096 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.837152  4130 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:04.837723  4130 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:04.839262  1738 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:04.839022  4130 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.840965  1733 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:04.841162  1741 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:04.842026  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.842347  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.842816  1741 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:04.843559  4130 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.844125  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.845641  4045 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.849922  4069 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.844774  4160 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 13:02:04.822333  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:04.825359  4123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.009s	sys 0.016s
I20260709 13:02:04.815650  4118 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:04.817236  4185 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.853212  4069 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:04.852319  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:04.879642  3951 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.054s	user 0.019s	sys 0.000s
I20260709 13:02:04.879751  3805 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:04.880694  3805 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:04.880693  3951 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.882164  3805 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.883033  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 1979) took 3098 ms. Trace:
I20260709 13:02:04.885437  4071 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:04.885594  1584 rpcz_store.cc:276] 0709 13:02:01.784825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.674149 (+889324us) service_pool.cc:224] Handling call
0709 13:02:02.674521 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:03.766287 (+1091766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:03.766333 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:03.767142 (+   809us) log.cc:844] Serialized 52 byte log entry
0709 13:02:04.882526 (+1115384us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.882619 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.882730 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.882780 (+    50us) raft_consensus.cc:1685] finished
0709 13:02:04.882792 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.882967 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22833,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":10387,"thread_start_us":1923854,"threads_started":2,"wal-append.queue_time_us":931120}
I20260709 13:02:04.869223  4123 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:04.829438  4033 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.889772  4123 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.890044  4033 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.893003  4033 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.884447  4063 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:04.836485  4096 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.893759  4063 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:04.894713  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 861643us
W20260709 13:02:04.884887  3907 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.892s	user 0.014s	sys 0.002s
W20260709 13:02:04.881970  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 126487us
I20260709 13:02:04.898119  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1988) took 4141 ms. Trace:
I20260709 13:02:04.898001  3805 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:04.905144  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2002) took 3237 ms. Trace:
I20260709 13:02:04.904637  3907 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:04.906153  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2013) took 1217 ms. Trace:
I20260709 13:02:04.906625  1668 rpcz_store.cc:276] 0709 13:02:03.689102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.689300 (+   198us) service_pool.cc:224] Handling call
0709 13:02:04.905835 (+1216535us) spinlock_profiling.cc:243] Waited 989 ms on lock 0x7b540013e2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:04.906035 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2275680640}
I20260709 13:02:04.907440  3907 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:04.907598  4172 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 13:02:04.904623  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:04.910429  4054 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:04.911736  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 569) took 2130 ms. Trace:
I20260709 13:02:04.912410  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:04.913326  1740 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:02:04.915663  4166 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 13:02:04.921003  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.905593  1736 rpcz_store.cc:276] 0709 13:02:01.667285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.741305 (+2074020us) service_pool.cc:224] Handling call
0709 13:02:03.742023 (+   718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.904539 (+1162516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:04.904596 (+    57us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.904665 (+    69us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:04.904703 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.904827 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.905068 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4942,"raft.queue_time_us":151822,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":45828,"thread_start_us":1067743,"threads_started":1}
I20260709 13:02:04.945261  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 1999) took 3702 ms. Trace:
I20260709 13:02:04.946301  1739 rpcz_store.cc:276] 0709 13:02:01.242984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.535322 (+2292338us) service_pool.cc:224] Handling call
0709 13:02:03.535720 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:03.537919 (+  2199us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:03.537950 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:03.538473 (+   523us) log.cc:844] Serialized 26 byte log entry
0709 13:02:04.822604 (+1284131us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:04.843676 (+ 21072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.843779 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:04.942029 (+ 98250us) spinlock_profiling.cc:243] Waited 9.46 ms on lock 0x7b1800482c80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:04.944879 (+  2850us) raft_consensus.cc:1685] finished
0709 13:02:04.944911 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.945190 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1587,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":90,"spinlock_wait_cycles":21748480,"thread_start_us":1230768,"threads_started":1,"wal-append.queue_time_us":1239572}
W20260709 13:02:04.952961  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48538.6us
I20260709 13:02:04.953797  1739 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 6. Ops: []
W20260709 13:02:04.941413  4169 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.195s	user 0.003s	sys 0.000s
I20260709 13:02:04.926250  3405 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.955513  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.928610  1733 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:04.959240  1742 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.4-3.4]
W20260709 13:02:04.898546  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76824us
I20260709 13:02:04.961695  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:04.964033  3939 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:04.964548  3939 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:04.898033  3951 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:04.966116  3951 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:04.966560  3951 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:04.966919  3951 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:04.930048  4183 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:04.970077  4183 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.970598  1739 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:02:04.971825  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.972518  4183 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:04.973121  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1449) took 4407 ms. Trace:
I20260709 13:02:04.932904  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:04.975263  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:04.976599  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:04.932685  4154 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:04.934031  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 320) took 1333 ms. Trace:
I20260709 13:02:04.970067  4161 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:04.912094  2311 rpcz_store.cc:276] 0709 13:02:02.781430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:02.781643 (+   213us) service_pool.cc:224] Handling call
0709 13:02:04.465771 (+1684128us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.791940 (+326169us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:04.791977 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.792027 (+    50us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.792068 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.792188 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.911504 (+119316us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400719650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:04.911651 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24954,"raft.queue_time_us":4147,"raft.run_cpu_time_us":513,"raft.run_wall_time_us":75713,"spinlock_wait_cycles":3833694080}
I20260709 13:02:04.903052  1737 rpcz_store.cc:276] 0709 13:02:00.756573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.154596 (+398023us) service_pool.cc:224] Handling call
0709 13:02:01.154983 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:01.155217 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:03.382000 (+2226783us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.103925 (+721925us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.820756 (+716831us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:04.866391 (+ 45635us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.897658 (+ 31267us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:04.897770 (+   112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.897894 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.898051 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2940691,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":166,"raft.queue_time_us":199248,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":198,"spinlock_wait_cycles":54672256,"thread_start_us":3635388,"threads_started":3,"wal-append.queue_time_us":1322497}
I20260709 13:02:04.978302  4161 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:04.979691  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:04.980739  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:04.936038  4189 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:04.977835  1660 rpcz_store.cc:276] 0709 13:02:03.600765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.604195 (+  3430us) service_pool.cc:224] Handling call
0709 13:02:03.604646 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:04.808292 (+1203646us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.933712 (+125420us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:04.933949 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2643589504}
I20260709 13:02:04.983644  1663 rpcz_store.cc:276] 0709 13:02:00.566167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:01.769639 (+1203472us) service_pool.cc:224] Handling call
0709 13:02:01.769983 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:02.506782 (+736799us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:03.946480 (+1439698us) log.cc:844] Serialized 14 byte log entry
0709 13:02:04.751384 (+804904us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:04.751545 (+   161us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:04.751674 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:04.972785 (+221111us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:04.972834 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:04.973052 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":417822,"prepare.queue_time_us":1520589,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":423639,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":31899,"spinlock_wait_cycles":732349312,"thread_start_us":2114417,"threads_started":2,"wal-append.queue_time_us":764947}
I20260709 13:02:04.981233  1737 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:04.955250  4177 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:02:04.939786  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:04.923877  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2014) took 1233 ms (client timeout 1540 ms). Trace:
I20260709 13:02:04.982013  4189 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:04.936539  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.18986e+06us
I20260709 13:02:04.966418  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:04.934707  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:04.979898  4154 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:04.987234  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:04.981058  4161 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:04.989717  1740 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:04.991128  1734 raft_consensus.cc:2376] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4. Re-sending same reply.
W20260709 13:02:04.987075  1665 rpcz_store.cc:269] 0709 13:02:03.690157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.690361 (+   204us) service_pool.cc:224] Handling call
0709 13:02:04.923261 (+1232900us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:04.923773 (+   512us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1794799872}
W20260709 13:02:05.001921  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.003433  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.005890  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:05.007584  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.008762  4156 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election won for term 5
W20260709 13:02:05.009816  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.010555  1733 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:05.013386  1741 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:05.015209  1737 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:05.009305  4156 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:05.023281  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.026196  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.024294  1735 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:05.020967  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.031710  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.010180  4177 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:05.034545  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 137447us
I20260709 13:02:05.035238  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.013396  4056 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:05.040987  1739 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:05.039444  1736 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:02:05.046739  1742 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:05.046911  1737 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:05.034183  4172 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:04.966364  3939 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.060187  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.063169  1734 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 13:02:05.081614  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.082739  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.083871  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101917us
I20260709 13:02:05.072088  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.089825  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:05.090939  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.091249  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.092406  1663 raft_consensus.cc:2410] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:05.092563  1665 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5.
I20260709 13:02:05.093925  3600 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:05.094293  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.095206  1733 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:05.094348  1660 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:05.096109  1660 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.097432  1660 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.099936  1660 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:05.103839  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.104521  1742 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:05.085491  1739 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.059605  4056 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:05.125612  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:05.125854  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 79955.4us
I20260709 13:02:05.130784  1736 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:05.130681  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.132635  1741 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.134831  3600 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:05.135432  3600 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.136493  3600 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.133123  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.125334  3805 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
W20260709 13:02:05.139595  4035 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.884s	user 0.009s	sys 0.009s
I20260709 13:02:05.140440  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.141201  1736 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:05.144770  4045 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.145295  4209 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 13:02:05.145483  4045 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.146346  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:05.148345  4045 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:05.114223  4172 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.150465  4172 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.151258  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.152484  1665 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:02:05.153815  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.154388  4056 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.155543  1663 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:05.131500  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.159946  1740 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:05.138389  1737 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:05.166481  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73297.7us
I20260709 13:02:05.168715  4209 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:05.169472  4209 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:05.097589  3405 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 13:02:05.173970  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:05.171489  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48160us
I20260709 13:02:05.170614  4209 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.178181  4209 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:05.181113  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.182437  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.129205  4156 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.181915  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:05.187148  1734 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:05.187053  3903 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:05.196246  1736 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:05.193096  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 228991us
I20260709 13:02:05.137841  3951 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:05.201656  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 112580us
I20260709 13:02:05.192827  3903 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.168597  4155 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:05.241808  4204 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 13:02:05.253764  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2025) took 1485 ms. Trace:
I20260709 13:02:05.254536  1661 rpcz_store.cc:276] 0709 13:02:03.768058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:03.881250 (+113192us) service_pool.cc:224] Handling call
0709 13:02:05.253168 (+1371918us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540013ddd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:05.253429 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2621585280}
I20260709 13:02:05.263037  4189 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.269958  4189 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:05.282320  1708 leader_election.cc:336] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.672s (SENT)
W20260709 13:02:05.283895  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51207.9us
I20260709 13:02:05.286979  4204 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:05.287475  4204 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.288977  3951 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:05.193301  4045 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
W20260709 13:02:05.298491  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.283078  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:05.302918  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:05.309331  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 71607.7us
I20260709 13:02:05.308254  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:05.288777  4204 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.224315  4177 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:05.373412  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52575.4us
I20260709 13:02:05.308787  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.370594  1739 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:05.372434  1742 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:05.320163  4045 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:05.379673  1660 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:02:05.375308  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136872us
I20260709 13:02:05.377187  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.397069  2311 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:05.401780  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88327.7us
W20260709 13:02:05.393322  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.353685  4119 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:05.280416  4155 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:05.408555  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 185883us
I20260709 13:02:05.303670  4205 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.214s
I20260709 13:02:05.371335  3405 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:05.388918  4130 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.545s	user 0.014s	sys 0.003s
I20260709 13:02:05.254488  3903 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.377066  3600 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.430903  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:05.431397  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 62645us
W20260709 13:02:05.434657  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 152564us
I20260709 13:02:05.436499  1737 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:05.439142  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.430903  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.444347  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.443686  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.441268  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.454582  1663 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.454743  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.456550  1660 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:05.457479  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:05.458747  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.455538  1664 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:05.460062  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:05.461151  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.463804  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:05.464006  4045 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:05.463917  1666 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:05.464671  2311 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:05.464598  4045 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:05.465123  4045 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:05.468549  1734 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:05.469991  1665 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:05.437167  4205 consensus_queue.cc:1243] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:05.459388  4195 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:05.478174  4195 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.479604  4195 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:05.462154  1708 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.743s (SENT)
W20260709 13:02:05.481442  1708 leader_election.cc:336] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.758s (SENT)
I20260709 13:02:05.443439  4210 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.238s
W20260709 13:02:05.472082  1706 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 2.882s (SENT) [suppressed 42 similar messages]
W20260709 13:02:05.482560  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 146661us
I20260709 13:02:05.488727  3903 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.490218  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.491403  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.492810  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.494278  1586 pending_rounds.cc:85] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:02:05.494647  1586 pending_rounds.cc:107] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 13:02:05.494917  1586 raft_consensus.cc:2889] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:05.498097  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.499253  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:05.483693  1557 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.876s (SENT)
I20260709 13:02:05.504894  1740 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:05.506556  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:05.454149  3405 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.472991  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:05.526531  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.526759  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.531024  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.527244  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:05.534840  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61954.1us
I20260709 13:02:05.542587  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:05.541893  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92103.3us
W20260709 13:02:05.526407  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45559.8us
I20260709 13:02:05.551252  1664 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:05.535562  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.539088  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.533408  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.534029  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.563359  1663 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:05.563834  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.564527  1590 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:05.456260  3600 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:05.531518  4063 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.637s	user 0.015s	sys 0.000s
I20260709 13:02:05.534636  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.535981  4161 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:05.528258  4123 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:05.563992  1661 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:05.572120  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:05.529613  4054 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:05.537063  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:05.575196  1659 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:05.581730  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:05.531270  4119 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:05.539264  4180 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.594257  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:05.595544  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48283.1us
I20260709 13:02:05.598335  2316 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:05.597046  4123 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.605939  4119 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:05.606637  4119 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.606676  4123 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.598431  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.604962  2317 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:05.612833  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 131450us
I20260709 13:02:05.618113  1667 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.618791  2312 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:05.619239  1663 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:05.572846  4096 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:05.623821  1660 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:05.620271  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.627067  1668 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:05.630772  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:05.633991  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69679.5us
I20260709 13:02:05.635322  2315 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:05.636864  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73951.7us
W20260709 13:02:05.638110  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.639393  4033 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:05.640136  4033 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.641345  4033 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.643160  4210 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:05.644039  4210 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.644716  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.645439  1588 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:05.648690  1660 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 13:02:05.626060  1662 raft_consensus.cc:2376] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7. Re-sending same reply.
I20260709 13:02:05.680991  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.586432  4063 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.674901  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.528054  4155 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:05.693972  1589 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:05.693804  1632 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.611132  4180 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.594902  4054 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.646572  4096 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.633165  4203 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 13:02:05.713824  1663 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:05.700179  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.698901  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:05.698679  1661 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:02:05.709532  1666 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:05.699519  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:05.735409  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 97728.9us
I20260709 13:02:05.736294  1584 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:02:05.736586  1587 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:05.699743  1662 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:05.767216  1590 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:05.718777   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:05.770231  1667 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:05.772907  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:05.766201  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54798.5us
W20260709 13:02:05.703073  4212 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.239s	user 0.004s	sys 0.000s
W20260709 13:02:05.784927  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48660.1us
W20260709 13:02:05.768386  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 196769us
I20260709 13:02:05.702837  4045 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.778132  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:05.778944  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.780771  4203 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:05.781253  4214 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 13:02:05.790902  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.786849  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:05.787834  1740 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:05.704191  3405 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:05.726953  3600 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:05.790535  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.787166  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 187259us
I20260709 13:02:05.793653  4096 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.797317  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:05.765933  4031 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:05.776276  4035 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:05.807181  4203 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.825845  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.832355  4054 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:05.832487  4203 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.834606  4063 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:05.835639  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 138539us
I20260709 13:02:05.802197  4181 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:05.802556  1661 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:05.836410  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:05.838116  1661 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260709 13:02:05.799605  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.802452  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.804065  4045 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:05.705251  4210 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:05.841281  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.833060  4054 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:02:05.851686  1740 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:02:05.834477  1557 connection.cc:256] RPC call timeout handler was delayed by -1.5020733549999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 13:02:05.869575  1557 connection.cc:256] RPC call timeout handler was delayed by -1.144488431999946s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
I20260709 13:02:05.842327  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:05.851186  1592 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:02:05.830201  4219 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 13:02:05.841420  1661 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:02:05.870211  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:05.830068  1584 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:05.831100  4180 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:05.897696  1588 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:05.849845  4031 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.902572  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.904071  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.851145  4035 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.868958  4181 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:05.873289  4063 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:05.907428  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:05.912612  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.927603  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.928452  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.929462  1588 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:05.930374  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.931545  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.932726  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.933990  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.934980  1587 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:05.936519  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.936257  4181 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.937548  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.939729  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.940516  1586 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:05.943361  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:05.944442  1593 raft_consensus.cc:2376] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6. Re-sending same reply.
I20260709 13:02:05.947973  4035 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:05.949695  1592 raft_consensus.cc:2376] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5. Re-sending same reply.
I20260709 13:02:05.938587  3939 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:05.900223  4161 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:05.952101  4161 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:05.953208  4161 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:05.938511  1584 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:05.937179  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:05.966156  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 334978us
I20260709 13:02:05.966581  1588 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:05.937192  1589 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:02:05.971997  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.931402  4119 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:05.957719  4063 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:05.965775  4187 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 13:02:05.982812  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 343994us
W20260709 13:02:05.986100  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:05.962436  4226 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:05.995602  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.997009  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:05.998502  4063 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.015051  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.016052  1584 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:06.017418  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:05.994712  1661 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:06.021999  1592 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:06.029423  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 187174us
W20260709 13:02:06.030509  1708 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.828s (SENT)
I20260709 13:02:05.972216  4210 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.055s
I20260709 13:02:06.007063  4226 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.035732  1664 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:06.031833  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.063627  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:06.064226  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46957.5us
W20260709 13:02:06.061050  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 264798us
I20260709 13:02:06.075335  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.080605  1589 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:06.087312  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 300554us
I20260709 13:02:06.051734  4226 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:06.031294  4209 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 13:02:06.093457  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:06.093757  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:06.094403  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.093475  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.094936  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:06.097187  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.097514  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.106004  1740 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:06.097689  4189 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:06.052141  4156 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:02:06.100878  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:06.101684  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.106942  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:06.098709  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:06.108808  1739 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:06.119596  1736 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:06.110189  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:06.110189  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.120939  1584 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:06.111944  1592 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:06.112176  2317 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 3
I20260709 13:02:06.113011  1593 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:06.113708  1740 pending_rounds.cc:85] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 2
I20260709 13:02:06.088147  4231 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:06.122530  1740 pending_rounds.cc:107] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:02:06.123754  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:06.123940  1740 raft_consensus.cc:2889] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:06.123431  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:06.124063  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.122102  2317 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20260709 13:02:06.130014  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.131502  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:06.132547  2317 raft_consensus.cc:2917] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20260709 13:02:06.133057  2317 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:06.142563  1586 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:06.038175  4031 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:06.173797  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.174886  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77497.4us
I20260709 13:02:06.176111  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.178583  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:06.182607  1586 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 2: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:06.179427  2315 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:06.194509  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.195719  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.196779  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:02:06.220611  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.247267  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48148.1us
W20260709 13:02:06.262084  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 77333.3us
I20260709 13:02:06.266916  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:06.268354  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 193727us
I20260709 13:02:06.185815  4035 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.288302  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:06.266646  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.302651  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:06.322054  1556 leader_election.cc:400] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5.
I20260709 13:02:06.250260  4031 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:06.340337  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73805.6us
I20260709 13:02:06.344170  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.345702  1584 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:06.246898  4239 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:06.351717  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:06.249818  4227 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:06.301699  4054 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:06.358006  2314 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:06.306185  4238 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:06.360224  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.360184  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 337508us
I20260709 13:02:06.368201  1586 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:06.351080  4180 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:06.373490  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110341us
I20260709 13:02:06.358460  4227 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.385208  1556 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:02:06.385726  1556 leader_election.cc:403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:06.365939  3805 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.367326  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:06.371073  4237 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:06.402614  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 208204us
W20260709 13:02:06.410028  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65979.7us
I20260709 13:02:06.376252  4180 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.368206  4155 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:06.375013  4234 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:06.420145  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:06.415493  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.945s (SENT)
I20260709 13:02:06.352715  4035 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.422803  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.388572  4156 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:06.387281  4054 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:06.389348  4227 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:06.430226  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.424348  4180 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:06.435735  4054 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:06.463681  4054 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:06.464996  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:06.466131  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46140.6us
I20260709 13:02:06.468683  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:06.470217  4156 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:06.471213  4156 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:06.377600  4238 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.474490  1586 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:06.475006  1586 pending_rounds.cc:85] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:02:06.475277  1586 pending_rounds.cc:107] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:06.475557  1586 raft_consensus.cc:2889] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:06.476634  4238 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:06.381352  4239 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20260709 13:02:06.493268  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.432305  4209 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:06.477229  1586 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260709 13:02:06.477800  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 76492.5us
I20260709 13:02:06.412081  3805 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.518414  4210 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:06.518936  4210 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:06.519840  3805 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:06.457973  4240 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.087s
I20260709 13:02:06.523741  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:06.527969  3939 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.577s	user 0.007s	sys 0.007s
I20260709 13:02:06.477314  4238 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.529413  4238 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.530052  1589 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:06.507395  1665 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:06.519928  4210 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:06.513651  4230 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.334s	user 0.001s	sys 0.001s
I20260709 13:02:06.461156  4063 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:06.479100  4254 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:06.478032  4236 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.298s	user 0.001s	sys 0.000s
I20260709 13:02:06.533424  4035 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:06.567767  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:06.541719  4242 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:02:06.536012  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:06.482337  4249 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:06.549324  4262 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:06.556857  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 351) took 1066 ms. Trace:
I20260709 13:02:06.417966  4237 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:06.557778  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49445.3us
I20260709 13:02:06.559047  4180 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.592203  1666 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:06.560456  4189 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:06.594448  4189 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:06.596060  4180 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:06.561848  4209 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:02:06.597113  4235 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.435248  4204 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:06.597882  4235 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.600172  4235 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:06.532202  4238 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:06.516626  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:06.569967  1661 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:06.585633  1590 rpcz_store.cc:276] 0709 13:02:05.490197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.492499 (+  2302us) service_pool.cc:224] Handling call
0709 13:02:06.556587 (+1064088us) spinlock_profiling.cc:243] Waited 71.1 ms on lock 0x7b540031db54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:06.556745 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163551104}
I20260709 13:02:06.566054  4063 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:06.595804  4189 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:06.562074  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 174322us
I20260709 13:02:06.611318  1661 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:06.596076  1662 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:06.585736  4239 consensus_queue.cc:1243] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:06.611323  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.628150  1663 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:02:06.630861  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:06.602447  4262 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:02:06.626410  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.633543  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.612905  4210 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:06.611568  4063 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:06.637683  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.002s	sys 0.013s
W20260709 13:02:06.639984  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.995s (SENT)
W20260709 13:02:06.641319  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.002s	sys 0.013s
W20260709 13:02:06.642192  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 560892us
I20260709 13:02:06.646183  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.644421  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:06.652071  1666 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:06.656158  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.641875  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:06.651453  4209 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:06.652734  3405 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:06.677026  1664 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:06.679060  2310 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:06.706043  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:06.708525  4267 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 13:02:06.713860  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73929us
I20260709 13:02:06.715452  4183 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:06.716150  4183 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:06.719259  1666 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:06.719848  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49646.4us
I20260709 13:02:06.667346  4189 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:06.679418  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:06.724974  1557 leader_election.cc:400] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:02:06.728677  1557 leader_election.cc:403] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:06.717285  4183 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.668588  4239 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:06.732169  4240 consensus_queue.cc:1243] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:06.657444  4255 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:06.675562  4209 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:06.752552  4209 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:06.744556  4259 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:02:06.746070  3405 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:06.751600  4255 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:06.674865  3939 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260709 13:02:06.674960  3805 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
W20260709 13:02:06.774578  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101338us
I20260709 13:02:06.730340  4155 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 13:02:06.760205  4123 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.763912  4255 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:06.814083  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 89323.8us
I20260709 13:02:06.764338  4240 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:02:06.826537  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 219237us
W20260709 13:02:06.809219  4248 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.083s	user 0.004s	sys 0.002s
I20260709 13:02:06.776136  4054 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.890332  4271 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:06.890990  4271 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:06.921941  4268 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.248s	user 0.001s	sys 0.000s
I20260709 13:02:06.922106  4249 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:06.922896  4249 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:06.924003  4249 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.926076  4231 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:06.926571  4231 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:06.927634  4231 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:06.929087  4231 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:06.930624  4265 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
W20260709 13:02:06.932366  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102890us
I20260709 13:02:06.933909  1705 leader_election.cc:400] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:06.934331  1705 leader_election.cc:403] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:06.884469  4269 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:06.938298  4269 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:06.880846  4270 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:06.880288  4257 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.280s
I20260709 13:02:06.951884  2310 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:06.846904  4250 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:06.964650  4249 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:06.816869  4123 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:06.898492  4226 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.765s	user 0.010s	sys 0.005s
I20260709 13:02:06.941248  4269 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:07.017704  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 189316us
W20260709 13:02:07.038693  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 318113us
I20260709 13:02:06.905359  4237 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.041464  4156 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6.
I20260709 13:02:07.018625  4226 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.041476  4238 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:07.053855  4270 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
W20260709 13:02:07.072147  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 138506us
I20260709 13:02:07.042104  4231 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:07.066242  3600 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:07.037134  4250 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:07.068652  4282 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:07.056464  4271 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:07.085386  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.062534  3939 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:07.075317  4277 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.110936  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 259864us
W20260709 13:02:07.109234  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 394417us
I20260709 13:02:07.101471  2313 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:07.090099  4282 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:07.130963  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.093743  4156 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.136475  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0726224090000187s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16290000000000002s for a response
W20260709 13:02:07.132656  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.132545  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:07.089058  3600 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:02:07.138034  2310 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 13:02:07.140173  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.141252  4283 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.141917  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.141825  4283 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.143304  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:07.125031  4031 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.143949  1733 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:07.144057  4031 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.144754  4281 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.145289  4156 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.145313  4281 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.147601  4281 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:07.153296  1708 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.689s (SENT)
I20260709 13:02:07.093048  4155 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 13:02:07.155460  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:07.158573  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.075912  4238 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.100169  3939 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:07.174400  1740 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:07.081045  4231 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.180760  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.181704  1737 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:07.134316  4271 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.147634  4156 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:07.180510  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 287049us
I20260709 13:02:07.158232  4281 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.144140  4283 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.217867  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:07.186815  4227 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.690s	user 0.001s	sys 0.013s
I20260709 13:02:07.208376  1705 leader_election.cc:400] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:07.218653  1705 leader_election.cc:403] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 13:02:07.196919  1708 leader_election.cc:336] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.851s (SENT)
I20260709 13:02:07.155033  4257 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:02:07.186504  3600 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.126417  4282 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:07.234771  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.197597  4054 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:07.129375  4277 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.241293  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.241976  1740 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 1.
W20260709 13:02:07.239327  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51597.3us
I20260709 13:02:07.191546  4155 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 13:02:07.231197  4250 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
W20260709 13:02:07.250205  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61521.3us
I20260709 13:02:07.195312  4271 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:07.240518  4210 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.597s	user 0.014s	sys 0.000s
I20260709 13:02:07.224042  4183 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 13:02:07.266366  1733 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:07.218145  4196 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.197s
I20260709 13:02:07.221942  4281 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.247714  4282 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
W20260709 13:02:07.277209  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83348.4us
I20260709 13:02:07.223912  4294 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:02:07.252976  4123 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.226200  4227 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:07.280851  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91167.6us
W20260709 13:02:07.290105  4279 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260709 13:02:07.290843  4288 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 13:02:07.293015  4271 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:07.294174  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47657.3us
I20260709 13:02:07.242998  3600 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:07.311517  1557 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.608s (SENT)
I20260709 13:02:07.316411  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.312891  1740 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 13:02:07.314549  1733 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260709 13:02:07.241812  3939 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.311479  2310 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:07.321664  1737 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:07.323407  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:07.367075  3939 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.274960  4210 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20260709 13:02:07.309939  4189 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.575s	user 0.005s	sys 0.009s
I20260709 13:02:07.327436  4292 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:02:07.338953  4277 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:07.344281  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2006) took 1247 ms. Trace:
I20260709 13:02:07.345007  4096 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.347028  4272 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.349997  4284 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:07.351643  4234 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:07.341228  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.355597  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.356722  4283 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:07.308915  4226 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:07.375722  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54461.5us
I20260709 13:02:07.380657  1589 rpcz_store.cc:276] 0709 13:02:06.096727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.096902 (+   175us) service_pool.cc:224] Handling call
0709 13:02:07.343965 (+1247063us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b540042e354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:07.344175 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":974376192}
W20260709 13:02:07.325102  4255 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.501s	user 0.012s	sys 0.000s
I20260709 13:02:07.325681  3600 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.399426  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.425786  1740 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:07.401716  4281 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.413616  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.395254  4284 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:07.414253  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 88933.4us
I20260709 13:02:07.377588  4227 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:07.397275  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.106s (SENT)
I20260709 13:02:07.454572  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.454989  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.394747  4155 consensus_queue.cc:1243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:07.404860  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.422005  4210 consensus_queue.cc:1243] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 7.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:07.402005  4272 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.422883  1741 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:07.424661  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:07.399652  1667 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:07.411059  4231 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.426853  1736 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:07.462160  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.379202  4031 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.459980  1664 pending_rounds.cc:85] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 2
I20260709 13:02:07.464516  1666 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 13:02:07.402144  4283 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:07.403652  4203 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.452s	user 0.019s	sys 0.000s
I20260709 13:02:07.456350  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.407219  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:07.419013  4189 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:07.462813  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:07.464535  1664 pending_rounds.cc:107] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:07.478911  1664 raft_consensus.cc:2889] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:07.402693  4234 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:07.412771  4297 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20260709 13:02:07.464321  4254 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.888s	user 0.011s	sys 0.005s
I20260709 13:02:07.468014  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.469385  1741 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:07.471591  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:07.397866  4096 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.472894  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:07.484786  1665 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:07.474463  1661 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:07.458487  4226 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:07.475114  4203 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.523s	user 0.023s	sys 0.016s
I20260709 13:02:07.488600  1736 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:02:07.472942  4292 consensus_queue.cc:1243] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:02:07.494107  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:07.492988  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116299us
W20260709 13:02:07.490612  4209 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.736s	user 0.002s	sys 0.013s
I20260709 13:02:07.472114  4155 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:02:07.429065  4255 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.490332  4231 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:07.498306  4294 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:07.512816  1708 leader_election.cc:336] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.280s (SENT)
I20260709 13:02:07.513825  1736 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260709 13:02:07.501483  4227 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.495268  4299 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.527565  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.528407  1591 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:07.516990  1741 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:07.517284  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:07.518545  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:07.519420  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 114813us
I20260709 13:02:07.493155  4254 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:07.524302  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.529090  1741 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:07.509550  4189 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:07.511355  4209 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.546310  4303 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.547027  4303 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:07.530791  4227 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.549033  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.524106  4294 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.551800  1589 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:02:07.551909  4294 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:07.539052  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:07.539428  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:07.566449  1585 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:07.576237  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:07.609555  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:07.572590  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2145) took 1766 ms (client timeout 1945 ms). Trace:
I20260709 13:02:07.603854  1741 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.529942  4130 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
W20260709 13:02:07.618952  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 70769.5us
I20260709 13:02:07.572201  4303 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.530493  4283 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.603298  4037 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:02:07.626245  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 113508us
W20260709 13:02:07.631349  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.636688  4231 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:07.638334  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:07.613209  1585 rpcz_store.cc:269] 0709 13:02:05.806124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.826154 (+ 20030us) service_pool.cc:224] Handling call
0709 13:02:07.490480 (+1664326us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:07.572521 (+ 82041us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34523520}
I20260709 13:02:07.640983  4231 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:07.644510  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:07.649049  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 118368us
I20260709 13:02:07.649818  4291 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:07.651991  4272 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.573424  4254 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:07.614136  4226 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.652624  4272 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.673357  1741 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:07.700300  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.567267  4189 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.672047  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.661180  4282 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:07.662019  4254 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:07.714919  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70663.8us
I20260709 13:02:07.694655  4203 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.706250  4272 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:07.723170  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:07.725770  4203 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.726455  4282 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.728111  4203 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.729571  4254 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:07.734266  4096 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:07.738016  4063 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.087s	user 0.012s	sys 0.005s
I20260709 13:02:07.738548  4063 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.740772  4255 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.717730  4227 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.745929  4231 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:07.712459  4283 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:07.725258  4272 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.729264  4282 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:07.753772  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:07.720635  4305 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:02:07.757381  1734 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:07.714368  4037 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:07.762095  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.749773  4180 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.770025  1741 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:07.773725  4180 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:07.779568  4302 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:07.780108  4302 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:07.783607  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:07.784559  1739 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:07.750613  4063 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:07.792944  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:07.798291  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.799134  1588 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.785557  4227 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:07.801893  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 129946us
I20260709 13:02:07.802390  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.803076  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.803699  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.805691  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:07.811893  4037 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:07.812533  4037 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:07.812206  4231 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:07.813061  4037 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.813205  4231 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:07.816303  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:07.532862  4299 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.818684  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:07.798269  4312 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:07.819849  4312 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.821138  1667 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:07.821408  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:07.822184  2310 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEARNER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:07.805243  4196 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:07.822750  4196 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:07.823164  4196 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:07.824604  4274 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:07.825028  4243 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.449s	user 0.009s	sys 0.004s
I20260709 13:02:07.825235  4274 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:07.826216  4274 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:07.826864  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:07.827667  1634 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.679s (SENT)
I20260709 13:02:07.827888  4130 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:07.828168  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:07.829377  4130 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:07.830711  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:07.832216  4277 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:07.776453  4284 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:07.835551  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:07.844996  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:07.798352  4180 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:07.928772  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 84641.6us
W20260709 13:02:07.929894  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:07.956372  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 120921us
I20260709 13:02:07.959520  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:07.965597  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:07.969308  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:07.970045  1593 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:07.969041  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:07.781078  4302 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:07.964802  4231 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:07.967670  4303 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:07.973932  4249 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.004s	user 0.011s	sys 0.004s
I20260709 13:02:07.993274  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:07.999228  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.004988  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.006110  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:07.977783  4045 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.008410  1735 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 13:02:08.011929  1735 pending_rounds.cc:85] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:02:08.012226  1735 pending_rounds.cc:107] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:08.012472  1735 raft_consensus.cc:2889] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:07.979328  4235 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:08.010443  1593 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:08.020639  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.024370  1592 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:08.001360  4180 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:08.007628  3939 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.640s	user 0.016s	sys 0.000s
I20260709 13:02:08.027042  1593 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260709 13:02:08.027592  1593 pending_rounds.cc:85] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:02:08.007860  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:08.027953  1593 pending_rounds.cc:107] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:02:08.028441  1593 raft_consensus.cc:2889] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:08.029727  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.013815  4249 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.018185  4272 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:08.010715  4130 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 13:02:08.026887  4311 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.217s	user 0.001s	sys 0.000s
W20260709 13:02:08.032855  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.020782  4302 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:07.989132  4037 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.004760  4303 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.028622  4180 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.007169  1588 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:08.031847  4035 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:08.050485  1593 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:08.019811  4045 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.052474  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.052793  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.054028  1586 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.053246  1592 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:08.040316  4096 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.042431  4272 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:08.074419  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 115061us
I20260709 13:02:08.027470  4235 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.049610  4180 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.045601  4063 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:08.050503  4035 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.091120  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.086366  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:08.083081  4272 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.075520  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:08.072642  4249 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:08.102424  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:08.103443  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 258500us
I20260709 13:02:08.104372  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:08.092818  4181 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.151s	user 0.050s	sys 0.057s
I20260709 13:02:08.109562  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:08.107285  4181 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.168s	user 0.051s	sys 0.057s
I20260709 13:02:08.118432  3939 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.112502  1665 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
W20260709 13:02:08.123446  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48059.8us
I20260709 13:02:08.126506  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.099735  4035 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:08.132439  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.133473  1737 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:08.135015  1741 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:08.136226  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.137090  1734 raft_consensus.cc:2410] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260709 13:02:08.139209  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.106786  3600 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:08.144552  1586 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:08.139394  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.150823  1591 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:08.108968  4231 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:08.159869  1585 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:08.164426  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.190313  1736 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.166934  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.174348  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.194552  1739 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:08.188728  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.196363  1742 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:08.186594  1660 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
W20260709 13:02:08.203557  4119 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:02:08.222285  1734 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:08.103559  4063 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:08.229795  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.59456e+06us
I20260709 13:02:08.235018  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:08.238423  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.158886  3600 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:08.242357  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 140000us
W20260709 13:02:08.224584  4298 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:02:08.123098  4177 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.105s
I20260709 13:02:08.186371  4161 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.255219  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.261850  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.268899  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:08.269660  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:08.271189  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 144758us
I20260709 13:02:08.232129  4319 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:08.226730  4291 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:08.281672  1734 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.285874  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.288400  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.280117  1736 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
W20260709 13:02:08.289788  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.141731  3939 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.265452  4204 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.267073  3600 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.242637  4318 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:08.290058  4321 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.291092  4319 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.294021  4119 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.303047  4299 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.353309  1734 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:08.354552  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.355636  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.356395  1585 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
W20260709 13:02:08.358541  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70215.5us
I20260709 13:02:08.359535  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.291910  4161 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.327078  4037 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:08.363448  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97641us
W20260709 13:02:08.361706  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86686.3us
I20260709 13:02:08.342360  4318 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:08.341801  4303 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:08.342729  4319 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.350894  4045 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.353240  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:08.414068  2314 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:02:08.421759  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.430529  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.404693  4235 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:08.434815  4254 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.705s	user 0.005s	sys 0.007s
I20260709 13:02:08.352850  4321 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.412654  4037 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:08.349964  4231 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.393904  4195 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:02:08.429469  4119 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:08.464147  4283 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.713s	user 0.014s	sys 0.000s
I20260709 13:02:08.468116  4119 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.469177  1590 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:08.472640  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.477072  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.478147  2318 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:08.478734  4119 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:08.480675  1489 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 126) took 12084 ms (client timeout 15000 ms). Trace:
I20260709 13:02:08.420442  4177 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 13:02:08.483932  4321 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.485795  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.497098  2318 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:08.483491  1741 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:08.502851  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.503695  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.511731  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.513679  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.516299  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.515767  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.540400  1741 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:08.501883  4234 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.012s	user 0.014s	sys 0.000s
I20260709 13:02:08.539160  1589 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:08.501827  4226 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.801s	user 0.008s	sys 0.004s
I20260709 13:02:08.441671  4254 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.551584  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.553957  1734 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:08.554848  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.555708  1735 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:08.554594  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:08.556519  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 197098us
I20260709 13:02:08.558000  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:08.547008  4320 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.320s	user 0.002s	sys 0.000s
I20260709 13:02:08.557482  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.560864  4037 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.561064  1589 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:08.553054  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.516604  4318 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.515249  4325 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.561170  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.564497  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.570581  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.504345  4327 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.552843  4226 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.573390  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.593536  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:08.556165  4234 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.576943  1735 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
W20260709 13:02:08.489866  1489 rpcz_store.cc:269] 0709 13:01:56.404942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:56.405174 (+   232us) service_pool.cc:224] Handling call
0709 13:02:08.480426 (+12075252us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:04.202379 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:04.203335 (+   956us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:04.203401 (+    66us) write_op.cc:435] Acquired schema lock
0709 13:02:04.203444 (+    43us) tablet.cc:662] Decoding operations
0709 13:02:04.238209 (+ 34765us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:04.238329 (+   120us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:04.238373 (+    44us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:02:04.240264 (+  1891us) tablet.cc:701] Row locks acquired
0709 13:02:04.240326 (+    62us) write_op.cc:260] PREPARE: finished
0709 13:02:04.240560 (+   234us) write_op.cc:270] Start()
0709 13:02:04.240766 (+   206us) write_op.cc:276] Timestamp: P: 1783602124240472 usec, L: 0
0709 13:02:04.240798 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 13:02:04.283638 (+ 42840us) log.cc:844] Serialized 6341 byte log entry
0709 13:02:05.315418 (+1031780us) op_driver.cc:464] REPLICATION: finished
0709 13:02:05.794884 (+479466us) write_op.cc:301] APPLY: starting
0709 13:02:05.795015 (+   131us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:05.795503 (+   488us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:05.795683 (+   180us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:05.839203 (+ 43520us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:05.839914 (+   711us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 13:02:05.840138 (+   224us) write_op.cc:312] APPLY: finished
0709 13:02:05.875448 (+ 35310us) log.cc:844] Serialized 113 byte log entry
0709 13:02:05.919714 (+ 44266us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:05.940057 (+ 20343us) write_op.cc:454] Released schema lock
0709 13:02:05.940148 (+    91us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:05.940656 (+   508us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:05.940689 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":417682,"apply.run_cpu_time_us":7309,"apply.run_wall_time_us":145756,"num_ops":16,"prepare.queue_time_us":1215396,"prepare.run_cpu_time_us":33793,"prepare.run_wall_time_us":622541,"raft.queue_time_us":232823,"raft.run_cpu_time_us":60885,"raft.run_wall_time_us":462657,"replication_time_us":1025856,"thread_start_us":2148946,"threads_started":4,"wal-append.queue_time_us":593992,"wal-append.run_cpu_time_us":17153,"wal-append.run_wall_time_us":2159132}]]}
I20260709 13:02:08.577786  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.602936  1734 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:08.597090  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.562404  1591 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:08.588811  4180 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.592870  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.604254  1741 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:08.604746  1736 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:08.606037  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.608081  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.609336  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.610586  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.611218  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:08.611397  4302 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.566s	user 0.009s	sys 0.007s
I20260709 13:02:08.612200  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.612730  1584 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:02:08.596175  4196 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.773s	user 0.011s	sys 0.001s
I20260709 13:02:08.627128  4234 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:08.627908  1738 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.628746  4319 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.629802  3939 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:08.631534  4227 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.825s	user 0.011s	sys 0.000s
I20260709 13:02:08.631973  4227 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.632994  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.633688  1735 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:08.634701  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.635715  1735 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260709 13:02:08.636144  4196 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:08.638614  2313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 612) took 3260 ms (client timeout 2882 ms). Trace:
I20260709 13:02:08.639019  4196 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.639333  4161 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.606870  4119 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:08.640012  4334 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.640127  4119 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.640911  4119 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.640700  4334 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.615608  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:08.620157  4284 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.643072  4334 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.615437  1733 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:08.622085  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.644142  1737 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.593770  4037 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:02:08.645702  4259 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.646293  4259 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.647312  4255 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.647840  4255 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.648692  4259 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.648895  4254 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.650691  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:08.651553  2314 raft_consensus.cc:2368] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
W20260709 13:02:08.652544  4310 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.927s	user 0.000s	sys 0.007s
I20260709 13:02:08.656195  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1497) took 1201 ms. Trace:
I20260709 13:02:08.636554  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.668061  1738 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:08.668450  1735 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 13:02:08.639082  2313 rpcz_store.cc:269] 0709 13:02:05.383173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.417976 (+ 34803us) service_pool.cc:224] Handling call
0709 13:02:08.638386 (+3220410us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:08.638498 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3818711424}
I20260709 13:02:08.656605  1664 rpcz_store.cc:276] 0709 13:02:07.454936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.459375 (+  4439us) service_pool.cc:224] Handling call
0709 13:02:07.459762 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.484035 (+ 24273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.484068 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.484543 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.629320 (+144777us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:07.629357 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.629452 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.651779 (+1022327us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:08.655839 (+  4060us) raft_consensus.cc:1685] finished
0709 13:02:08.655863 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.656129 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68723968,"thread_start_us":119415,"threads_started":1,"wal-append.queue_time_us":171832}
I20260709 13:02:08.671634  1737 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:08.679688  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:08.680255  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79021.7us
I20260709 13:02:08.682502  1739 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:08.667673  4334 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.677412  4226 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:08.683672  4334 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:08.684490  3805 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:08.684901  3805 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:08.685962  3805 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.687121  3805 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.695349  4035 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.699504  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.667665  4234 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.612776  1589 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:08.678078  4294 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.074s	user 0.013s	sys 0.004s
I20260709 13:02:08.704460  1735 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [2.2-5.3]
W20260709 13:02:08.701390  4249 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.012s	sys 0.016s
W20260709 13:02:08.706979  4249 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.012s	sys 0.016s
I20260709 13:02:08.707244  4249 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:08.712736  4180 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.697376  4119 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.708184  4249 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.718761  4238 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.706054  4294 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.722224  4292 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:08.722352  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:08.722827  4292 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:08.723119  4119 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.723526  4231 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.723798  4119 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:08.724260  4119 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.724020  4231 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.724627  4119 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:08.725531  4314 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.999s	user 0.002s	sys 0.000s
I20260709 13:02:08.725800  4035 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.726822  4231 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.726991  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 343) took 1265 ms. Trace:
I20260709 13:02:08.727725  1661 rpcz_store.cc:276] 0709 13:02:07.461638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.491077 (+ 29439us) service_pool.cc:224] Handling call
0709 13:02:07.491578 (+   501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.510639 (+ 19061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.510676 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.527259 (+ 16583us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.713786 (+186527us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:07.713849 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.713956 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.724636 (+1010680us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:08.726670 (+  2034us) raft_consensus.cc:1685] finished
0709 13:02:08.726696 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.726925 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":5397,"spinlock_wait_cycles":40025472,"thread_start_us":110246,"threads_started":1,"wal-append.queue_time_us":173582}
I20260709 13:02:08.721215  4294 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.739869  4344 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:08.740378  4306 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.021s	user 0.000s	sys 0.004s
I20260709 13:02:08.740784  4344 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.727087  4037 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:02:08.749655  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.753052  1588 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:08.754160  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 341) took 1355 ms. Trace:
I20260709 13:02:08.764942  4344 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.769876  4304 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.770805  4304 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.772133  3405 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:08.772753  3405 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:08.771131  4312 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:08.774329  3405 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.776048  1742 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:08.776328  3405 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.715672  4255 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.714671  4226 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.778475  1742 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:02:08.755062  1668 rpcz_store.cc:276] 0709 13:02:07.398833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.398977 (+   144us) service_pool.cc:224] Handling call
0709 13:02:07.414335 (+ 15358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:07.426724 (+ 12389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:07.426756 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:07.453409 (+ 26653us) log.cc:844] Serialized 26 byte log entry
0709 13:02:07.525377 (+ 71968us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:07.525437 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:07.525565 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:08.739736 (+1214171us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:08.743292 (+  3556us) raft_consensus.cc:1685] finished
0709 13:02:08.743320 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:08.754083 (+ 10763us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":540,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":31531,"spinlock_wait_cycles":64633728,"thread_start_us":37424,"threads_started":1,"wal-append.queue_time_us":231833}
I20260709 13:02:08.703732  4234 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.785473  4234 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:08.787472  4272 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.000s
I20260709 13:02:08.788002  4272 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.785658  4209 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:08.791198  3600 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.793910  1740 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:08.794765  3600 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.796058  1736 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 13:02:08.800611  1585 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260709 13:02:08.800184  1736 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:08.801127  1585 pending_rounds.cc:85] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:02:08.801682  1585 pending_rounds.cc:107] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260709 13:02:08.799391  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.801877  1585 raft_consensus.cc:2889] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:08.798031  4308 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.038s	user 0.023s	sys 0.017s
I20260709 13:02:08.774390  4304 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:08.801146  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 78864.4us
I20260709 13:02:08.808139  1706 leader_election.cc:400] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:08.808495  1706 leader_election.cc:403] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:08.802453  1588 raft_consensus.cc:2376] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5. Re-sending same reply.
I20260709 13:02:08.804489  1742 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:08.804827  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:08.806918  4308 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.061s	user 0.024s	sys 0.017s
I20260709 13:02:08.811314  1584 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:02:08.815176  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.815246  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.816188  1588 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:08.818256  4281 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:08.818735  4281 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:08.818836  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.819434  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.819756  1588 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.820163  1584 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:08.819813  4281 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.821058  4281 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.816404  4226 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:08.822610  4272 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:08.824306  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.723752  4292 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:08.827888  4226 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:08.828372  4226 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:08.828837  4226 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:08.831075  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 344) took 1088 ms. Trace:
I20260709 13:02:08.831503  1733 rpcz_store.cc:276] 0709 13:02:07.742505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.742738 (+   233us) service_pool.cc:224] Handling call
0709 13:02:07.747487 (+  4749us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:08.643745 (+896258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.830795 (+187050us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b5400435654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:08.831006 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1994971904}
I20260709 13:02:08.842985  4292 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.848035  4334 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.849535  4319 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:08.850667  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1489) took 3125 ms. Trace:
I20260709 13:02:08.854513  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:08.826539  4325 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:08.857067  1660 rpcz_store.cc:276] 0709 13:02:05.724809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:05.900971 (+176162us) service_pool.cc:224] Handling call
0709 13:02:05.921488 (+ 20517us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:08.198170 (+2276682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:08.850368 (+652198us) spinlock_profiling.cc:243] Waited 2.24 s on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:08.850602 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5158015360}
I20260709 13:02:08.859006  1588 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:08.860088  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.860865  4161 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:08.860739  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.860966  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:08.861387  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.862654  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:08.864461  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.865291  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.867550  2317 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
W20260709 13:02:08.869099  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.866863  4161 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:08.872855  4296 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:08.861779  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:08.873769  4319 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:08.884938  4319 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:08.875663  4296 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:08.886469  4296 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.887984  4296 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.872956  4161 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.882718  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.890251  1588 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.893296  4281 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:08.894285  4281 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.897631  4281 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:08.897804  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:08.898492  4319 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.901201  4063 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:08.908156  1588 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:08.909940  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.899302  1584 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:08.909369  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.913054  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:08.915709  1588 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:08.913034  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.922396  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.923796  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:08.921653  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:08.925813  1736 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.927996  4321 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:08.928912  4321 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:08.930177  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.930913  1584 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:08.930402  4321 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:08.933267  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.938027  1584 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:08.912720  4172 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 13:02:08.935998  1739 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:08.912051  4249 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:08.941459  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.946920  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.943933  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:08.952066  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.954052  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2060) took 1630 ms. Trace:
I20260709 13:02:08.954509  1738 rpcz_store.cc:276] 0709 13:02:07.317545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.317731 (+   186us) service_pool.cc:224] Handling call
0709 13:02:08.946967 (+1629236us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54001081d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:08.947314 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2902782208}
I20260709 13:02:08.955018  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.956398  4255 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.956454  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:08.957271  1584 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:08.957486  4312 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:08.958417  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.958981  1584 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:08.958925  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:08.960537  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.961334  3405 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.962873  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:08.963791  3405 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.965382  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:08.968963  4334 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:08.949934  4304 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:08.969470  4334 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.955426  4031 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:08.970963  4334 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:08.971254  4031 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:08.971683  4031 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:08.967796  3405 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.952338  4209 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:08.982425  1584 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:08.939862  4281 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:08.981050  4334 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:09.008303  1738 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:09.004103  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.012183  1740 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:09.005836  3405 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 13:02:09.022542  4308 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:09.024840  3405 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:09.025317  3405 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:09.022710  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.027467  1587 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:09.029675  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.031042  1589 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:09.031390  1736 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:08.982244  4249 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:09.036265  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:08.992712  4209 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.040855  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.042409  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.042450  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.048448  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.045241  4308 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:09.049561  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.052857  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2017) took 1254 ms. Trace:
I20260709 13:02:09.053244  4308 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:09.050969  1585 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.053527  1590 rpcz_store.cc:276] 0709 13:02:07.798456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.802064 (+  3608us) service_pool.cc:224] Handling call
0709 13:02:09.052551 (+1250487us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:09.052775 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525483776}
I20260709 13:02:09.055347  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.056432  1590 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:09.060387  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.060742  1586 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:02:09.061280  1586 pending_rounds.cc:85] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:02:09.061594  1586 pending_rounds.cc:107] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 13:02:09.061941  1586 raft_consensus.cc:2889] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:09.062343  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.063390  4334 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:09.063531  3805 raft_consensus.cc:2764] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 13:02:09.067142  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.067922  1591 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:09.074046  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.074774  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.065049  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.079679  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:09.080667  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.081161  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.081471  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.081806  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.065150  3805 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:09.083424  4308 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:09.084263  4255 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:09.084151  4031 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.076480  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.086189  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.086722  1584 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.086757  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.069123  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.091867  1591 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:09.095450  1557 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.100s (SENT)
I20260709 13:02:09.083791  3805 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:09.096027  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.096478  3805 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:09.096210  4226 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.096866  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 344) took 1277 ms. Trace:
I20260709 13:02:09.097520  1740 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:09.083209  1585 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:09.099255  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.099040  4119 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.100395  1742 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:09.101306  1739 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 13:02:09.102602  1585 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:09.102427  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.094396  1593 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:09.041606  1587 raft_consensus.cc:2376] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6. Re-sending same reply.
I20260709 13:02:09.116150  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:09.119831  1591 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:09.117825  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.121836  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.122582  1593 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.121315  1584 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:09.125087  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.126026  1588 raft_consensus.cc:2376] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8. Re-sending same reply.
I20260709 13:02:09.037897  4281 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:09.040160  4209 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.097313  1662 rpcz_store.cc:276] 0709 13:02:07.819279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:07.823287 (+  4008us) service_pool.cc:224] Handling call
0709 13:02:07.823601 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:07.823809 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:08.104990 (+281181us) log.cc:844] Serialized 14 byte log entry
0709 13:02:08.223658 (+118668us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.096287 (+872629us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x7b18003b9bc0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:09.096441 (+   154us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:09.096495 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:09.096544 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.096602 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.096796 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":927204,"prepare.queue_time_us":447239,"prepare.run_cpu_time_us":4181,"prepare.run_wall_time_us":35068,"raft.queue_time_us":1007779,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":31780,"spinlock_wait_cycles":12302720,"thread_start_us":396213,"threads_started":2}
I20260709 13:02:09.144738  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:09.115293  4308 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:09.106472  4291 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.151811  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:09.151944  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 764140us
I20260709 13:02:09.146077  3903 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 13:02:09.154377  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.156172  3903 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:09.156751  3903 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:09.158391  4291 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.148728  4304 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.161718  4045 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:09.162129  4119 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.162047  4304 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.149333  4284 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 13:02:09.164330  4304 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.168126  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.169374  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.169761  4063 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:09.170385  4063 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.170881  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.171555  1659 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:09.173575  4330 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:09.173998  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.174605  4330 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.175331  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.175556  1663 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:09.175918  1666 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.177071  4330 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.170774  4184 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:09.177899  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.178292  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.178494  4096 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:09.171494  4237 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 13:02:09.179668  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.179975  3600 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:09.180660  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.181877  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.182569  1659 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:09.182789  4226 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.174166  4130 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:09.183903  4130 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.184866  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.185566  1664 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:09.177841  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.173445  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.187067  1585 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:09.160987  4249 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.186542  1667 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.190855  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.188750  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.198058  1742 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:09.194361  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.199198  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.200176  1593 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:09.200531  1738 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:09.202687  4130 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.150004  4196 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.204418  1664 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: [3.3-6.4]
I20260709 13:02:09.204288  4196 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.209952  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.210173  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.208668  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.212249  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.210098  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.214751  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.216496  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:09.217224  1742 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:09.211036  1738 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
W20260709 13:02:09.219981  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.221310  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.173013  4063 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.142082  4277 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 13:02:09.196429  4096 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:09.235692  1667 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:09.232568  1589 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:09.231549  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.237977  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:09.232576  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.196606  4304 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.246198  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.244307  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.239625  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.245215  4324 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:09.246137  4304 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.244977  4119 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:09.248225  4119 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:09.248790  4119 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.250172  4334 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:09.250872  4334 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:09.251657  4189 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.251937  4334 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.250386  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.255929  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.256497  4196 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.257231  4180 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:09.257680  4180 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.258038  4180 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:09.257966  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.258957  1584 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:09.259336  4324 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:09.259895  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.260072  4274 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:09.260708  1584 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:09.261016  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.206130  4330 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:09.255117  1488 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 4 to 7, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:09.265703  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.273846  1742 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:09.276044  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.279361  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.280202  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.280175  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.281749  1584 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:09.282871  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.283699  1584 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:09.215292  4130 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:09.290383  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.292905  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.295351  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.208259  4263 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260709 13:02:09.229595  4287 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:09.306802  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.307466  1586 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:09.259897  4324 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.281185  4277 consensus_queue.cc:1243] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:09.331043  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.290482  4130 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:09.290333  4286 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.013s
I20260709 13:02:09.290254  4262 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:09.304107  4304 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.304945  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.223371  4322 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.116s
I20260709 13:02:09.305444  4266 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:09.305908  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.317225  4287 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.228901  4123 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:09.262874  4096 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.317456  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.295652  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.318809  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:09.365468  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46755.9us
I20260709 13:02:09.364188  4294 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.350049  4324 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.368780  4294 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.372104  4294 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.378928  3805 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.346426  4161 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.388392  4161 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.391425  1663 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:09.398852  4037 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:09.399613  4037 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.400094  4037 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.349453  4119 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.404568  4119 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:09.242635  3903 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.324488  1737 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:09.324407  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.296870  4319 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:09.215903  4255 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.075s
I20260709 13:02:09.415741  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.319890  1488 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:09.268200  4330 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.260730  4274 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.290102  4235 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 13:02:09.223978  4035 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 13:02:09.279510  4189 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.334882  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.435099  4189 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.335513  1662 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:09.229204  4237 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.088s
I20260709 13:02:09.305137  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.334932  4130 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:09.345393  1589 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.443536  1588 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.242030  4281 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.344000  4031 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:09.347138  1668 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:09.346856  4238 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:09.477074  4196 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:09.477542  4196 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:09.344096  4262 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.325150  4349 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:09.478446  4196 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.479029  4349 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.412295  1591 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 3. (term mismatch)
I20260709 13:02:09.479660  1591 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 2
I20260709 13:02:09.480114  1591 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 13:02:09.480381  1591 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:09.414047  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 118490us
I20260709 13:02:09.481617  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2042) took 1009 ms. Trace:
I20260709 13:02:09.393296  1663 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:09.414465  1590 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.322655  1659 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:09.484834  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.420804  4296 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:09.420970  1733 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:09.421633  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 273791us
I20260709 13:02:09.358033  4287 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.426644  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.357528  4096 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:09.428493  1667 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:09.445590  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:09.502225  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 211242us
W20260709 13:02:09.502276  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57019.2us
W20260709 13:02:09.503454  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.505276  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.459442  1589 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:09.446015  4063 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 13:02:09.473714  4281 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.508373  1593 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:09.461704  4226 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.509801  4226 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.474678  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.511978  1734 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:09.475625  4180 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.476311  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.421305  4330 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.424988  4266 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:09.486156  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.486519  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.530833  1742 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
W20260709 13:02:09.492991  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.482012  1591 rpcz_store.cc:276] 0709 13:02:08.472042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.411438 (+939396us) service_pool.cc:224] Handling call
0709 13:02:09.411805 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.481319 (+ 69514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.481558 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:09.480312  4358 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:02:09.512075  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.479480  4349 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.509145  4281 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.516835  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.517313  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.494148  1488 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:09.420104  1660 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:09.519939  1584 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:09.480463  4209 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:09.462613  4319 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:09.545172  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59091.4us
W20260709 13:02:09.573829  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57135.1us
I20260709 13:02:09.567370  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.562646  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:09.575105  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:09.575917  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.576853  1590 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:09.575914  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.578218  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.568538  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.582096  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:09.582437  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.550551  4196 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:09.497014  4296 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.491063  4321 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:09.515812  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.494949  4263 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:09.521030  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.567291  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.568104  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.568493  1586 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:09.568137  1588 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:09.569811  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.479483  4262 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.584854  4272 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:09.571955  1663 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:09.584889  4281 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:09.550981  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:09.581254  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.652729  1738 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
W20260709 13:02:09.653419  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.654062  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:09.654709  1738 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:09.541464  1666 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:02:09.557858  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 413267us
I20260709 13:02:09.608421  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.583952  1488 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 2 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:09.561275  4181 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:09.590337  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.578295  4319 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.664280  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.665159  1590 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:09.607227  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.610810  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.667356  1591 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:09.617653  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.618026  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:09.561887  4266 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.673674  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:09.573580  4255 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:09.624341  1587 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:09.624419  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.628014  1739 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:09.542984  4292 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.699s	user 0.006s	sys 0.007s
I20260709 13:02:09.648267  4209 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:09.571265  4364 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 13:02:09.571731  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:09.703567  1734 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:09.585084  3805 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.585316  4184 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:09.521571  4304 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:09.606062  4327 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:09.619087  4296 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.657855  1584 raft_consensus.cc:2376] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7. Re-sending same reply.
I20260709 13:02:09.611300  4237 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:09.666107  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.585943  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:09.672480  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:09.580073  4342 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:02:09.627370  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:09.860572  1660 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:09.678457  1590 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:09.861210  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.862347  4119 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:09.862897  4119 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.631256  4189 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.583817  4323 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
I20260709 13:02:09.703229  4324 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:09.866312  4180 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:09.583979  4291 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 13:02:09.659597  4249 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:09.868573  4249 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:09.647212  4226 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:09.698755  1589 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:09.699235  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 79586.7us
I20260709 13:02:09.705003  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:09.705988  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 143943us
I20260709 13:02:09.894718  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:09.904554  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:09.784850  1738 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:09.907197  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:09.908258  1738 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:09.670529  4181 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.707373  1488 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 2 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:09.726842  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:09.946553  1591 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:09.669271  4281 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:09.640995  4321 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.606052  4274 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:09.682557  4255 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:09.810835  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1491) took 1479 ms. Trace:
W20260709 13:02:09.811928  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:09.813835  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:09.989818  2318 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:09.814590  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:09.818881  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:09.684820  4272 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:09.822898  1666 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:09.824761  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:09.657914  3939 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:09.927230  1734 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:09.870019  4249 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:02:09.937359  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52708.9us
I20260709 13:02:09.996771  1662 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:09.987130  1735 rpcz_store.cc:276] 0709 13:02:08.331226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.709806 (+378580us) service_pool.cc:224] Handling call
0709 13:02:08.710163 (+   357us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:08.744199 (+ 34036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:08.744256 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:08.975797 (+231541us) spinlock_profiling.cc:243] Waited 4.38 ms on lock 0x7b2c00077ef8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:08.976476 (+   679us) log.cc:844] Serialized 52 byte log entry
0709 13:02:09.191749 (+215273us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:09.499088 (+307339us) log.cc:844] Serialized 14 byte log entry
0709 13:02:09.500761 (+  1673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.500873 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:09.709674 (+208801us) raft_consensus.cc:1685] finished
0709 13:02:09.810461 (+100787us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:09.810735 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":465915,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":7379,"raft.queue_time_us":3186,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":34669,"spinlock_wait_cycles":10069248,"thread_start_us":490882,"threads_started":2,"wal-append.queue_time_us":364327}
I20260709 13:02:09.870795  4031 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.179s
I20260709 13:02:09.821228  4130 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:10.044217  1738 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
W20260709 13:02:10.564396  1705 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.650s (SENT) [suppressed 29 similar messages]
W20260709 13:02:10.056407  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.566656  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:10.114758  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 300312us
I20260709 13:02:10.097970  2318 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:10.445230  4037 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.040s	user 0.015s	sys 0.000s
I20260709 13:02:10.570873  4037 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:10.571933  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:10.572800  1735 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 2.
I20260709 13:02:09.911357  4196 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.668s
I20260709 13:02:10.176152  4266 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.083966  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:10.344825  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:10.343253  4325 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:10.187603  4180 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.589726  4180 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:10.294288  1591 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:10.592209  4180 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:10.593078  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:10.601817  1741 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:10.602766  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1513) took 1189 ms (client timeout 1581 ms). Trace:
I20260709 13:02:10.213915  4373 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.564301  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2050) took 1985 ms. Trace:
I20260709 13:02:10.605022  1584 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:10.085948  4367 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.431s	user 0.001s	sys 0.000s
I20260709 13:02:10.595459  3405 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.599339  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:10.617095  1659 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
W20260709 13:02:10.344278  4376 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.459s	user 0.000s	sys 0.001s
W20260709 13:02:10.634835  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.975s	user 0.007s	sys 0.075s
W20260709 13:02:10.138671  4358 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 13:02:09.995741  4255 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:10.601817  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 357) took 1697 ms. Trace:
I20260709 13:02:10.646751  1733 rpcz_store.cc:276] 0709 13:02:08.903937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.934233 (+ 30296us) service_pool.cc:224] Handling call
0709 13:02:08.934571 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.490822 (+556251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.601561 (+1110739us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54002030d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:10.601747 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118230912}
I20260709 13:02:10.603788  4373 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.593384  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1489) took 2311 ms. Trace:
I20260709 13:02:10.210875  4161 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:10.073833  1740 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:10.606166  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2045) took 2132 ms. Trace:
I20260709 13:02:10.214993  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:10.752130  1659 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:10.751984  1584 rpcz_store.cc:276] 0709 13:02:08.473375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.604455 (+2131080us) service_pool.cc:224] Handling call
0709 13:02:10.604822 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.605849 (+  1027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.606061 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:10.058044  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 163472us
W20260709 13:02:10.753477  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:10.755462  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:10.755764  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.754637  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:10.603340  1741 rpcz_store.cc:269] 0709 13:02:09.413054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.592633 (+1179579us) service_pool.cc:224] Handling call
0709 13:02:10.602703 (+ 10070us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.754637  1584 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:10.747257  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2177) took 2017 ms. Trace:
I20260709 13:02:10.758489  1587 rpcz_store.cc:276] 0709 13:02:08.729361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.145967 (+416606us) service_pool.cc:224] Handling call
0709 13:02:10.746937 (+1600970us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54002701d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:10.747168 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1060178304}
I20260709 13:02:10.264248  1488 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 3, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:10.757376  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:10.573396  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2080) took 1063 ms. Trace:
I20260709 13:02:10.760639  1735 rpcz_store.cc:276] 0709 13:02:09.509875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.571561 (+1061686us) service_pool.cc:224] Handling call
0709 13:02:10.573332 (+  1771us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.608960  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:10.086375  4272 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.625633  3405 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:10.625903  4308 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.629202  1588 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:10.630618  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:10.635726  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.007s	sys 0.075s
I20260709 13:02:10.565472  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2082) took 1051 ms. Trace:
I20260709 13:02:10.005342  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:10.604218  1593 rpcz_store.cc:276] 0709 13:02:08.579286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.279023 (+699737us) service_pool.cc:224] Handling call
0709 13:02:10.564080 (+1285057us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400144454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:10.564238 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":523226880}
I20260709 13:02:10.098835  4383 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:10.049741  3939 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:10.786931  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.12714e+06us
I20260709 13:02:10.592550  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2043) took 2120 ms. Trace:
I20260709 13:02:10.662396  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:10.788339  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:10.788636  1733 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:10.649431  1737 rpcz_store.cc:276] 0709 13:02:08.282479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.691122 (+408643us) service_pool.cc:224] Handling call
0709 13:02:08.691489 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:09.413999 (+722510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.593151 (+1179152us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:10.593323 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1455375616}
I20260709 13:02:10.647982  4334 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:10.702970  4370 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.073s	user 0.002s	sys 0.000s
I20260709 13:02:10.606189  4181 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:10.712970  4096 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.189s	user 0.010s	sys 0.001s
I20260709 13:02:10.799460  4385 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:10.799284  4096 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.800089  4385 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:10.722714  4123 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.361s	user 0.010s	sys 0.004s
I20260709 13:02:10.801779  4123 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:10.801841  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 364) took 1709 ms. Trace:
I20260709 13:02:10.802611  4123 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:10.803156  4123 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:10.804035  4123 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.457732  4281 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.728119  1738 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:10.812414  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:10.636900  4284 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.899s	user 0.010s	sys 0.014s
W20260709 13:02:10.648509  4327 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.004s	sys 0.012s
W20260709 13:02:10.648695  1556 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.067s	user 0.004s	sys 0.024s
I20260709 13:02:10.762953  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2053) took 2182 ms. Trace:
W20260709 13:02:10.827778  1556 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.246s	user 0.009s	sys 0.024s
I20260709 13:02:10.764497  1488 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:10.828168  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.24643e+06us
I20260709 13:02:10.828377  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:10.766570  1735 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260709 13:02:10.830170  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1517) took 1415 ms. Trace:
I20260709 13:02:10.830605  1735 rpcz_store.cc:276] 0709 13:02:09.415106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.765906 (+1350800us) service_pool.cc:224] Handling call
0709 13:02:10.766305 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.829771 (+ 63466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.830094 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:10.738874  4349 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.166s	user 0.013s	sys 0.000s
I20260709 13:02:10.784116  1742 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:10.733408  4287 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.208s	user 0.012s	sys 0.000s
I20260709 13:02:10.844338  4287 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.818342  4304 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:10.846762  4287 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:10.847477  4287 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:10.770143  1587 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:10.848951  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 357) took 2247 ms. Trace:
I20260709 13:02:10.849287  1587 rpcz_store.cc:276] 0709 13:02:08.601788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.759398 (+2157610us) service_pool.cc:224] Handling call
0709 13:02:10.769944 (+ 10546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.848657 (+ 78713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.848871 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.850049  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:10.850320  2318 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:10.788091  1591 rpcz_store.cc:276] 0709 13:02:08.472369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.136732 (+1664363us) service_pool.cc:224] Handling call
0709 13:02:10.293724 (+156992us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.592000 (+298276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.592378 (+   378us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.790360  1737 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:10.790639  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2048) took 2212 ms. Trace:
I20260709 13:02:10.793597  1734 rpcz_store.cc:276] 0709 13:02:09.513846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.550912 (+ 37066us) service_pool.cc:224] Handling call
0709 13:02:10.565400 (+1014488us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.852246  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2078) took 1345 ms. Trace:
I20260709 13:02:10.852802  1737 rpcz_store.cc:276] 0709 13:02:09.506669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.789361 (+1282692us) service_pool.cc:224] Handling call
0709 13:02:10.790149 (+   788us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.851996 (+ 61847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.852189 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.739308  4031 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.856395  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:10.589748  4325 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.859619  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:10.826115  4284 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.129s	user 0.011s	sys 0.015s
I20260709 13:02:10.797430  4379 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:10.805855  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 197019us
I20260709 13:02:10.802312  1736 rpcz_store.cc:276] 0709 13:02:09.092754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.093368 (+   614us) service_pool.cc:224] Handling call
0709 13:02:09.093764 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.094204 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:09.094236 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:09.094735 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 13:02:09.497767 (+403032us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:09.497921 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:09.498005 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:10.801587 (+1303582us) raft_consensus.cc:1685] finished
0709 13:02:10.801608 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:10.801777 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4008,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":402489,"threads_started":1,"wal-append.queue_time_us":1410439}
I20260709 13:02:10.810020  1665 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:10.870402  1557 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.581s (SENT)
I20260709 13:02:10.871035  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:10.741705  4373 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:10.813241  1593 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:10.794531  3939 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:10.814842  4130 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:10.816129  4296 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.878s	user 0.008s	sys 0.004s
I20260709 13:02:10.767562  4312 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:10.816787  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:10.913291  1668 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:10.820564  1733 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:10.908376  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:10.785116  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:10.768057  4063 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:10.827910  1584 rpcz_store.cc:276] 0709 13:02:08.580316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.753755 (+2173439us) service_pool.cc:224] Handling call
0709 13:02:10.754401 (+   646us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.757934 (+  3533us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.762888 (+  4954us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.863723  4096 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:10.843281  4349 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.789031  1741 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:02:10.850433  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:10.944738  1587 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:10.857113  1742 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:10.946266  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2085) took 1391 ms. Trace:
I20260709 13:02:10.946776  1742 rpcz_store.cc:276] 0709 13:02:09.555081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.630269 (+1075188us) service_pool.cc:224] Handling call
0709 13:02:10.946177 (+315908us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.948460  1742 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:10.852089  1588 rpcz_store.cc:276] 0709 13:02:08.578581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.628417 (+2049836us) service_pool.cc:224] Handling call
0709 13:02:10.628872 (+   455us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.790282 (+161410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.790570 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.860731  2317 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:10.740777  4119 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:10.805053  4334 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:10.802325  4385 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:10.826799  4327 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.004s	sys 0.013s
I20260709 13:02:10.861689  2315 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:10.839265  4383 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:10.899681  1488 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:10.922093  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:10.790174  4308 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:10.833601  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:10.936136  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:10.936941  1584 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:10.968148  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 359) took 2365 ms. Trace:
I20260709 13:02:10.968703  1584 rpcz_store.cc:276] 0709 13:02:08.602367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.936194 (+2333827us) service_pool.cc:224] Handling call
0709 13:02:10.936500 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.967734 (+ 31234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.968061 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.935294  4255 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:10.937592  1591 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:10.970896  1585 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:10.971084  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 358) took 2369 ms. Trace:
I20260709 13:02:10.971684  1591 rpcz_store.cc:276] 0709 13:02:08.602087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.896981 (+2294894us) service_pool.cc:224] Handling call
0709 13:02:10.937310 (+ 40329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.970729 (+ 33419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.971010 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.973214  4255 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:10.975353  4255 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:10.950441  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:10.949316  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2091) took 1338 ms. Trace:
I20260709 13:02:10.875607  4161 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:10.985249  2313 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:10.986238  4161 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:10.984189  1593 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:10.985118  1742 rpcz_store.cc:276] 0709 13:02:09.611255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.947746 (+1336491us) service_pool.cc:224] Handling call
0709 13:02:10.948198 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.948993 (+   795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:10.949240 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:10.962709  2315 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:10.865466  4031 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.013386  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:10.987398  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 192704us
I20260709 13:02:10.977665  4304 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:11.012872  1488 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:10.927505  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 363147us
I20260709 13:02:10.927449  4063 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:11.017854  1488 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 3 to 5, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:11.019778  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:11.021684  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:10.952533  4130 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:11.040382  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118456us
I20260709 13:02:11.020479  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:11.043222  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:11.044636  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:11.050071  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:11.012344  4330 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.450s	user 0.015s	sys 0.003s
I20260709 13:02:10.897330  4379 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:11.052791  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46818.8us
I20260709 13:02:11.058565  1488 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:10.998828  4327 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:10.915139  4296 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:11.047562  4274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.006s	sys 0.010s
W20260709 13:02:11.064050  4274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.006s	sys 0.010s
I20260709 13:02:11.065560  1488 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:10.936692  4312 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.046015  4096 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:11.067251  1488 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 2 to 5, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:11.039363  4063 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:11.039043  4385 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.168859  1488 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 2 to 4, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:11.170848  1488 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:02:10.902055  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
W20260709 13:02:11.152678  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.158293  1668 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 13:02:11.173882  2312 raft_consensus.cc:606] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:11.040386  4130 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:11.165922  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62525.3us
I20260709 13:02:11.015985  2315 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.182160  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:11.175635  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.177692  1488 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:02:11.058284  4255 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.188676  1668 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:11.150825  4373 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.148586  4387 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:11.255868  1588 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:11.182937  4318 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.183189  4272 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:11.166154  4393 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:11.062201  4330 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:11.168569  4277 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:11.284960  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 234986us
I20260709 13:02:11.165699  4386 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.113s
W20260709 13:02:11.307911  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 258513us
I20260709 13:02:11.315755  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:11.258651  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 388282us
I20260709 13:02:11.269172  4387 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:11.339318  4380 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.474s	user 0.000s	sys 0.001s
W20260709 13:02:11.334427  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.339926  4386 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 13:02:11.340461  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.340617  4386 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:11.188954  4385 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.291064  4401 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:11.377164  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 195072us
I20260709 13:02:11.378602  1661 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 13:02:11.299875  4403 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.253181  4255 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:11.384709  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 315092us
I20260709 13:02:11.386088  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:11.260556  4382 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.114s
I20260709 13:02:11.401225  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:11.329380  4389 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:11.252761  4383 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.334726  4387 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:11.298820  4318 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:11.278982  4373 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:11.456492  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 117740us
I20260709 13:02:11.475333  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:11.365818  4274 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260709 13:02:11.268946  4304 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.310331  4393 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.336449  4277 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.434923  4415 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:11.387666  4384 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260709 13:02:11.362594  4130 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:11.388156  4394 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
I20260709 13:02:11.396956  4294 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
I20260709 13:02:11.541318  4325 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:11.371779  4386 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.595566  4405 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:11.372766  4327 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.433641  4403 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:11.595805  4123 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:11.401906  4401 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:11.607134  1587 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.000s
W20260709 13:02:11.621753  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 223929us
W20260709 13:02:11.627746  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 226611us
I20260709 13:02:11.546141  4409 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.548079  4308 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:11.552863  4031 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:11.638648  1587 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:11.596725  4349 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:11.642266  1737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.004s	sys 0.008s
I20260709 13:02:11.649650  1593 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
W20260709 13:02:11.654439  1737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.004s	sys 0.008s
I20260709 13:02:11.655366  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2088) took 2092 ms. Trace:
W20260709 13:02:11.657063  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 181852us
I20260709 13:02:11.655869  1737 rpcz_store.cc:276] 0709 13:02:09.563387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.854897 (+1291510us) service_pool.cc:224] Handling call
0709 13:02:10.855182 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.855399 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:11.058137 (+202738us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.654947 (+596810us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.655015 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:11.655067 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.655143 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.655297 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42012,"prepare.queue_time_us":642198,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":24729,"thread_start_us":741596,"threads_started":2}
I20260709 13:02:11.658569  1587 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: [3.3-6.4]
I20260709 13:02:11.659224  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2179) took 2928 ms. Trace:
I20260709 13:02:11.659538  1587 rpcz_store.cc:276] 0709 13:02:08.730538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.657915 (+2927377us) service_pool.cc:224] Handling call
0709 13:02:11.658321 (+   406us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:11.658963 (+   642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.659168 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:11.662379  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106832us
I20260709 13:02:11.663189  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:11.625268  4334 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:11.668681  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:11.669464  1737 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:11.670795  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:11.647836  4296 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:11.685108  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2092) took 2059 ms. Trace:
I20260709 13:02:11.708549  1742 rpcz_store.cc:276] 0709 13:02:09.626072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.998776 (+1372704us) service_pool.cc:224] Handling call
0709 13:02:10.999325 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.999855 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.999900 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:11.015426 (+ 15526us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.378714 (+363288us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:11.378761 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.378855 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:11.684843 (+305988us) raft_consensus.cc:1685] finished
0709 13:02:11.684865 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.685032 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11703,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":23767,"thread_start_us":232441,"threads_started":1,"wal-append.queue_time_us":582363}
W20260709 13:02:11.628456  4263 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.000s	sys 0.009s
I20260709 13:02:11.678220  4312 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:11.715543  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52417.2us
W20260709 13:02:11.715749  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.671439  1737 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:11.777689  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:11.664115  4382 consensus_queue.cc:1243] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:11.718590  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:11.787043  1742 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:11.667343  4349 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:11.804134  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:11.804709  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.810854  4349 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.811474  4349 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:11.727401  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1521) took 1894 ms (client timeout 1898 ms). Trace:
I20260709 13:02:11.680858  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:11.847662  1662 rpcz_store.cc:269] 0709 13:02:09.832737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.832918 (+   181us) service_pool.cc:224] Handling call
0709 13:02:11.727088 (+1894170us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b5400012a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:11.727312 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277517184}
I20260709 13:02:11.749372  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1523) took 1807 ms. Trace:
W20260709 13:02:11.684353  4402 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.190s	user 0.000s	sys 0.000s
I20260709 13:02:11.812776  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 364) took 3137 ms. Trace:
I20260709 13:02:11.860004  1593 rpcz_store.cc:276] 0709 13:02:08.675730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.001216 (+2325486us) service_pool.cc:224] Handling call
0709 13:02:11.649260 (+648044us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.654645 (+  5385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.812464 (+157819us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b540025ce50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:11.812677 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1489356160}
W20260709 13:02:11.818195  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:11.859747  1660 rpcz_store.cc:276] 0709 13:02:09.942044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.614107 (+672063us) service_pool.cc:224] Handling call
0709 13:02:10.614551 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.614783 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:11.022807 (+408024us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.745584 (+722777us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.745673 (+    89us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:11.745738 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.745860 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.749263 (+  3403us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11390,"prepare.queue_time_us":399298,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":305362,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":1085966,"threads_started":3,"wal-append.queue_time_us":515887}
W20260709 13:02:11.724423  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 338436us
W20260709 13:02:11.729092  4263 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.000s	sys 0.010s
I20260709 13:02:11.679633  4334 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.868456  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:11.683997  4424 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.792598  4330 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:11.801621  4296 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.803756  4312 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:11.908483  1742 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
W20260709 13:02:11.911190  1734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.005s	sys 0.021s
W20260709 13:02:11.922390  1734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.005s	sys 0.021s
I20260709 13:02:11.952147  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2087) took 2391 ms. Trace:
I20260709 13:02:11.954310  1734 rpcz_store.cc:276] 0709 13:02:09.560962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.852937 (+1291975us) service_pool.cc:224] Handling call
0709 13:02:10.853348 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.853662 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:11.185390 (+331728us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.951809 (+766419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.951874 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:11.951920 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.951986 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:11.952090 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28902,"prepare.queue_time_us":580329,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"thread_start_us":1025248,"threads_started":2}
W20260709 13:02:11.952009  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83670.2us
I20260709 13:02:11.952342  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:11.962910  1742 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:11.951588  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1514) took 2509 ms. Trace:
I20260709 13:02:11.966158  1663 rpcz_store.cc:276] 0709 13:02:09.442746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.558912 (+116166us) service_pool.cc:224] Handling call
0709 13:02:09.571558 (+ 12646us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.646593 (+ 75035us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.951229 (+2304636us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54002e6920. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0a02 00007fcdedfdcdd6 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:11.951515 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":514989824}
W20260709 13:02:11.894444  4411 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.396s	user 0.000s	sys 0.002s
I20260709 13:02:11.951452  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:11.883985  4123 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:11.986511  1740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1516) took 2538 ms (client timeout 1550 ms). Trace:
I20260709 13:02:11.654639  4409 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:11.992626  1740 rpcz_store.cc:269] 0709 13:02:09.414799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.536855 (+122056us) service_pool.cc:224] Handling call
0709 13:02:11.952912 (+2416057us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54003534d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:11.953065 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028485888}
W20260709 13:02:11.951148  4390 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.497s	user 0.000s	sys 0.002s
I20260709 13:02:11.988166  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:12.022500  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.023751  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:12.024467  1734 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:11.951355  4274 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:11.952286  4385 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:11.993343  4448 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.099s
I20260709 13:02:11.976188  4119 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:12.069068  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:11.992515  4123 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:11.895746  4334 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:11.904351  4263 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.115s
I20260709 13:02:12.054078  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:12.106652  1593 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:11.976701  4255 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.088406  1740 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:11.953392  4312 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:11.941661  4296 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.011518  4277 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:12.062487  4412 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:12.068630  4404 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.403s	user 0.001s	sys 0.000s
I20260709 13:02:11.896214  4424 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.071295  4318 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:12.063155  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 591) took 1205 ms. Trace:
I20260709 13:02:12.112814  4184 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:12.063023  4426 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:12.071200  4274 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:12.171641  4445 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.172072  4407 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:12.173940  4419 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:02:12.174285  4428 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.290083  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2089) took 2699 ms. Trace:
I20260709 13:02:12.290596  1733 rpcz_store.cc:276] 0709 13:02:09.590322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.914749 (+1324427us) service_pool.cc:224] Handling call
0709 13:02:10.915241 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.944450 (+ 29209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.944486 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.945213 (+   727us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.389276 (+444063us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:11.389319 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.389400 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.289821 (+900421us) raft_consensus.cc:1685] finished
0709 13:02:12.289843 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.290012 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12405,"raft.queue_time_us":20140,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":13567,"thread_start_us":335930,"threads_started":1,"wal-append.queue_time_us":936076}
I20260709 13:02:12.293164  4334 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.295243  4409 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:12.174893  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2084) took 2656 ms (client timeout 2060 ms). Trace:
W20260709 13:02:12.178243  4418 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.267s	user 0.000s	sys 0.000s
W20260709 13:02:12.179566  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1519) took 2482 ms (client timeout 1714 ms). Trace:
I20260709 13:02:12.182814  4431 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:12.096446  4123 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:12.174314  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 360) took 3569 ms. Trace:
I20260709 13:02:12.308048  1588 rpcz_store.cc:276] 0709 13:02:08.605332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.950158 (+2344826us) service_pool.cc:224] Handling call
0709 13:02:11.010868 (+ 60710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.266732 (+255864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.174002 (+907270us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:12.174241 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":562289408}
I20260709 13:02:12.187222  1737 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:12.188758  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:12.196692  1708 connection.cc:256] RPC call timeout handler was delayed by -1.3511929920000512s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
I20260709 13:02:12.257789  4226 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:12.248091  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2051) took 3668 ms. Trace:
I20260709 13:02:12.250962  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:12.255555  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:12.283214  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.279428  4445 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:12.296335  1738 rpcz_store.cc:269] 0709 13:02:09.518438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.566263 (+1047825us) service_pool.cc:224] Handling call
0709 13:02:12.174596 (+1608333us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54003511d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:12.174712 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369424640}
I20260709 13:02:12.250454  4255 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:12.303133  1665 rpcz_store.cc:269] 0709 13:02:09.697645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.698778 (+  1133us) service_pool.cc:224] Handling call
0709 13:02:12.179307 (+2480529us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:12.179460 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118197504}
I20260709 13:02:12.265249  4184 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:12.233736  4393 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:12.309813  1708 connection.cc:256] RPC call timeout handler was delayed by -1.2169970109999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1724s for a response
I20260709 13:02:12.306010  4431 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:12.316381  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:12.316488  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:12.334296  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1508) took 2943 ms. Trace:
I20260709 13:02:12.379778  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:12.299014  4403 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:12.275354  4441 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.323141  4447 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:12.343200  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2052) took 3113 ms (client timeout 3309 ms). Trace:
W20260709 13:02:12.382447  1664 rpcz_store.cc:269] 0709 13:02:09.230296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.230580 (+   284us) service_pool.cc:224] Handling call
0709 13:02:12.323404 (+3092824us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:12.343089 (+ 19685us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822331776}
W20260709 13:02:12.305732  3939 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.385s	user 0.014s	sys 0.002s
I20260709 13:02:12.394299  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:12.390578  3939 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.387995  4447 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.398739  3939 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:12.400166  4447 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:12.361847  1740 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:12.407226  4405 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.749s	user 0.010s	sys 0.000s
I20260709 13:02:12.407445  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 370) took 1240 ms. Trace:
I20260709 13:02:12.407675  4405 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:12.407826  1740 rpcz_store.cc:276] 0709 13:02:11.166780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.154016 (+987236us) service_pool.cc:224] Handling call
0709 13:02:12.407366 (+253350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.362349  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:12.359930  1590 rpcz_store.cc:276] 0709 13:02:08.579631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.748080 (+2168449us) service_pool.cc:224] Handling call
0709 13:02:10.748617 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.748945 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.748983 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.754946 (+  5963us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.144239 (+389293us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:11.144278 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.144349 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.205824 (+1061475us) raft_consensus.cc:1685] finished
0709 13:02:12.247455 (+ 41631us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.247917 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":327489,"threads_started":1,"wal-append.queue_time_us":668709}
W20260709 13:02:12.364615  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.308382  4409 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:12.316279  4428 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.371815  4274 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.097457  4448 consensus_queue.cc:1243] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 3.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:12.375999  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.431818  1665 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:12.263633  4412 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.384558  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:12.383610  4318 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.436333  4318 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.436534  4431 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.437884  4431 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:12.438310  4037 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.438634  4318 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:12.253587  2310 rpcz_store.cc:276] 0709 13:02:10.846928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.862580 (+ 15652us) service_pool.cc:224] Handling call
0709 13:02:10.927086 (+ 64506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.998240 (+ 71154us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:11.581076 (+582836us) log.cc:844] Serialized 14 byte log entry
0709 13:02:11.951727 (+370651us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.986366 (+ 34639us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:11.993549 (+  7183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.021264 (+ 27715us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.052375 (+ 31111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12974,"prepare.queue_time_us":366443,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":30219,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":13166,"thread_start_us":626832,"threads_started":2}
W20260709 13:02:12.418197  3405 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.622s	user 0.014s	sys 0.002s
I20260709 13:02:12.441150  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1519) took 2864 ms. Trace:
I20260709 13:02:12.442009  1592 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 13:02:12.442521  1592 pending_rounds.cc:85] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 1
I20260709 13:02:12.442862  1592 pending_rounds.cc:107] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260709 13:02:12.441596  1736 rpcz_store.cc:276] 0709 13:02:09.576905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.870223 (+1293318us) service_pool.cc:224] Handling call
0709 13:02:10.908348 (+ 38125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:10.928308 (+ 19960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:10.928359 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:10.953088 (+ 24729us) log.cc:844] Serialized 26 byte log entry
0709 13:02:11.305289 (+352201us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:11.305364 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:11.305484 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:12.440778 (+1135294us) raft_consensus.cc:1685] finished
0709 13:02:12.440804 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.441061 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15857,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":65488,"thread_start_us":260164,"threads_started":1,"wal-append.queue_time_us":669762}
I20260709 13:02:12.443150  1592 raft_consensus.cc:2889] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:12.446306  1666 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:12.409466  4366 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.780s	user 0.019s	sys 0.028s
W20260709 13:02:12.403647  4181 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.605s	user 0.009s	sys 0.007s
I20260709 13:02:12.376544  4334 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.376302  4401 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.376065  4349 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:12.469269  4334 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.472556  4334 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.446079  4308 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.478364  4312 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:12.391073  2309 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:12.411202  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1504) took 3097 ms. Trace:
I20260709 13:02:12.430732  4409 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:12.479063  1661 rpcz_store.cc:276] 0709 13:02:09.314155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.314366 (+   211us) service_pool.cc:224] Handling call
0709 13:02:09.324471 (+ 10105us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:11.387945 (+2063474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.410931 (+1022986us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:12.411137 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4712399616}
I20260709 13:02:12.479717  4409 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:12.480517  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.481170  1661 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:12.415766  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2046) took 3910 ms. Trace:
I20260709 13:02:12.481665  1585 rpcz_store.cc:276] 0709 13:02:08.505227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.590560 (+2085333us) service_pool.cc:224] Handling call
0709 13:02:10.591132 (+   572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.971425 (+380293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.415485 (+1444060us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b540031e554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:12.415698 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":872889088}
I20260709 13:02:12.483330  1585 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:12.482966  4096 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.483925  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 369) took 3625 ms. Trace:
I20260709 13:02:12.484289  1585 rpcz_store.cc:276] 0709 13:02:08.858165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.482600 (+3624435us) service_pool.cc:224] Handling call
0709 13:02:12.482977 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.483650 (+   673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.483843 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.406090  2315 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:12.418340  1590 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:12.379209  1667 rpcz_store.cc:276] 0709 13:02:09.391242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.391368 (+   126us) service_pool.cc:224] Handling call
0709 13:02:09.391704 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:09.517926 (+126222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.323081 (+2805155us) spinlock_profiling.cc:243] Waited 36.5 ms on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:12.334186 (+ 11105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":83899648}
I20260709 13:02:12.435231  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:12.443022  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.447614  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2038) took 3976 ms. Trace:
W20260709 13:02:12.380153  4161 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.065s	sys 0.041s
I20260709 13:02:12.382298  4373 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.478041  4308 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:12.380044  4394 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.044s	sys 0.057s
I20260709 13:02:12.373992  4255 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.414356  4321 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:12.486403  1585 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:12.615504  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 370) took 3751 ms. Trace:
I20260709 13:02:12.547725  2315 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:12.392355  4441 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:12.622439  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 238001us
I20260709 13:02:12.429379  4385 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:12.621133  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:12.631101  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:12.631601  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 368) took 3775 ms. Trace:
I20260709 13:02:12.632234  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 640) took 1783 ms. Trace:
I20260709 13:02:12.568686  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:12.632653  1593 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.001s
I20260709 13:02:12.632881  1585 rpcz_store.cc:276] 0709 13:02:08.864184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.484992 (+3620808us) service_pool.cc:224] Handling call
0709 13:02:12.485328 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.615196 (+129868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.615434 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.629757  1592 rpcz_store.cc:276] 0709 13:02:08.470761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.473607 (+  2846us) service_pool.cc:224] Handling call
0709 13:02:09.244496 (+770889us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:02:09.244774 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.444065 (+3199291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.445319 (+  1254us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b540025e754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:12.447482 (+  2163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7385782272}
I20260709 13:02:12.654356  1590 rpcz_store.cc:276] 0709 13:02:08.856512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.417774 (+3561262us) service_pool.cc:224] Handling call
0709 13:02:12.418154 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.631342 (+213188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.631545 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.655408  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:12.625440  4394 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.974s	user 0.047s	sys 0.062s
I20260709 13:02:12.658277  4394 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:12.657037  1593 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:12.562482  4396 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 13:02:12.655544  2315 rpcz_store.cc:276] 0709 13:02:10.848850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.850765 (+  1915us) service_pool.cc:224] Handling call
0709 13:02:10.851061 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.405793 (+1554732us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b580061f1d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:02:12.631985 (+226192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.632168 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420217600}
W20260709 13:02:12.561650  4415 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.969s	user 0.006s	sys 0.007s
W20260709 13:02:12.552691  4063 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.364s	user 0.012s	sys 0.003s
I20260709 13:02:12.660074  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.613185  4412 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:12.351374  1734 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:12.658438  4318 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:12.730064  4318 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:12.730502  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:12.556363  4445 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:12.669631  1587 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.971s	user 0.071s	sys 0.092s
W20260709 13:02:12.737296  1587 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.039s	user 0.072s	sys 0.093s
I20260709 13:02:12.666446  1585 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:12.739692  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2196) took 3292 ms. Trace:
I20260709 13:02:12.362757  4226 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.519675  4450 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:12.638509  4161 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.071s	sys 0.047s
I20260709 13:02:12.721220  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.727072  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2101) took 1456 ms. Trace:
I20260709 13:02:12.755800  1734 rpcz_store.cc:276] 0709 13:02:11.270352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.991192 (+720840us) service_pool.cc:224] Handling call
0709 13:02:12.726998 (+735806us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.727907  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 363) took 1914 ms. Trace:
I20260709 13:02:12.760465  1665 rpcz_store.cc:276] 0709 13:02:10.813760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.659490 (+1845730us) service_pool.cc:224] Handling call
0709 13:02:12.727236 (+ 67746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.727497 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.727533 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.727594 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:12.727637 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.727706 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.727835 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.518347  4424 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:12.661391  1742 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.697s	user 0.012s	sys 0.000s
I20260709 13:02:12.732442  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2192) took 3467 ms. Trace:
I20260709 13:02:12.763082  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.741022  1585 rpcz_store.cc:276] 0709 13:02:09.447387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.665665 (+3218278us) service_pool.cc:224] Handling call
0709 13:02:12.666051 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.739373 (+ 73322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.739631 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.700209  1590 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:12.742125  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:12.743144  4427 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 13:02:12.727638  4415 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:12.612957  4123 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.743419  4394 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:12.594255  4387 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.101s	user 0.015s	sys 0.000s
I20260709 13:02:12.518494  4181 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:12.757951  4430 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
W20260709 13:02:12.740556  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 109556us
I20260709 13:02:12.765257  1742 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:12.764032  1592 rpcz_store.cc:276] 0709 13:02:09.264750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.674134 (+3409384us) service_pool.cc:224] Handling call
0709 13:02:12.728815 (+ 54681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.729214 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.729254 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.729308 (+    54us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:12.729354 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.729425 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.732356 (+  2931us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2181,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":3936}
I20260709 13:02:12.768803  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:12.770444  1667 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:12.740237  4385 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:12.774150  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2195) took 3327 ms. Trace:
I20260709 13:02:12.603826  4184 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.732096  4063 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.800699  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.802161  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:12.813906  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.344s	user 0.011s	sys 0.053s
W20260709 13:02:12.858595  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.389s	user 0.011s	sys 0.054s
I20260709 13:02:12.826581  1734 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:12.826637  1634 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.051s	user 0.013s	sys 0.022s
I20260709 13:02:12.828064  2315 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 13:02:12.862860  1634 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.013s	sys 0.023s
I20260709 13:02:12.863011  2315 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:12.863442  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.08791e+06us
W20260709 13:02:12.864634  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:12.829173  1708 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:12.798784  4394 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:12.865149  1708 leader_election.cc:403] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:12.838126  1590 rpcz_store.cc:276] 0709 13:02:09.447084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.699503 (+3252419us) service_pool.cc:224] Handling call
0709 13:02:12.699857 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.772711 (+ 72854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.774068 (+  1357us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:12.808349  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2104) took 1507 ms (client timeout 1823 ms). Trace:
I20260709 13:02:12.857977  1665 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:12.865888  1742 rpcz_store.cc:269] 0709 13:02:11.301120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.952077 (+650957us) service_pool.cc:224] Handling call
0709 13:02:12.808253 (+856176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.861276  1661 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:12.864066  2315 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:12.869326  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.093s	user 0.007s	sys 0.021s
W20260709 13:02:12.873543  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.097s	user 0.007s	sys 0.021s
I20260709 13:02:12.874904  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:12.906000  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.280s	user 0.009s	sys 0.016s
W20260709 13:02:12.906726  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.280s	user 0.010s	sys 0.016s
I20260709 13:02:12.759759  4318 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:12.912815  2315 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:12.924611  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2081) took 3431 ms. Trace:
I20260709 13:02:12.925246  1590 rpcz_store.cc:276] 0709 13:02:09.492771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.866054 (+3373283us) service_pool.cc:224] Handling call
0709 13:02:12.866402 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.924140 (+ 57738us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.924185 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:12.924246 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:12.924294 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.924367 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:12.924525 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.855371  1592 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:02:12.934453  4255 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:12.934774  4349 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:12.939286  1585 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:12.940726  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2078) took 3449 ms. Trace:
I20260709 13:02:12.941282  1592 rpcz_store.cc:276] 0709 13:02:09.491853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.808581 (+3316728us) service_pool.cc:224] Handling call
0709 13:02:12.854661 (+ 46080us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.940396 (+ 85735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.940644 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.954098  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2080) took 3461 ms. Trace:
I20260709 13:02:12.955338  1585 rpcz_store.cc:276] 0709 13:02:09.492474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.844173 (+3351699us) service_pool.cc:224] Handling call
0709 13:02:12.938441 (+ 94268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.940058 (+  1617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:12.953683 (+ 13625us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:12.957440  1664 raft_consensus.cc:2368] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
W20260709 13:02:12.857285  4363 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.587s	user 0.020s	sys 0.018s
I20260709 13:02:12.936278  4161 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:12.976621  1662 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:12.867638  4394 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:12.985194  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:12.807749  4387 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:12.968014  4417 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:02:13.000276  4161 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:13.003752  4379 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:13.002875  4417 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:13.004987  4417 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:13.006129  4389 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:13.006646  4389 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:12.977484  4363 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.713s	user 0.021s	sys 0.018s
W20260709 13:02:13.015062  4364 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260709 13:02:13.241873  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2083) took 3725 ms (client timeout 2059 ms). Trace:
W20260709 13:02:13.517688  2281 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.199s	user 0.000s	sys 0.030s
I20260709 13:02:13.786568  1584 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
W20260709 13:02:13.785526  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 956638us
W20260709 13:02:13.697870  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.314s	user 0.019s	sys 0.023s
W20260709 13:02:13.521873  1660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.224s	user 0.020s	sys 0.020s
W20260709 13:02:13.758246  1741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.735s	user 0.036s	sys 0.016s
I20260709 13:02:13.044708  2316 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:13.792910  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 918074us
W20260709 13:02:13.517207  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:13.801440  2281 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.482s	user 0.001s	sys 0.036s
W20260709 13:02:13.802546  1660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.505s	user 0.023s	sys 0.022s
W20260709 13:02:13.071131  1488 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.007s	user 0.014s	sys 0.031s
W20260709 13:02:13.787209  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3425591529999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2086s for a response
W20260709 13:02:13.802721  1741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.780s	user 0.037s	sys 0.016s
W20260709 13:02:13.044858  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.000s	sys 0.043s
W20260709 13:02:13.415357  1734 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.553s	user 0.008s	sys 0.004s
I20260709 13:02:13.293991  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 360) took 2534 ms. Trace:
I20260709 13:02:13.801676  1584 pending_rounds.cc:85] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
W20260709 13:02:13.068486  1591 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.433s	user 0.001s	sys 0.015s
I20260709 13:02:13.795176  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:13.815611  1584 pending_rounds.cc:107] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260709 13:02:13.801285  1739 rpcz_store.cc:269] 0709 13:02:09.516629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.050149 (+533520us) service_pool.cc:224] Handling call
0709 13:02:13.241623 (+3191474us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b540039dfd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:13.241787 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":735850240}
I20260709 13:02:13.786490  1590 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:13.817827  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2082) took 3052 ms. Trace:
I20260709 13:02:13.818360  1734 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
W20260709 13:02:13.819120  1591 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.191s	user 0.008s	sys 0.030s
I20260709 13:02:13.819199  1742 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:13.814868  1662 rpcz_store.cc:276] 0709 13:02:10.759312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.882865 (+1123553us) service_pool.cc:224] Handling call
0709 13:02:11.883299 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.992461 (+1109162us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.293562 (+301101us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54002f1f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:13.293892 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2499370752}
W20260709 13:02:13.802373  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.419s	user 0.021s	sys 0.023s
I20260709 13:02:13.821154  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:13.818236  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:13.822293  1589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2073) took 4310 ms (client timeout 1527 ms). Trace:
W20260709 13:02:13.804479  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.49779e+06us
I20260709 13:02:13.819986  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1524) took 1167 ms. Trace:
I20260709 13:02:13.823863  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2072) took 1169 ms. Trace:
W20260709 13:02:13.770880  4453 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.735s	user 0.001s	sys 0.001s
W20260709 13:02:13.820096  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.708s	user 0.000s	sys 0.065s
W20260709 13:02:13.812562  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.822183  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:13.459913  4327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.275s	user 0.007s	sys 0.011s
I20260709 13:02:13.824757  1734 rpcz_store.cc:276] 0709 13:02:12.653011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.768423 (+115412us) service_pool.cc:224] Handling call
0709 13:02:13.819924 (+1051501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.825030  1661 rpcz_store.cc:276] 0709 13:02:12.653982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.798748 (+1144766us) service_pool.cc:224] Handling call
0709 13:02:13.823794 (+ 25046us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:13.816088  1584 raft_consensus.cc:2889] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:13.821410  1488 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.026s	sys 0.040s
I20260709 13:02:13.818241  1590 rpcz_store.cc:276] 0709 13:02:10.765980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.926410 (+2160430us) service_pool.cc:224] Handling call
0709 13:02:13.786095 (+859685us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.817514 (+ 31419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.817765 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:13.831369  1733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.535s	user 0.012s	sys 0.011s
W20260709 13:02:13.241850  4451 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.329s	user 0.001s	sys 0.000s
W20260709 13:02:13.712167  4417 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.707s	user 0.007s	sys 0.006s
W20260709 13:02:13.823593  1589 rpcz_store.cc:269] 0709 13:02:09.483083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:09.517019 (+ 33936us) service_pool.cc:224] Handling call
0709 13:02:13.284448 (+3767429us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5400443754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:13.793239 (+508791us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307368832}
I20260709 13:02:13.819981  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 363) took 5149 ms. Trace:
W20260709 13:02:13.223837   436 debug-util.cc:398] Leaking SignalData structure 0x7b080000ff80 after lost signal to thread 1556
W20260709 13:02:13.059144  4304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.988s	user 0.004s	sys 0.029s
W20260709 13:02:13.238360  4409 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.758s	user 0.011s	sys 0.000s
I20260709 13:02:13.898550  1589 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:13.899566  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2088) took 3110 ms. Trace:
W20260709 13:02:13.847265  4327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.655s	user 0.009s	sys 0.014s
I20260709 13:02:13.823751  1662 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:13.903931  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2077) took 1237 ms. Trace:
I20260709 13:02:13.904402  1662 rpcz_store.cc:276] 0709 13:02:12.666118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.820890 (+1154772us) service_pool.cc:224] Handling call
0709 13:02:13.903845 (+ 82955us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:13.842787  1733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.547s	user 0.012s	sys 0.011s
W20260709 13:02:13.836818  4274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.258s	user 0.000s	sys 0.047s
I20260709 13:02:13.816545  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2078) took 1150 ms. Trace:
W20260709 13:02:13.866398  1740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.244s	user 0.024s	sys 0.020s
W20260709 13:02:13.845207  1705 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.025s	sys 0.012s
I20260709 13:02:13.868224  1590 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260709 13:02:13.870393  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:13.609521  4383 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.269s	user 0.009s	sys 0.022s
W20260709 13:02:13.556821  4446 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.693s	user 0.002s	sys 0.000s
I20260709 13:02:13.905603  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:13.872380  1587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.068s	user 0.003s	sys 0.018s
I20260709 13:02:13.872613  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:13.798714  4226 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:13.711531  4387 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:13.235463  4184 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:13.567348  4181 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:13.685251  4431 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.232s	user 0.012s	sys 0.003s
I20260709 13:02:13.556489  4441 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:13.817545  4319 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:13.897595  2318 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.892s	user 0.006s	sys 0.014s
I20260709 13:02:13.979267  4226 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:13.835083  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:13.899998  1589 rpcz_store.cc:276] 0709 13:02:10.788910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.897837 (+3108927us) service_pool.cc:224] Handling call
0709 13:02:13.898300 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.899205 (+   905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.899492 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:14.020203  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.021101  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2092) took 2241 ms. Trace:
W20260709 13:02:13.478821  4281 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.806s	user 0.005s	sys 0.023s
W20260709 13:02:13.901436  1668 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.013s	sys 0.033s
I20260709 13:02:13.978094  1661 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:13.896602  1591 rpcz_store.cc:276] 0709 13:02:08.671314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.972698 (+2301384us) service_pool.cc:224] Handling call
0709 13:02:10.973142 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:10.973433 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:11.269205 (+295772us) log.cc:844] Serialized 14 byte log entry
0709 13:02:13.819637 (+2550432us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:13.819695 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:13.819738 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.819818 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:13.819920 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":733694,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":36671,"raft.queue_time_us":401659,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":37029,"thread_start_us":2839972,"threads_started":3,"wal-append.queue_time_us":834652,"wal-append.run_cpu_time_us":1416,"wal-append.run_wall_time_us":1164524}
W20260709 13:02:13.979988  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.124s	user 0.018s	sys 0.029s
W20260709 13:02:13.979624  1587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.175s	user 0.008s	sys 0.020s
I20260709 13:02:13.981084  1662 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:13.983592  1667 rpcz_store.cc:276] 0709 13:02:12.666383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.835927 (+169544us) service_pool.cc:224] Handling call
0709 13:02:13.816430 (+980503us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:13.984189  1705 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.315s	user 0.038s	sys 0.017s
I20260709 13:02:13.983834  1734 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
W20260709 13:02:13.977732  1740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.356s	user 0.024s	sys 0.028s
I20260709 13:02:13.867242  4035 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:13.893949  4249 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.449s	user 0.017s	sys 0.026s
I20260709 13:02:13.856448  4161 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:13.819868  4406 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:14.013576  2318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.008s	user 0.008s	sys 0.014s
I20260709 13:02:14.098923  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 362) took 3313 ms. Trace:
W20260709 13:02:13.898622  4304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.828s	user 0.009s	sys 0.046s
W20260709 13:02:13.842794  4394 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.839s	user 0.014s	sys 0.000s
I20260709 13:02:13.034493  4385 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.102213  1663 rpcz_store.cc:276] 0709 13:02:10.786047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.042956 (+1256909us) service_pool.cc:224] Handling call
0709 13:02:12.087180 (+ 44224us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:12.097631 (+ 10451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.097842 (+   211us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:12.098940 (+  1098us) log.cc:844] Serialized 52 byte log entry
0709 13:02:13.833633 (+1734693us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:13.833699 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.833838 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.098645 (+264807us) raft_consensus.cc:1685] finished
0709 13:02:14.098669 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.098849 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17492,"raft.queue_time_us":15540,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":15759,"thread_start_us":1720893,"threads_started":1,"wal-append.queue_time_us":1984242}
I20260709 13:02:14.159596  1663 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:14.160207  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:14.104164  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:14.011427  4383 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.691s	user 0.014s	sys 0.036s
W20260709 13:02:14.161331  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57275.3us
I20260709 13:02:14.161713  4383 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:14.162264  4383 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:14.162727  4383 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:14.104257  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.4352e+06us
I20260709 13:02:14.164886  2314 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:14.167068  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 639) took 3318 ms. Trace:
I20260709 13:02:14.167528  4319 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.167488  2314 rpcz_store.cc:276] 0709 13:02:10.848570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.851599 (+  3029us) service_pool.cc:224] Handling call
0709 13:02:10.869601 (+ 18002us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.165429 (+3295828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.166752 (+  1323us) spinlock_profiling.cc:243] Waited 3.29 s on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:14.166981 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7578430208}
I20260709 13:02:14.168190  4319 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.018862  4181 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.169785  4181 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.170292  2309 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEARNER]: Advancing to term 7
I20260709 13:02:14.170729  4319 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:14.171288  4319 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.153640  4035 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.173827  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59800 (request call id 0) took 3316 ms. Trace:
I20260709 13:02:14.174240  2312 rpcz_store.cc:276] 0709 13:02:10.867074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.878870 (+ 11796us) service_pool.cc:224] Handling call
0709 13:02:14.173398 (+3294528us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b54006a2320. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf5a520 00007fcdedfd7652 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:14.173737 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21788416}
I20260709 13:02:13.981303  4063 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:14.105288  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.250s	user 0.023s	sys 0.035s
I20260709 13:02:14.104957  1591 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:14.175882  4458 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:14.176048  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 2.49531e+06us
I20260709 13:02:14.176352  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2093) took 2396 ms. Trace:
I20260709 13:02:14.176579  4458 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.100414  1589 rpcz_store.cc:276] 0709 13:02:11.779928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.020026 (+2240098us) service_pool.cc:224] Handling call
0709 13:02:14.020431 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.020654 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.020690 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.020751 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.020795 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.020865 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.021003 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.016212  4123 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:14.178896  1589 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260709 13:02:13.934703   436 kernel_stack_watchdog.cc:198] Thread 1556 stuck at /apache/dev/git/kudu/src/kudu/rpc/outbound_call.cc:307 for 490ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260709 13:02:14.179503  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2094) took 2399 ms. Trace:
I20260709 13:02:14.009899  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2087) took 3219 ms. Trace:
I20260709 13:02:14.175931  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 365) took 3313 ms. Trace:
I20260709 13:02:13.911708  4409 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:14.117889  4281 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.461s	user 0.007s	sys 0.037s
W20260709 13:02:14.102789  4249 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.658s	user 0.022s	sys 0.038s
I20260709 13:02:14.121304  4161 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:13.902981  4417 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.123102  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:14.192075  1661 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:14.160468  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 366) took 3250 ms. Trace:
I20260709 13:02:14.174671  4035 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.105540  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 376) took 1151 ms. Trace:
W20260709 13:02:13.977732  4274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.399s	user 0.000s	sys 0.051s
I20260709 13:02:13.910341  4454 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.176998  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:14.100464  1668 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.374s	user 0.016s	sys 0.044s
I20260709 13:02:14.179910  1589 rpcz_store.cc:276] 0709 13:02:11.780478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.178161 (+2397683us) service_pool.cc:224] Handling call
0709 13:02:14.178504 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.179261 (+   757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.179448 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.176839  1591 rpcz_store.cc:276] 0709 13:02:11.780197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.104003 (+2323806us) service_pool.cc:224] Handling call
0709 13:02:14.104447 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.176092 (+ 71645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.176295 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.181030  1667 rpcz_store.cc:276] 0709 13:02:10.872004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.121104 (+3249100us) service_pool.cc:224] Handling call
0709 13:02:14.175371 (+ 54267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.175582 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.175611 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.175667 (+    56us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.175704 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.175764 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.175870 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.180264  1590 rpcz_store.cc:276] 0709 13:02:10.788622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.865943 (+3077321us) service_pool.cc:224] Handling call
0709 13:02:13.867863 (+  1920us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:14.007575 (+139712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.007942 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.088299  4441 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.121649  4184 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:14.122264  1662 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:02:14.123909  2316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 645) took 1818 ms (client timeout 2121 ms). Trace:
I20260709 13:02:14.192513  1663 rpcz_store.cc:276] 0709 13:02:10.910483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.158888 (+3248405us) service_pool.cc:224] Handling call
0709 13:02:14.159341 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.160136 (+   795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.160400 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.194304  1734 rpcz_store.cc:276] 0709 13:02:12.954113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.869876 (+915763us) service_pool.cc:224] Handling call
0709 13:02:14.105456 (+235580us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.026484  4387 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.159s
I20260709 13:02:14.207168  1735 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:14.026530  4226 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.207646  1588 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:14.179520  4458 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:14.171741  4319 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.124604  4394 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.190068  4181 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.122341  4406 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.222573  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:14.223961  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2102) took 2930 ms (client timeout 1815 ms). Trace:
I20260709 13:02:14.235853  4308 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.238255  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2095) took 4530 ms. Trace:
I20260709 13:02:14.543200  1740 rpcz_store.cc:276] 0709 13:02:09.707692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.408595 (+2700903us) service_pool.cc:224] Handling call
0709 13:02:12.409085 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.621049 (+211964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.621090 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.621772 (+   682us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.105734 (+1483962us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:14.105794 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.105885 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.237898 (+132013us) raft_consensus.cc:1685] finished
0709 13:02:14.237932 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.238166 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":290136,"raft.queue_time_us":228537,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":5790,"thread_start_us":1483492,"threads_started":1,"wal-append.queue_time_us":1613799}
I20260709 13:02:14.545082  4383 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:14.239128  4399 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.971s	user 0.010s	sys 0.008s
W20260709 13:02:14.240375  4466 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:02:14.206853  4454 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:14.241915  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.210867  4161 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:14.243775  1736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.473s	user 0.033s	sys 0.062s
I20260709 13:02:14.242583  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.287766  4417 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:14.291002  4472 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
W20260709 13:02:14.291827  4465 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
W20260709 13:02:14.293947  4468 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 13:02:14.294870  4409 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:14.298146  4470 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
W20260709 13:02:14.298820  4096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.735s	user 0.032s	sys 0.045s
W20260709 13:02:14.313306  1593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.544s	user 0.038s	sys 0.037s
W20260709 13:02:14.327997  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.003s	sys 0.033s
W20260709 13:02:14.328640  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.017s	sys 0.020s
I20260709 13:02:14.324918  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:14.331724  1557 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.005s	sys 0.032s
W20260709 13:02:14.332734  1660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.010s	sys 0.027s
I20260709 13:02:14.321717  4037 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:02:14.324332  4249 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.010s
W20260709 13:02:14.415014  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.436007  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.016s	sys 0.077s
W20260709 13:02:14.439864  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.000s	sys 0.011s
W20260709 13:02:14.441342  1741 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.020s	sys 0.022s
W20260709 13:02:14.440821  4479 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 13:02:14.523341  4184 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:14.444317  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.000s	sys 0.011s
W20260709 13:02:14.445808  4475 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 13:02:14.449863  2309 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEARNER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:14.451216  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:14.446329  4482 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:14.554744  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:14.457356  2316 rpcz_store.cc:269] 0709 13:02:12.305762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.305934 (+   172us) service_pool.cc:224] Handling call
0709 13:02:14.108052 (+1802118us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b540059e7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:14.123839 (+ 15787us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1516992640}
W20260709 13:02:14.557689  1736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.795s	user 0.034s	sys 0.062s
I20260709 13:02:14.522492  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 364) took 3683 ms. Trace:
I20260709 13:02:14.592696  1662 rpcz_store.cc:276] 0709 13:02:10.838884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.121193 (+3282309us) service_pool.cc:224] Handling call
0709 13:02:14.121795 (+   602us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.457125 (+335330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.522395 (+ 65270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.593281  4426 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:14.594014  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.594609  1662 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:14.592852  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2096) took 4792 ms. Trace:
I20260709 13:02:14.595559  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.595255  1736 rpcz_store.cc:276] 0709 13:02:09.800971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.443499 (+2642528us) service_pool.cc:224] Handling call
0709 13:02:12.443918 (+   419us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.670720 (+226802us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.670768 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.719146 (+ 48378us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.592468 (+1873322us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:14.592511 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.592606 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.592651 (+    45us) raft_consensus.cc:1685] finished
0709 13:02:14.592661 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.592797 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":208870,"raft.queue_time_us":227090,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":18209,"thread_start_us":1829516,"threads_started":1,"wal-append.queue_time_us":1689825}
I20260709 13:02:14.597234  1662 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:14.598134  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2061) took 2606 ms. Trace:
I20260709 13:02:14.598450  1662 rpcz_store.cc:276] 0709 13:02:11.992285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.596632 (+2604347us) service_pool.cc:224] Handling call
0709 13:02:14.596910 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.597558 (+   648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.598073 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.599774  1662 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:02:14.600265  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2062) took 2439 ms. Trace:
I20260709 13:02:14.600363  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.600600  1662 rpcz_store.cc:276] 0709 13:02:12.160643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.599202 (+2438559us) service_pool.cc:224] Handling call
0709 13:02:14.599478 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.600045 (+   567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.600218 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.514132  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 359) took 3755 ms. Trace:
W20260709 13:02:14.520300  1585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.039s	sys 0.114s
I20260709 13:02:14.526378  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:14.601765  1585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.040s	sys 0.114s
I20260709 13:02:14.602483  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2085) took 3814 ms. Trace:
I20260709 13:02:14.601598  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:14.538296  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:14.540906  1737 rpcz_store.cc:269] 0709 13:02:11.293793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.777318 (+483525us) service_pool.cc:224] Handling call
0709 13:02:14.223746 (+2446428us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b5400435654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:14.223883 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":940147456}
I20260709 13:02:14.634776  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.635314  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.635975  1737 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:14.637207  1734 raft_consensus.cc:2410] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260709 13:02:14.520076  4464 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:14.639748  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:14.520740  4441 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.638954  4383 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:14.643113  4383 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:14.552351  1739 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:14.646214  4441 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:14.552933  4281 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.647348  4281 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.553399  4304 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.553606  4394 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:14.591475  4482 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:14.650379  4482 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:14.650082  4304 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.652124  1591 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 13:02:14.513945  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 337067us
I20260709 13:02:14.654101  4304 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:14.654907  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:14.655871  4304 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:14.602927  1585 rpcz_store.cc:276] 0709 13:02:10.787964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.799580 (+3011616us) service_pool.cc:224] Handling call
0709 13:02:13.840906 (+ 41326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.841457 (+   551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.841514 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.842140 (+   626us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.602163 (+760023us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:14.602202 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.602265 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.602310 (+    45us) raft_consensus.cc:1685] finished
0709 13:02:14.602320 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.602423 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3060,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":35290,"thread_start_us":590742,"threads_started":1,"wal-append.queue_time_us":674785}
I20260709 13:02:14.536794  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:14.657203  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.657907  1585 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:14.658002  1667 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:14.651578  4482 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.604447  1668 rpcz_store.cc:276] 0709 13:02:10.759060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.191143 (+432083us) service_pool.cc:224] Handling call
0709 13:02:11.191465 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.191666 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:11.265566 (+ 73900us) log.cc:844] Serialized 14 byte log entry
0709 13:02:12.726289 (+1460723us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b18002af760. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:02:14.461017 (+1734728us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.513730 (+ 52713us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.513784 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.513857 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.514058 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":73714,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":7161,"raft.queue_time_us":360,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":4823,"spinlock_wait_cycles":4971904,"thread_start_us":1734073,"threads_started":1,"wal-append.queue_time_us":3063022}
W20260709 13:02:14.609299  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 284501us
I20260709 13:02:14.659436  4482 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:14.659976  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:14.662230  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:14.662650  4394 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:14.615959  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:14.663245  4394 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:14.663705  4394 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:14.664002  4464 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.664372  4441 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:14.664558  4464 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:14.558717  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.564553  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2093) took 4913 ms. Trace:
I20260709 13:02:14.666487  1733 rpcz_store.cc:276] 0709 13:02:09.651005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.292350 (+2641345us) service_pool.cc:224] Handling call
0709 13:02:12.292751 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.293203 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:12.293239 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.295219 (+  1980us) log.cc:844] Serialized 26 byte log entry
0709 13:02:13.906487 (+1611268us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:13.906548 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.906650 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.564031 (+657381us) raft_consensus.cc:1685] finished
0709 13:02:14.564188 (+   157us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.564448 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":765,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":153,"thread_start_us":1610578,"threads_started":1,"wal-append.queue_time_us":1910721}
I20260709 13:02:14.620237  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 367) took 5764 ms. Trace:
W20260709 13:02:14.572824  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.017s	sys 0.020s
W20260709 13:02:14.572515  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.017s	sys 0.077s
W20260709 13:02:14.668671  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 868775us
I20260709 13:02:14.669495  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2084) took 3901 ms. Trace:
I20260709 13:02:14.558558  4226 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:14.573035  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.011s	sys 0.028s
I20260709 13:02:14.575071  4486 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:14.575389  1741 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.020s	sys 0.022s
W20260709 13:02:14.576743  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.000s	sys 0.011s
W20260709 13:02:14.577559  4096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.013s	user 0.033s	sys 0.046s
W20260709 13:02:14.577436  1593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.808s	user 0.038s	sys 0.038s
I20260709 13:02:14.705276  1661 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:14.708012  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2077) took 5223 ms. Trace:
I20260709 13:02:14.708537  1593 rpcz_store.cc:276] 0709 13:02:09.484349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.699812 (+3215463us) service_pool.cc:224] Handling call
0709 13:02:12.700095 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.714522 (+ 14427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.714555 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.715095 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.707481 (+1992386us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:14.707525 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.707608 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.707673 (+    65us) raft_consensus.cc:1685] finished
0709 13:02:14.707692 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.707924 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5095,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":11357,"thread_start_us":1937798,"threads_started":1,"wal-append.queue_time_us":1700263}
W20260709 13:02:14.579907  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.004s	sys 0.034s
I20260709 13:02:14.581362  4308 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.584947  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2094) took 4911 ms. Trace:
I20260709 13:02:14.748888  1738 rpcz_store.cc:276] 0709 13:02:09.673614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.357841 (+2684227us) service_pool.cc:224] Handling call
0709 13:02:12.358295 (+   454us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:12.358754 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.358791 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:12.359517 (+   726us) log.cc:844] Serialized 52 byte log entry
0709 13:02:13.821072 (+1461555us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:13.821131 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:13.821225 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.584609 (+763384us) raft_consensus.cc:1685] finished
0709 13:02:14.584641 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.584864 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3075,"raft.queue_time_us":121,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":13988,"thread_start_us":1437564,"threads_started":1,"wal-append.queue_time_us":1746472}
I20260709 13:02:14.624202  1589 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:14.750638  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 361) took 3981 ms. Trace:
I20260709 13:02:14.751060  1660 rpcz_store.cc:276] 0709 13:02:10.768911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.901556 (+1132645us) service_pool.cc:224] Handling call
0709 13:02:11.902435 (+   879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:12.256211 (+353776us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:13.804773 (+1548562us) log.cc:844] Serialized 14 byte log entry
0709 13:02:14.750232 (+945459us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.750293 (+    61us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:14.750353 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.750435 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.750581 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43971,"prepare.queue_time_us":1979428,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":2408027,"threads_started":2,"wal-append.queue_time_us":648423}
I20260709 13:02:14.752166  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.585428  4319 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:14.589669  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.000s	sys 0.011s
I20260709 13:02:14.583647  4184 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:14.590878  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 647) took 1971 ms. Trace:
I20260709 13:02:14.626466  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2079) took 5118 ms. Trace:
W20260709 13:02:14.591269  2310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 651) took 1586 ms (client timeout 2013 ms). Trace:
I20260709 13:02:14.559149  4161 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.590095  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 283) took 3204 ms. Trace:
I20260709 13:02:14.622577  4428 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.764753  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:14.632664  4389 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.626s	user 0.019s	sys 0.000s
I20260709 13:02:14.540304  4381 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:14.786403  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 362) took 6163 ms. Trace:
I20260709 13:02:14.637812  1736 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:14.662765  4281 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:14.663434  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:14.786995  1584 rpcz_store.cc:276] 0709 13:02:08.622709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.969744 (+2347035us) service_pool.cc:224] Handling call
0709 13:02:13.786220 (+2816476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:13.841385 (+ 55165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.786111 (+944726us) spinlock_profiling.cc:243] Waited 2.82 s on lock 0x7b5400443750. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:14.786316 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6476894976}
W20260709 13:02:14.788327  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 124974us
I20260709 13:02:14.789151  1584 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:14.670337  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.617903  4325 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:14.572935  1557 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.005s	sys 0.032s
I20260709 13:02:14.668211  1588 rpcz_store.cc:276] 0709 13:02:08.855909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.410250 (+3554341us) service_pool.cc:224] Handling call
0709 13:02:12.410578 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.534539 (+2123961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.619959 (+ 85420us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540042e354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:14.620172 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4132666624}
I20260709 13:02:14.664208  1586 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:14.641391  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.545490  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.693125  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:14.794006  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2086) took 4005 ms. Trace:
W20260709 13:02:14.794516  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101570us
I20260709 13:02:14.794394  1586 rpcz_store.cc:276] 0709 13:02:10.788333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.616758 (+3828425us) service_pool.cc:224] Handling call
0709 13:02:14.663508 (+ 46750us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.793724 (+130216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.793934 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.796057  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.796658  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:14.748296  4486 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:14.644187  4383 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:02:14.707118  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 902027us
I20260709 13:02:14.750223  4226 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.755162  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.670001  1592 rpcz_store.cc:276] 0709 13:02:10.768352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.942384 (+2174032us) service_pool.cc:224] Handling call
0709 13:02:12.998384 (+ 56000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.042938 (+ 44554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.042992 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.801512 (+758520us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.668987 (+867475us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:14.669039 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.669160 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:14.669227 (+    67us) raft_consensus.cc:1685] finished
0709 13:02:14.669247 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.669419 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23391,"raft.queue_time_us":35435,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":8273,"thread_start_us":824712,"threads_started":1,"wal-append.queue_time_us":755107}
I20260709 13:02:14.759034  1735 rpcz_store.cc:276] 0709 13:02:09.508562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.831544 (+1322982us) service_pool.cc:224] Handling call
0709 13:02:10.831965 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.531973 (+3700008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.626211 (+ 94238us) spinlock_profiling.cc:243] Waited 3.37 s on lock 0x7b5400434c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:14.626405 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7762428800}
I20260709 13:02:14.761775  1667 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:14.630761  4296 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.759543  2313 rpcz_store.cc:276] 0709 13:02:11.386059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.386219 (+   160us) service_pool.cc:224] Handling call
0709 13:02:11.386559 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.551975 (+165416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:11.552030 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:11.552099 (+    69us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:11.552149 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.589826 (+3037677us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002e8454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:14.589889 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.590034 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379783424}
I20260709 13:02:14.768131  4161 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:14.779973  4389 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.759749  4488 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:14.781751  4426 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:14.747617  4249 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:14.784430  1693 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:39071 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:39071 timed out after 15.000s (SENT)
I20260709 13:02:14.786538  4381 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:14.660128  1668 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: [3.3-6.4]
I20260709 13:02:14.789490  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:14.871613  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82258.1us
W20260709 13:02:14.613298  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:14.793452  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 988477us
I20260709 13:02:14.753608  4037 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.806030  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.881590  1738 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:14.796881  1737 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:14.857792  1667 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:14.811419  4304 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:14.884455  4381 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.885861  4381 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:14.856846  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52127.4us
I20260709 13:02:14.890475  4409 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:14.892004  4409 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:14.892421  4409 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:14.899075  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:14.901429  1736 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:14.898792  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.860396  1733 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:14.747772  4387 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:02:14.857278  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.910218  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:14.861217  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:14.904340  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:14.911283  1734 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:14.766844  4319 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:14.754699  4308 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.811924  4281 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:14.907982  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.859772  4161 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:14.919266  1586 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:14.883299  1667 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.908573  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:14.910344  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:14.762181  2310 rpcz_store.cc:269] 0709 13:02:13.004708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.450821 (+1446113us) service_pool.cc:224] Handling call
0709 13:02:14.591186 (+140365us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.917057  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 369) took 3745 ms. Trace:
I20260709 13:02:14.869262  4426 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:14.804327  4458 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:14.762354  2309 rpcz_store.cc:276] 0709 13:02:12.619436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.159795 (+1540359us) service_pool.cc:224] Handling call
0709 13:02:14.590787 (+430992us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.917613  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:14.918349  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2067) took 2603 ms (client timeout 1650 ms). Trace:
I20260709 13:02:14.921049  1735 rpcz_store.cc:276] 0709 13:02:11.166555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.911095 (+3744540us) service_pool.cc:224] Handling call
0709 13:02:14.911382 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.911562 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:14.911586 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.911647 (+    61us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:14.911687 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.911760 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:14.911880 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.923568  1584 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:14.793164  4325 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:14.804761  4486 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:14.925087  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:14.925792  1588 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 13:02:14.930181  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.897583  4483 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:14.936919  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.962306  1586 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:14.963634  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.964675  1586 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:14.965736  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 417192us
I20260709 13:02:14.966434  1592 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 13:02:14.915215  4460 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:02:14.974790  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.976150  1586 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:14.976311  1660 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:14.977087  1663 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:14.961730  1664 rpcz_store.cc:269] 0709 13:02:12.315017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.383640 (+ 68623us) service_pool.cc:224] Handling call
0709 13:02:14.918154 (+2534514us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54002f3854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:14.918277 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1292319104}
I20260709 13:02:14.977665  1663 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:14.977857  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.919023  4304 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.913033  4428 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.914023  4296 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:14.986294  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.982437  1735 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:14.987949  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:14.988590  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 373) took 3809 ms. Trace:
I20260709 13:02:14.988596  1588 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:14.988993  1735 rpcz_store.cc:276] 0709 13:02:11.179697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.981094 (+3801397us) service_pool.cc:224] Handling call
0709 13:02:14.981548 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.988323 (+  6775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.988527 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:14.993351  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:14.906102  4037 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.867911  4226 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.918893  4281 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:15.021063  4325 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.895382  4488 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.983392  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:14.905030  4249 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.893472  4409 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.976859  4458 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:15.043357  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 429897us
I20260709 13:02:14.924037  4319 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.996299  4464 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.059259  1740 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:15.026500  4281 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.073710  4488 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.074600  4488 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:15.077616  4319 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.078048  4454 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.078328  4308 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:14.992425  4428 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:14.991392  4296 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.054150  4458 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:14.992323  4304 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.080571  4409 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:15.069702  4495 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:02:15.114533  4495 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:15.117786  4496 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:15.118450  4496 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:15.118923  4496 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.150097  4428 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.185874  4483 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:15.202566  4496 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:14.987149  1586 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:15.294353  4037 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.295892  4364 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:15.241842  1590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.015s	sys 0.026s
I20260709 13:02:15.302614  4304 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.242350  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:15.248420  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 367868us
I20260709 13:02:15.197171  4486 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.354895  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:15.356206  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 372854us
I20260709 13:02:15.363003  1667 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:15.260892  4426 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.249125  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.283841  4296 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.289855  4389 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.291694  4226 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.078246  4319 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 13:02:15.301453  4249 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:15.301288  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 357606us
I20260709 13:02:15.296537  4364 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:15.296352  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:15.302958  1590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.016s	sys 0.026s
I20260709 13:02:14.984385  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.377558  1668 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:15.379499  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.379226  4364 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.380314  4406 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.324236  4496 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.328560  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:15.335204  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2097) took 5465 ms. Trace:
I20260709 13:02:15.335356  4119 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:02:15.305003  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.343335  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.341624  1659 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: []
I20260709 13:02:15.342988  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2090) took 5733 ms. Trace:
I20260709 13:02:15.338160  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.377724  4130 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:15.378049  1592 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:15.196020  4458 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.381479  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.413260  1741 rpcz_store.cc:276] 0709 13:02:09.609387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.944882 (+1335495us) service_pool.cc:224] Handling call
0709 13:02:11.953094 (+1008212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:11.953337 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:12.022548 (+ 69211us) log.cc:844] Serialized 14 byte log entry
0709 13:02:14.706436 (+2683888us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:14.706513 (+    77us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:14.706580 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.706682 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.342602 (+635920us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54003534d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:15.342788 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75328,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":79,"raft.queue_time_us":210581,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":48627,"spinlock_wait_cycles":2187300224,"thread_start_us":2683428,"threads_started":2,"wal-append.queue_time_us":2060493,"wal-append.run_cpu_time_us":1174,"wal-append.run_wall_time_us":1001887}
W20260709 13:02:15.421945  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 374118us
I20260709 13:02:15.480786  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:15.468465  1593 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:15.431394  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.382045  4383 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260709 13:02:15.488175  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:15.489365  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112960us
I20260709 13:02:15.491356  1705 leader_election.cc:400] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:15.491817  1705 leader_election.cc:403] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 13:02:15.492834  4119 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:15.493949  4119 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:15.494845  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.496058  1589 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:15.496480  4458 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.480929  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 650) took 2543 ms. Trace:
W20260709 13:02:15.502553  1584 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.524s	user 0.013s	sys 0.012s
I20260709 13:02:15.503015  1593 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:15.503192  1584 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:15.504791  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 600378us
I20260709 13:02:15.506126  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:15.506223  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:15.433784  1734 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:15.506825  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.508128  4226 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:15.509042  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 371) took 4342 ms. Trace:
I20260709 13:02:15.434944  4426 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.509945  4456 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:15.510186  4426 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:15.510957  4456 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:15.511736  4226 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.512430  4456 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.514508  1590 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 13:02:15.501763  2318 rpcz_store.cc:276] 0709 13:02:12.937041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.937208 (+   167us) service_pool.cc:224] Handling call
0709 13:02:12.937710 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.004633 (+ 66923us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:13.004664 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:13.005203 (+   539us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.102251 (+1097048us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:14.102298 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.480000 (+1377702us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b54006a0254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:15.480100 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.480205 (+   105us) raft_consensus.cc:1685] finished
0709 13:02:15.480229 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.480633 (+   404us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1158,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":152952064,"thread_start_us":1096385,"threads_started":1,"wal-append.queue_time_us":1243781}
I20260709 13:02:15.518200  4272 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:15.518821  4272 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:15.519388  4272 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.518244  1734 rpcz_store.cc:276] 0709 13:02:11.167031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.914119 (+3747088us) service_pool.cc:224] Handling call
0709 13:02:14.947671 (+ 33552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.507082 (+559411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.508701 (+  1619us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b540012d9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:15.508944 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1116527488}
I20260709 13:02:15.521242  1734 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:15.521938  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 374) took 2585 ms. Trace:
I20260709 13:02:15.523227  1734 rpcz_store.cc:276] 0709 13:02:12.936990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.520320 (+2583330us) service_pool.cc:224] Handling call
0709 13:02:15.520844 (+   524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.521658 (+   814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.521856 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.435997  4381 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.550s	user 0.003s	sys 0.019s
W20260709 13:02:15.444574  4478 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.100s	user 0.008s	sys 0.000s
I20260709 13:02:15.445426  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.457294  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 372) took 4289 ms. Trace:
I20260709 13:02:15.497267  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.381768  4386 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:15.464481  4491 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:15.465116  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:15.465801  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49167.9us
I20260709 13:02:15.471295  4454 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:15.472514  4516 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.382500  1742 rpcz_store.cc:276] 0709 13:02:09.869469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:12.917153 (+3047684us) service_pool.cc:224] Handling call
0709 13:02:13.818927 (+901774us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:13.819652 (+   725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.334568 (+1514916us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54000e1d50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:15.334855 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069878528}
I20260709 13:02:15.478085  4037 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:15.478236  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:15.478940  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2059) took 4828 ms (client timeout 1724 ms). Trace:
I20260709 13:02:15.479121  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.481139  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 367) took 1601 ms. Trace:
I20260709 13:02:15.481006  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:15.423362  4394 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.759s	user 0.018s	sys 0.001s
I20260709 13:02:15.482028  4496 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:15.483631  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2098) took 5613 ms. Trace:
I20260709 13:02:15.423054  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.448907  4325 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:15.485653  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 365) took 4319 ms. Trace:
I20260709 13:02:15.504969  1584 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260709 13:02:15.433159  1586 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:15.432796  4486 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:15.439909  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2073) took 2782 ms (client timeout 3267 ms). Trace:
W20260709 13:02:15.546015  1665 rpcz_store.cc:269] 0709 13:02:12.657815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.337207 (+2679392us) service_pool.cc:224] Handling call
0709 13:02:15.439847 (+102640us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.538292  1666 rpcz_store.cc:269] 0709 13:02:10.668738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:10.784796 (+116058us) service_pool.cc:224] Handling call
0709 13:02:15.476072 (+4691276us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:15.476337 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3519237504}
I20260709 13:02:15.529600  1741 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:15.509052  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.538656  1659 rpcz_store.cc:276] 0709 13:02:13.837125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.601791 (+764666us) service_pool.cc:224] Handling call
0709 13:02:15.325470 (+723679us) spinlock_profiling.cc:243] Waited 3.88 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:02:15.340959 (+ 15489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.413324 (+ 72365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.437873 (+ 24549us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8933120}
I20260709 13:02:15.544631  1584 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 3
I20260709 13:02:15.554360  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.554625  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:15.555121  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74821.8us
I20260709 13:02:15.555979  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.556180  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.556516  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:15.558836  4431 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:15.559439  4431 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:15.542032  1739 rpcz_store.cc:276] 0709 13:02:09.869928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:13.875819 (+4005891us) service_pool.cc:224] Handling call
0709 13:02:13.911449 (+ 35630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.644302 (+732853us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.482899 (+838597us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b54001081d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:15.483336 (+   437us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1473488640}
I20260709 13:02:15.561931  4266 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:15.560902  4431 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.553925  4516 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.563231  1592 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:15.491360  4161 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.564931  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.560791  1663 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.557264  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.568596  1661 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:15.566098  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2100) took 4296 ms. Trace:
W20260709 13:02:15.577971  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0539378730000522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.284s for a response
I20260709 13:02:15.579483  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:15.582409  4308 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:15.582180  4272 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.583139  4308 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:15.584089  4409 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.584884  4272 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.585815  4286 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.586131  1585 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:15.586498  4286 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.588057  4506 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.589021  4286 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.588749  4506 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.584460  4308 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.577339  1741 rpcz_store.cc:276] 0709 13:02:11.254977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.484361 (+4229384us) service_pool.cc:224] Handling call
0709 13:02:15.486106 (+  1745us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.550164 (+ 64058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.551632 (+  1468us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.530351  1736 rpcz_store.cc:276] 0709 13:02:11.167445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.924462 (+3757017us) service_pool.cc:224] Handling call
0709 13:02:14.932295 (+  7833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:14.932509 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:15.288317 (+355808us) log.cc:844] Serialized 14 byte log entry
0709 13:02:15.436676 (+148359us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.456437 (+ 19761us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:15.456599 (+   162us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.456740 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.457049 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36659,"prepare.queue_time_us":213335,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":50091,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":194,"thread_start_us":382861,"threads_started":2}
I20260709 13:02:15.598125  4296 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:15.598737  4296 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:15.599177  4296 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:15.601985  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:15.602756  1741 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:15.603422  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2106) took 1805 ms. Trace:
I20260709 13:02:15.603778  1741 rpcz_store.cc:276] 0709 13:02:13.798246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.601443 (+1803197us) service_pool.cc:224] Handling call
0709 13:02:15.602130 (+   687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.603083 (+   953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.603287 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:15.604125  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95248.3us
I20260709 13:02:15.605705  4486 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:15.598835  1588 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:15.607532  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 126732us
W20260709 13:02:15.608291  4281 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.526s	user 0.019s	sys 0.000s
I20260709 13:02:15.609278  4517 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:15.609787  4517 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:15.605088  2281 connection.cc:256] RPC call timeout handler was delayed by -1.4333511610000187s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
I20260709 13:02:15.552317  4491 consensus_queue.cc:1243] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:15.605203  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.584800  4409 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.543403  1737 rpcz_store.cc:276] 0709 13:02:11.165546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.906251 (+3740705us) service_pool.cc:224] Handling call
0709 13:02:14.906697 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:15.247475 (+340778us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:15.248399 (+   924us) log.cc:844] Serialized 14 byte log entry
0709 13:02:15.365929 (+117530us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:15.378947 (+ 13018us) log.cc:844] Serialized 26 byte log entry
0709 13:02:15.379169 (+   222us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:15.379204 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.379340 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.484108 (+104768us) raft_consensus.cc:1685] finished
0709 13:02:15.484134 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.484391 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10796,"prepare.queue_time_us":10961,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":4344,"raft.queue_time_us":199985,"raft.run_cpu_time_us":637,"raft.run_wall_time_us":39364,"thread_start_us":435185,"threads_started":2,"wal-append.queue_time_us":227669}
I20260709 13:02:15.610136  4517 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:15.552141  1584 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
W20260709 13:02:15.623411  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 117388us
I20260709 13:02:15.625715  1663 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:15.550149  4394 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.627586  1584 raft_consensus.cc:2917] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20260709 13:02:15.631489  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:15.638772  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82384.7us
W20260709 13:02:15.639215  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.642889  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 366) took 6787 ms. Trace:
I20260709 13:02:15.643380  1587 rpcz_store.cc:276] 0709 13:02:08.855536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:11.667186 (+2811650us) service_pool.cc:224] Handling call
0709 13:02:11.697697 (+ 30511us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:12.737789 (+1040092us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:12.737978 (+   189us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:12.753261 (+ 15283us) log.cc:844] Serialized 26 byte log entry
0709 13:02:14.103284 (+1350023us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:14.103341 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:14.103432 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.443560 (+1340128us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:15.642266 (+198706us) raft_consensus.cc:1685] finished
0709 13:02:15.642292 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.642619 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":423170,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":55894,"spinlock_wait_cycles":5913472,"thread_start_us":2338367,"threads_started":2,"wal-append.queue_time_us":1548950}
I20260709 13:02:15.647069  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.647292  4394 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.648344  4516 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.650656  4506 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:15.651367  4119 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:15.652403  1738 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:15.658018  4161 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
W20260709 13:02:15.660375  1740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1525) took 1492 ms (client timeout 1607 ms). Trace:
W20260709 13:02:15.660871  1740 rpcz_store.cc:269] 0709 13:02:14.167436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.545172 (+377736us) service_pool.cc:224] Handling call
0709 13:02:15.659915 (+1114743us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400284454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:15.660160 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":588805888}
I20260709 13:02:15.682391  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:15.683372  1740 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:15.684378  4416 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:15.685570  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.685731  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.686597  4506 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:15.686796  1591 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:15.687234  4506 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:15.687374  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.687948  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:15.688680  1591 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.688512  4506 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.690047  4428 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:15.691228  4513 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.002s
I20260709 13:02:15.692185  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:15.675292  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:15.693887  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.621482  4454 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:15.661849  4458 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.664124  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 366) took 4498 ms. Trace:
I20260709 13:02:15.639590  4486 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:15.665232  1584 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:15.609195  4281 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.658457  1734 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 13:02:15.658736  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.659155  4394 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:15.659665  4464 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:15.672036  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2099) took 4488 ms. Trace:
I20260709 13:02:15.672092  4406 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.673767  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.675283  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.676515  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:15.657495  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.677985  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.680069  4286 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:15.631793  4381 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:02:15.684733  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:15.662840  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:15.689098  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.732888  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:15.641078  4508 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 13:02:15.760437  1741 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:15.741763  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 375) took 2776 ms. Trace:
I20260709 13:02:15.764386  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:15.729369  1735 rpcz_store.cc:276] 0709 13:02:11.183702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.991701 (+3807999us) service_pool.cc:224] Handling call
0709 13:02:14.992040 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:15.429617 (+437577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.429683 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:15.430560 (+   877us) log.cc:844] Serialized 26 byte log entry
0709 13:02:15.533746 (+103186us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:15.533821 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.538165 (+  4344us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.671511 (+133346us) raft_consensus.cc:1685] finished
0709 13:02:15.671545 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.671772 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86655,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":77,"thread_start_us":422319,"threads_started":2,"wal-append.queue_time_us":194656}
I20260709 13:02:15.764137  1734 rpcz_store.cc:276] 0709 13:02:12.938192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.545547 (+2607355us) service_pool.cc:224] Handling call
0709 13:02:15.631996 (+ 86449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:15.714447 (+ 82451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.714794 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:15.718052  4454 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.731094  4394 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:15.773717  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 163710us
I20260709 13:02:15.703312  1733 rpcz_store.cc:276] 0709 13:02:11.165839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.909838 (+3743999us) service_pool.cc:224] Handling call
0709 13:02:14.931095 (+ 21257us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:15.418056 (+486961us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:15.418104 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:15.419260 (+  1156us) log.cc:844] Serialized 52 byte log entry
0709 13:02:15.550262 (+131002us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:15.550294 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:15.552051 (+  1757us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:15.663711 (+111660us) raft_consensus.cc:1685] finished
0709 13:02:15.663740 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:15.663913 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147353,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":616551,"threads_started":2,"wal-append.queue_time_us":191798}
W20260709 13:02:15.777804  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.745620  4464 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:15.787823  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.791177  4308 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:15.783154  4443 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 13:02:15.779502  4304 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:15.838340  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67691.5us
I20260709 13:02:15.847115  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:15.848340  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:15.850852  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.792902  1585 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:15.849772  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 343913us
I20260709 13:02:15.790511  4394 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.845247  4431 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 13:02:15.858551  4387 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:15.861625  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.837945  4308 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:15.875100  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 216775us
W20260709 13:02:15.875893  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 219833us
I20260709 13:02:15.784514  4119 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:15.862712  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 173751us
I20260709 13:02:15.788938  4464 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:15.847191  4304 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:15.877193  1706 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 3.337s (SENT) [suppressed 18 similar messages]
I20260709 13:02:15.877830  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:15.890946  1587 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:15.862231  4409 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:15.763224  4458 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.883957  1588 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:15.901831  1585 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260709 13:02:15.873077  4387 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:15.885275  4308 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:15.908456  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 121727us
I20260709 13:02:15.910991  1587 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:15.897827  4304 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:15.909723  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 59791.8us
I20260709 13:02:15.922807  4266 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:15.925143  4226 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:15.935767  1737 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:15.863206  4406 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:15.898053  4119 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:15.876760  4508 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 13:02:15.947803  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:15.911082  4387 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:15.947625  1592 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:15.956809  4184 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:02:15.941612  4119 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:15.939426  4454 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:15.965657  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:15.968793  4458 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:15.909229  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "ff44a4643e01482cb6b6a79b099fa766"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:15.977628  4364 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:15.981079  1589 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:15.981864  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 104181us
I20260709 13:02:15.982445  4496 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:16.001924  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50437.5us
I20260709 13:02:16.005888  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.010749  1591 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:16.009414  2318 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Received request to transfer leadership to TS ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:16.007290  4130 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.522s	user 0.009s	sys 0.005s
I20260709 13:02:15.999065  4184 consensus_queue.cc:1243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:16.016314  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:16.025740  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 124611us
I20260709 13:02:16.011996  4381 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:16.050865  1593 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:16.052012  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2210) took 1132 ms. Trace:
I20260709 13:02:16.052459  1593 rpcz_store.cc:276] 0709 13:02:14.919254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.503927 (+584673us) service_pool.cc:224] Handling call
0709 13:02:16.042679 (+538752us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8eee 0000000000375939 00007fcdf2f2c1e0 00007fcdf2ff50c4 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:02:16.042912 (+   233us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.051639 (+  8727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.051930 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49163008}
I20260709 13:02:16.035953  4428 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:16.009773  4496 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.070079  1590 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
W20260709 13:02:16.071828  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73981.5us
I20260709 13:02:16.078059  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:16.079378  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.010203  4266 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:16.083344  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67145.7us
I20260709 13:02:16.085531  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:16.086126  4525 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:02:16.086162  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:16.090194  4530 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 13:02:16.093400  4514 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.460s	user 0.003s	sys 0.000s
I20260709 13:02:16.098878  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.077149  4130 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.093833  4456 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:16.106832  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:16.111922  2309 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEARNER]: Advancing to term 5
I20260709 13:02:16.064024  4454 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:16.113602  4266 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:16.114210  1662 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:16.121937  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:16.122332  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:16.127434  4406 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.132778  1665 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:16.133888  4386 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:16.134645  4386 consensus_queue.cc:1243] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:16.138688  2312 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:16.141654  4508 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:16.144063  4526 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.353s	user 0.000s	sys 0.002s
I20260709 13:02:16.146407  4266 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:16.146253  4491 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:16.146854  4266 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:16.147372  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:16.147950  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:16.147899  4266 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.148767  2311 raft_consensus.cc:2376] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEARNER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7. Re-sending same reply.
I20260709 13:02:16.141758  4294 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:16.151150  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:16.151198  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.151489  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 377) took 3136 ms. Trace:
I20260709 13:02:16.158643  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.168346  4508 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:16.170059  4508 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:16.183429  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 178917us
I20260709 13:02:16.185253  4456 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:16.188576  4456 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:16.190621  4456 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:16.191238  4456 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:16.192725  4456 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.194480  4456 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:16.163290  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 146388us
W20260709 13:02:16.208494  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77635.8us
I20260709 13:02:16.198087  4294 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:16.213801  4325 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.214339  4325 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:16.214834  4325 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:16.219384  4517 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.003s
I20260709 13:02:16.219934  4517 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.221611  4458 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:16.222887  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:16.208029  4386 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:16.224663  4386 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:16.225770  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:16.226006  4406 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.226128  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1522) took 5662 ms. Trace:
I20260709 13:02:16.226449  4406 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.213642  4454 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:16.226855  1738 rpcz_store.cc:276] 0709 13:02:10.563800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.907698 (+4343898us) service_pool.cc:224] Handling call
0709 13:02:14.908011 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.652875 (+744864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.225798 (+572923us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b540039dfd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:16.226054 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1711110912}
I20260709 13:02:16.133438  4381 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.228644  4381 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.230958  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.226871  4406 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.205051  4130 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.238704  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.240226  4130 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.242095  1666 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:16.224136  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.149468  4266 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:16.146687  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:16.252692  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 101573us
I20260709 13:02:16.253372  4161 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:02:16.253643  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:16.253858  4161 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:16.255375  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:16.255162  4161 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.256579  4161 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.256490  4486 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:16.257054  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:16.258014  2317 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:16.258262  4486 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:16.258862  4486 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:16.259608  4515 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:16.260345  4515 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:16.259936  4486 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.260977  4515 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:16.159092  1739 rpcz_store.cc:276] 0709 13:02:13.014819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:15.562001 (+2547182us) service_pool.cc:224] Handling call
0709 13:02:15.562649 (+   648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.699366 (+136717us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:15.699857 (+   491us) log.cc:844] Serialized 14 byte log entry
0709 13:02:15.854480 (+154623us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.854556 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:15.854623 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.151250 (+296627us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:16.151301 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.151427 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":214,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":62,"raft.queue_time_us":8273,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":26306,"spinlock_wait_cycles":313574784,"thread_start_us":153920,"threads_started":1,"wal-append.queue_time_us":207028}
I20260709 13:02:16.262431  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 370) took 2424 ms. Trace:
I20260709 13:02:16.263345  4272 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:16.264062  4521 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:16.264542  4521 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:16.265581  4521 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.267026  4517 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.268198  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2082) took 1957 ms. Trace:
W20260709 13:02:16.269694  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51974.3us
I20260709 13:02:16.262835  1663 rpcz_store.cc:276] 0709 13:02:13.838186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.648270 (+810084us) service_pool.cc:224] Handling call
0709 13:02:14.648725 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.559806 (+911081us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b4c00164c80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:02:16.042413 (+482607us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.042442 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:16.042483 (+    41us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:16.042528 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.262117 (+219589us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:16.262170 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.262352 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23310,"raft.queue_time_us":142726,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":6197,"spinlock_wait_cycles":775384064,"thread_start_us":103406,"threads_started":1}
W20260709 13:02:16.278614  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 142665us
I20260709 13:02:16.268596  1667 rpcz_store.cc:276] 0709 13:02:14.311140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.659106 (+347966us) service_pool.cc:224] Handling call
0709 13:02:14.659410 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:15.362206 (+702796us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b58005ecad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:02:15.640772 (+278566us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:15.640851 (+    79us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:15.718401 (+ 77550us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:15.718470 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.267891 (+549421us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540013d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:16.267955 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:16.268133 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5775,"raft.queue_time_us":9592,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":2237,"spinlock_wait_cycles":1244496128}
I20260709 13:02:16.289260  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:16.294302  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49026.3us
W20260709 13:02:16.287748  4296 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.682s	user 0.013s	sys 0.003s
I20260709 13:02:16.303628  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.279253  4521 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.305603  2317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:16.307603  4487 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:16.298374  4517 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 13:02:16.309928  4535 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 13:02:16.310315  4119 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.213287  4304 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.311098  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.319257  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2066) took 4058 ms. Trace:
I20260709 13:02:16.321674  4409 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:16.298761  4184 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
W20260709 13:02:16.335408  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111431us
I20260709 13:02:16.322630  4536 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:16.324841  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:16.327358  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72075.7us
W20260709 13:02:16.330087  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.331161  1662 rpcz_store.cc:276] 0709 13:02:12.260388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:14.601377 (+2340989us) service_pool.cc:224] Handling call
0709 13:02:14.609889 (+  8512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.114597 (+1504708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.318875 (+204278us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:16.319184 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3283841792}
I20260709 13:02:16.330307  2317 raft_consensus.cc:606] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:16.269412  4394 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.339989  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.344152  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.295154  4130 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 13:02:16.332811  4409 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.324160  4296 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.355113  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.357729  1667 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:16.360852  4381 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:16.346830  4454 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:16.382802  1589 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 13:02:16.370123  2281 connection.cc:256] RPC call timeout handler was delayed by -1.0976884260000475s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
I20260709 13:02:16.354202  4409 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:16.395192  1586 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:16.382210  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:16.385977  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 113143us
I20260709 13:02:16.398142  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:16.387435  4454 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:16.407817  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67810.2us
I20260709 13:02:16.387799  4428 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:16.408736  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:16.414230  4454 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:16.435989  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88410.7us
I20260709 13:02:16.441994  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.442266  4543 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 13:02:16.443131  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.444521  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.453979  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.419677  4544 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:16.455953  1734 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:16.447533  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.459087  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:16.459493  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53757.3us
I20260709 13:02:16.479243  1665 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:16.486716  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.488090  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.489144  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.489602  1662 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 3: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:16.490880  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.491679  1662 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:16.492432  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.492935  4428 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:16.493809  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.486431  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.493110  1662 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:16.479754  1739 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:02:16.457569  4506 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:16.471385  4308 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260709 13:02:16.502007  1739 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 13:02:16.506139  1739 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:16.500746  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:16.486469  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.508754  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:16.509723  4458 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:16.508971  4552 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:02:16.511720  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51433.7us
W20260709 13:02:16.520678  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83590.1us
I20260709 13:02:16.528810  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:16.530296  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.531888  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:16.548720  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.549925  4428 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:16.550262  1737 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:16.518560  4495 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:16.501133  4544 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.556062  4495 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:16.557046  4553 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.557654  4553 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.557335  4495 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.558058  4553 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.558881  4495 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:16.559309  4226 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:16.563561  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.565100  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.525079  4486 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:16.557991  4544 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:16.511083  4406 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.570315  4458 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.570573  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:16.572726  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:16.574468  1593 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:16.575217  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:16.581648  4464 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.677s	user 0.013s	sys 0.000s
I20260709 13:02:16.518425  4308 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.583078  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.505923  4506 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:16.591980  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:16.598616  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67439.6us
W20260709 13:02:16.640024  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:16.640647  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.600081  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.642364  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:16.643879  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73386.9us
W20260709 13:02:16.596946  4387 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.603s	user 0.013s	sys 0.000s
I20260709 13:02:16.651696  1586 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:16.653379  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.655143  1588 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:16.585331  4304 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.657970  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:16.659057  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 257714us
I20260709 13:02:16.659627  4515 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:16.661036  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.661264  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.586541  4394 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:16.673597  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 151732us
I20260709 13:02:16.595384  4458 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:16.585834  4119 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:16.708374  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 63599us
I20260709 13:02:16.712117  1733 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:16.646304  4506 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:16.718540  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 76320.2us
I20260709 13:02:16.716058  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.645125  4296 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:16.707959  4515 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:16.724666  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132821us
I20260709 13:02:16.725557  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:16.726159  1664 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:16.726346  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:16.727231  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59893.6us
I20260709 13:02:16.727495  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:16.709010  4526 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 13:02:16.715287  4458 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.636716  4464 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:16.717617  4428 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.792217  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:16.729833  4518 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:02:16.726457  4515 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:16.792217  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54899.9us
I20260709 13:02:16.728519  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:16.672843  4387 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.832954  1593 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 9 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:02:16.803966  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:16.847432  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.889123  1733 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:16.839428  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:16.895634  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 186070us
I20260709 13:02:16.740641  4304 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:16.803664  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:16.903266  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55962.1us
I20260709 13:02:16.868765   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:02:16.905098  2312 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:16.904122  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:16.875314  4563 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:02:16.837986  4486 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:16.941761  4486 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:16.942231  4486 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.885532  4428 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:16.949489  4428 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:16.950412  4406 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.951175  4406 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:16.952406  4406 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:16.952819  4406 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:16.953224  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49170.2us
I20260709 13:02:16.800715  4394 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.155s
I20260709 13:02:16.954181  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:16.954917  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:16.955854  2311 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:16.956751  4119 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:16.957330  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:16.958631  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:16.925837  2310 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:16.961813  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:16.931136  4496 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:16.963394  1736 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:16.964743  4394 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:02:16.873982  4573 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:16.927675  2313 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:16.938740  1735 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:16.967413  4386 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:16.938158  4308 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:16.963332  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.969949  1591 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:16.938907  1660 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:02:16.964893  1738 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:16.970695  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.971539  1589 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:16.972858  1589 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:02:16.957523  4119 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:16.975224  4416 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:16.975834  4416 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:16.976900  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:16.945283  4409 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.532s	user 0.002s	sys 0.006s
I20260709 13:02:16.936342  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.991984  1586 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:16.970772  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:16.994902  1591 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:17.009035  1588 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:17.012687  1734 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.537s	user 0.012s	sys 0.000s
I20260709 13:02:17.014250  1734 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:17.014946  1664 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:16.991333  4386 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:17.024158  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.025043  1735 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:16.986732  4409 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.028378  2315 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:17.029423  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:17.030480  1668 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260709 13:02:17.030539  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105285us
W20260709 13:02:16.948706  4325 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.733s	user 0.009s	sys 0.002s
I20260709 13:02:17.030036  1589 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:17.035619  1591 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:17.044781  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.043267  1588 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:17.049683  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.051705  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.059921  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.059795  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.059978  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:17.030002  4409 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.066318  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.063570  1664 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 4
I20260709 13:02:17.079423  1664 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
W20260709 13:02:17.070127  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 116019us
I20260709 13:02:17.071635  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.059790  4495 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.074752  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:17.075043  4281 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.080152  1734 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:02:17.086745  1733 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:17.081436  1741 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.049471  4325 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.093348  1664 raft_consensus.cc:2917] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
I20260709 13:02:17.093956  1664 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:17.095799  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.097414  2315 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:17.097072  4495 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.098721  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.087080  1734 pending_rounds.cc:85] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 3
I20260709 13:02:17.101140  1734 pending_rounds.cc:107] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:02:17.100564  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.101408  1734 raft_consensus.cc:2889] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:16.984948  4406 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.102377  4406 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:17.115376  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59130.7us
I20260709 13:02:16.977389  4416 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.118206  4416 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:17.122787  4161 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.866s	user 0.016s	sys 0.000s
W20260709 13:02:17.123808  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.124248  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.124339  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:17.124727  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.131922  4495 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:17.134897  4454 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.662s	user 0.007s	sys 0.004s
I20260709 13:02:17.137444  2315 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEARNER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:17.145102  4294 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.147868  4506 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.149683  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.152134  4387 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:17.153764  4541 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.847s	user 0.004s	sys 0.000s
W20260709 13:02:17.157732  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64301.5us
I20260709 13:02:17.167520  4294 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:17.168090  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 378) took 1192 ms. Trace:
I20260709 13:02:17.165598  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:17.168334  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:17.169229  1739 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 4. Current term is 5. Ops: []
W20260709 13:02:17.170840  4529 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.152s	user 0.002s	sys 0.003s
I20260709 13:02:17.189177  1668 rpcz_store.cc:276] 0709 13:02:15.975724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.928351 (+952627us) service_pool.cc:224] Handling call
0709 13:02:17.030239 (+101888us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.030941 (+   702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.167867 (+136926us) spinlock_profiling.cc:243] Waited 86.7 ms on lock 0x7b540013ecd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:17.168025 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":199466880}
W20260709 13:02:17.171026  4553 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.612s	user 0.008s	sys 0.004s
W20260709 13:02:17.170974  4540 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.865s	user 0.002s	sys 0.000s
I20260709 13:02:17.171324  4325 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:17.173493  1557 connection.cc:256] RPC call timeout handler was delayed by -2.5008742399999733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
I20260709 13:02:17.179646  1587 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:17.202418  1557 connection.cc:256] RPC call timeout handler was delayed by -2.3217742399999679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
I20260709 13:02:17.187592  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.203214  4291 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:17.192698  4515 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.186574  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:17.203284  4409 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.200372  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:17.187955  4521 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.854s	user 0.006s	sys 0.006s
I20260709 13:02:17.209043  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:17.220820  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.171985  4543 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 13:02:17.229875  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.230357  1738 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:17.231369  1736 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.233093  4576 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:17.233716  4576 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:17.234097  4576 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:17.235356  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49014.4us
I20260709 13:02:17.237993  4582 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:17.173102  4454 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.241883  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.244239  1738 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:17.242484  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74255.4us
I20260709 13:02:17.246114  1742 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 9. Ops: []
I20260709 13:02:17.298408  2314 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:17.299881  1587 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:17.301069  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 92123.2us
I20260709 13:02:17.297087  4291 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:02:17.310518  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.300114  4458 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.311874  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.312475  1665 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.239061  4296 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:17.201829  4553 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.314217  1739 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.245831  4515 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:17.321801  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91738.9us
I20260709 13:02:17.308302  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.330387  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:17.325260  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:17.336576  1736 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.341653  4520 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 13:02:17.345937  4582 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:17.346487  4291 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:17.347532  4291 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:17.347904  4291 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:17.348806  4291 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:17.351979  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 151765us
I20260709 13:02:17.352948  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:17.353457  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.354091  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.303496  4521 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:17.369889  4486 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.371646  1735 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.374871  4486 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:17.314677  4394 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:02:17.328239  4458 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:17.379235  1737 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.243048  4409 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.386174  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:17.386395  1739 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:02:17.387710  1734 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:17.380663  1738 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:17.329249  4119 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:17.378077  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49921.1us
W20260709 13:02:17.390827  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85330.7us
I20260709 13:02:17.353686  4325 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:17.424599  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:17.430883  1736 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.341248  4542 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
W20260709 13:02:17.433845  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81012.7us
I20260709 13:02:17.380781  4557 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:17.434226  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:17.381165  4582 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:17.381376  4296 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:17.438687  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 664) took 1296 ms. Trace:
I20260709 13:02:17.439929  1739 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 13:02:17.441176  2312 rpcz_store.cc:276] 0709 13:02:16.137042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.139796 (+  2754us) service_pool.cc:224] Handling call
0709 13:02:16.140112 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:16.908946 (+768834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.432984 (+524038us) spinlock_profiling.cc:243] Waited 762 ms on lock 0x7b54006a0254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:17.433192 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1752878976}
I20260709 13:02:17.444087  1733 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:17.380580  4464 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.432513  4572 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:17.437870  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.435942  4543 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:17.448458  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:17.440817  4506 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.472654  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:17.448062  4515 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.435168  4272 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 13:02:17.449626  4521 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:17.455279  4543 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.455399  4291 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.437538  4428 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.485018  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.463857  4574 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:02:17.487953  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50177.3us
I20260709 13:02:17.452459  4325 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:17.486904  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.455704  4572 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:17.448326  4578 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:17.451989  4226 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:02:17.483031  4184 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:17.480401  4454 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:17.472275  4294 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.485024  4521 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:17.492218  4543 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.514228  1592 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:17.499965  4520 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:17.489126  4582 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:17.486353  4515 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.518230  4521 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:17.543396  2313 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:17.524597  4184 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.549335  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.512434  4542 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 13:02:17.544092  4272 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 13:02:17.554831  4272 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:17.555482  4272 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:17.560679  4574 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.561300  4574 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.564128  4574 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:17.566293  4585 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:02:17.568187  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.568919  1662 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:17.569892  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.574081  4582 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.578662  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2102) took 1087 ms. Trace:
I20260709 13:02:17.579195  1659 rpcz_store.cc:276] 0709 13:02:16.491461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.929739 (+438278us) service_pool.cc:224] Handling call
0709 13:02:16.930196 (+   457us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:17.037522 (+107326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:17.037561 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:17.061945 (+ 24384us) log.cc:844] Serialized 52 byte log entry
0709 13:02:17.327322 (+265377us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:17.327370 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.327447 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.578364 (+250917us) raft_consensus.cc:1685] finished
0709 13:02:17.578393 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.578576 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62160,"raft.queue_time_us":88102,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":18233,"thread_start_us":193340,"threads_started":1,"wal-append.queue_time_us":388048}
I20260709 13:02:17.516530  4294 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.526800  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.525637  4520 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:17.590793  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:17.621786  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.603691  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.591375  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.582492  4184 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.603235  2317 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:17.636073  1668 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:17.605883  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.637557  1668 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 13:02:17.636742  1739 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:17.649904  1739 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:17.603142  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:17.650437  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2108) took 1094 ms. Trace:
I20260709 13:02:17.651556  1739 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.653081  1739 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.654830  4520 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.657819  4226 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:17.639940  4506 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.585068  4576 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:17.658502  4226 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:02:17.658638  4506 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.651192  1737 rpcz_store.cc:276] 0709 13:02:16.556139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.576622 (+ 20483us) service_pool.cc:224] Handling call
0709 13:02:16.576953 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.431711 (+854758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.650162 (+218451us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b540014ebd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:17.650367 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1844579584}
I20260709 13:02:17.660550  4576 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:17.589039  4564 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.766s	user 0.000s	sys 0.003s
W20260709 13:02:17.661322  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58270.5us
I20260709 13:02:17.580514  4572 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.662726  4383 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:17.663163  4383 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:17.663524  4383 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.586844  4515 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:17.603056  4325 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.590170  4428 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.666430  4496 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:17.667729  4364 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:17.668139  4364 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:17.669049  4548 raft_consensus.cc:993] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:02:17.669795  4491 raft_consensus.cc:1081] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:02:17.672138  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.672356  4521 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:17.669090  4364 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.674768  4364 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:17.675186  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.675917  1740 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:17.676796  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.677593  1668 pending_rounds.cc:85] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 3
I20260709 13:02:17.677933  1668 pending_rounds.cc:107] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 13:02:17.678231  1668 raft_consensus.cc:2889] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:17.581135  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.657891  2313 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:17.639602  4304 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.639772  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.616613  4096 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.640192  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.611493  4522 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:17.591413  4409 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:17.706401  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.707448  1740 raft_consensus.cc:2368] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:17.641104  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 653) took 1576 ms. Trace:
W20260709 13:02:17.614564  4554 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.097s	user 0.003s	sys 0.000s
I20260709 13:02:17.634935  2314 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
W20260709 13:02:17.598011  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.641821  2316 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:02:17.642403  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.644276  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:17.644570  4537 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.286s	user 0.004s	sys 0.004s
I20260709 13:02:17.622372  4556 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:17.606802  4539 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.299s	user 0.001s	sys 0.008s
I20260709 13:02:17.542373  4553 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:17.581189  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.715546  1742 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.609347  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.622921  1665 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:17.660622  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.637753  4184 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.621781  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:17.701692  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.624768  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.692921  4506 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:17.708124  2310 rpcz_store.cc:276] 0709 13:02:16.064568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.066559 (+  1991us) service_pool.cc:224] Handling call
0709 13:02:17.640882 (+1574323us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400573ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:17.641023 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1883512192}
W20260709 13:02:17.700289  4406 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.597s	user 0.009s	sys 0.004s
I20260709 13:02:17.715080  1733 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:02:17.715356  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.716228  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.719846  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.707247  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:17.721457  2317 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:17.727420  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.722054  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.726902  2318 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:17.722831  1666 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [4.2-5.3]
I20260709 13:02:17.723392  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.724893  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.728974  1587 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:17.728919  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.704391  4522 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:17.726895  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:17.730162  4294 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:17.733403  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.733711  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.734643  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.737792  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.735399  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.713068  4416 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:17.737987  1659 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:17.742946  1740 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.736956  1738 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.747058  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.752921  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.755329  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.755997  1659 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:17.758078  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.758693  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.760200  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.726156  4428 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:17.763737  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.764405  1589 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:02:17.760887  1665 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:17.758955  1735 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.766803  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.768134  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.769255  1738 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:17.762976   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:17.770663  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.770697  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.771744  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.772467  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.772801  1737 raft_consensus.cc:2376] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6. Re-sending same reply.
W20260709 13:02:17.771692  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 295096us
I20260709 13:02:17.791868  1592 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:17.704099  4304 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:17.795706  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:17.800078  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:17.803061  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.804194  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.804867  1738 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.718859  4096 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.858366  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.863289  1587 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:17.867853  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:17.868738  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.770047  4406 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:17.721664  4556 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:17.771790  4119 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 13:02:17.871048  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.766067  4281 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:17.735146  4506 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 13:02:17.877293  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.877116  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.881894  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.910907  1589 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:17.871163  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.911657  1592 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:17.859112  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.912439  1738 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:17.913276  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.917587  4296 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:17.917675  4574 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:17.918331  4296 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.920603  4296 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:17.892926  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 377329us
I20260709 13:02:17.896648  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:17.766320  4416 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:17.921113  4281 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:17.921779  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.925868  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.805858  4291 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
I20260709 13:02:17.736399  4409 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.123s
I20260709 13:02:17.737592  4486 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.136s
I20260709 13:02:17.929469  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:17.930405  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 388) took 1803 ms. Trace:
I20260709 13:02:17.952759  1660 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:17.953545  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.935397  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.937773  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.938697  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.939131  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.943559  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:17.944023  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 148758us
I20260709 13:02:17.962497  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.964643  1739 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:17.890172  1665 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:17.946888  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:17.970249  1659 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:17.971109  1667 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260709 13:02:17.967882  4542 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:17.984799  1740 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 13:02:17.987614  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:17.987722  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:17.990069  1738 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:17.991930  1739 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:17.992303  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:17.988343  1589 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:17.942344  4593 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:02:17.861025  4464 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.110s
I20260709 13:02:17.948807  1590 rpcz_store.cc:276] 0709 13:02:16.126658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.126862 (+   204us) service_pool.cc:224] Handling call
0709 13:02:16.199642 (+ 72780us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:16.303178 (+103536us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:16.317613 (+ 14435us) log.cc:844] Serialized 14 byte log entry
0709 13:02:16.404010 (+ 86397us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:16.448509 (+ 44499us) log.cc:844] Serialized 52 byte log entry
0709 13:02:16.448666 (+   157us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:16.448704 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:16.448776 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:17.613870 (+1165094us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:17.930023 (+316153us) raft_consensus.cc:1685] finished
0709 13:02:17.930148 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:17.930313 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5739,"prepare.queue_time_us":1247,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":43,"raft.queue_time_us":26383,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":6787,"spinlock_wait_cycles":4642560,"thread_start_us":85872,"threads_started":1,"wal-append.queue_time_us":160572}
W20260709 13:02:17.997766  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54320us
I20260709 13:02:17.929864  4294 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:18.002557  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:18.009124  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.953841  4521 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.014793  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:17.974642  4553 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.008229  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:18.012735  1587 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:18.012804  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:17.974368  4291 consensus_queue.cc:1243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:17.989444  4495 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:18.026355  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.027097  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 3.
I20260709 13:02:18.036005  1738 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:18.038477  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:17.997071  4517 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:17.936764  4281 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.053229  1587 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:18.045415  2310 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:18.047137  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.056799  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.046712  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 183512us
I20260709 13:02:18.047760  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:18.010700  4491 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.064323  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.068941  1733 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:18.016623  4557 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:18.046622  4184 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.100950  1591 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:18.100948  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2122) took 1029 ms. Trace:
I20260709 13:02:18.102104  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2127) took 1028 ms. Trace:
W20260709 13:02:18.109884  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62328.4us
I20260709 13:02:18.102931  1591 rpcz_store.cc:276] 0709 13:02:17.073318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.085899 (+1012581us) service_pool.cc:224] Handling call
0709 13:02:18.100679 (+ 14780us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.101762 (+  1083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.102025 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.101799  1584 rpcz_store.cc:276] 0709 13:02:17.071712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.142591 (+ 70879us) service_pool.cc:224] Handling call
0709 13:02:17.142951 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.143216 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:17.546480 (+403264us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.095307 (+548827us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:18.095386 (+    79us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:18.100176 (+  4790us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.100474 (+   298us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.100762 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28780,"prepare.queue_time_us":328514,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":2040,"thread_start_us":758859,"threads_started":2}
I20260709 13:02:18.053613  4096 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:18.017802  4294 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:18.116911  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 137904us
I20260709 13:02:18.090308  4557 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.091529  4582 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.129948  1591 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:18.141841  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2128) took 1011 ms. Trace:
I20260709 13:02:18.142191  2318 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:18.142383  1591 rpcz_store.cc:276] 0709 13:02:17.130158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.112563 (+982405us) service_pool.cc:224] Handling call
0709 13:02:18.129763 (+ 17200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.141454 (+ 11691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.141760 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.061450  4517 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.148036  2317 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260709 13:02:18.154948  1584 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 13:02:18.147512  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.162535  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51487.6us
W20260709 13:02:18.163614  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.164397  1591 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:18.086158  4491 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.163275  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2129) took 1032 ms. Trace:
I20260709 13:02:18.166229  1584 rpcz_store.cc:276] 0709 13:02:17.130542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.114764 (+984222us) service_pool.cc:224] Handling call
0709 13:02:18.154611 (+ 39847us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.162950 (+  8339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.163216 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.167655  1584 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:18.168334  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2131) took 1037 ms. Trace:
I20260709 13:02:18.168629  1584 rpcz_store.cc:276] 0709 13:02:17.131124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.167015 (+1035891us) service_pool.cc:224] Handling call
0709 13:02:18.167399 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.168060 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.168276 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.170053  1584 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260709 13:02:18.171581  1584 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:18.140177  4557 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:18.173480  1584 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 13:02:18.227173  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 254730us
I20260709 13:02:18.176386  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2123) took 1104 ms. Trace:
I20260709 13:02:18.198915  1557 leader_election.cc:400] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:18.232141  1557 leader_election.cc:403] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:02:18.167482  4491 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.181149  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2130) took 1050 ms. Trace:
I20260709 13:02:18.236374  1591 rpcz_store.cc:276] 0709 13:02:17.130837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.143656 (+1012819us) service_pool.cc:224] Handling call
0709 13:02:18.163835 (+ 20179us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.165217 (+  1382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.180863 (+ 15646us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:18.217569  4383 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.002s
W20260709 13:02:18.184274  4596 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.308s	user 0.001s	sys 0.001s
I20260709 13:02:18.209641  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.251801  1736 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:18.210368  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.185813  4294 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.217828  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 594) took 1629 ms. Trace:
I20260709 13:02:18.220041  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:18.179580  4542 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:18.233901  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2232) took 1022 ms. Trace:
I20260709 13:02:18.266965  1584 rpcz_store.cc:276] 0709 13:02:17.211951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.172674 (+960723us) service_pool.cc:224] Handling call
0709 13:02:18.173053 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.226597 (+ 53544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.233813 (+  7216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.239120  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:18.268375  2315 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:18.189471  4568 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:18.188278  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.239558  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:18.253363  1593 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.000s
I20260709 13:02:18.216741  4601 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:18.209726  4304 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.146026  4582 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.229274  1585 rpcz_store.cc:276] 0709 13:02:17.072045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.180095 (+108050us) service_pool.cc:224] Handling call
0709 13:02:17.185698 (+  5603us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.206432 (+ 20734us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:17.703040 (+496608us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.081138 (+378098us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.097057 (+ 15919us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.098977 (+  1920us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:18.099025 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.099117 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.176013 (+ 76896us) raft_consensus.cc:1685] finished
0709 13:02:18.176101 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.176300 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":270274,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":1781,"raft.run_cpu_time_us":376,"raft.run_wall_time_us":30959,"thread_start_us":837431,"threads_started":2,"wal-append.queue_time_us":453001}
I20260709 13:02:18.198238  4496 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 13:02:18.260030  2316 rpcz_store.cc:276] 0709 13:02:16.598647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.599151 (+   504us) service_pool.cc:224] Handling call
0709 13:02:17.641567 (+1042416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.709089 (+ 67522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.217583 (+508494us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400573ad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:18.217758 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2256098048}
W20260709 13:02:18.249927  4543 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.725s	user 0.009s	sys 0.004s
I20260709 13:02:18.277016  1593 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:18.264910  1591 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 13:02:18.254974  4464 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
W20260709 13:02:18.280969  4575 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:02:18.266754  4542 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.294250  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2235) took 1081 ms. Trace:
I20260709 13:02:18.278978  4096 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:17.941563  4522 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.296708  1591 rpcz_store.cc:276] 0709 13:02:17.212931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.237658 (+1024727us) service_pool.cc:224] Handling call
0709 13:02:18.264571 (+ 26913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.293891 (+ 29320us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.294181 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:18.279237  4601 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.257439  4383 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.313436  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2126) took 1240 ms. Trace:
I20260709 13:02:18.295377  4184 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:18.316167  4607 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:02:18.322273  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129371us
I20260709 13:02:18.323374  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:18.293368  4521 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 13:02:18.290529  4487 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 13:02:18.325775  2318 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:18.317889  4601 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.340684  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.314298  1590 rpcz_store.cc:276] 0709 13:02:17.073018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.045468 (+972450us) service_pool.cc:224] Handling call
0709 13:02:18.085990 (+ 40522us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.105486 (+ 19496us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:18.121617 (+ 16131us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.308259 (+186642us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.312545 (+  4286us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:18.312667 (+   122us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:18.312785 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.312922 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.313254 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7811,"prepare.queue_time_us":18426,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":113,"raft.queue_time_us":21426,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":2236,"thread_start_us":173338,"threads_started":1}
I20260709 13:02:18.289947  4520 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 13:02:18.310345  4600 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 13:02:18.293916  4543 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.353492  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.356621  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.357543  4506 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:18.357795  4614 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:02:18.352902  4601 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:18.361730  4409 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.357828  4517 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:18.363201  4383 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:18.363224  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.366008  4383 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:18.366240  4604 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:02:18.367380  1584 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 13:02:18.368260  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.341447  4428 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.340416  4495 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:18.377138  1662 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:18.369467  4597 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 13:02:18.356154  4096 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:18.385327  1593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2111) took 1500 ms (client timeout 1745 ms). Trace:
I20260709 13:02:18.385895  4506 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:18.385795  1593 rpcz_store.cc:269] 0709 13:02:16.885481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.921432 (+ 35951us) service_pool.cc:224] Handling call
0709 13:02:18.385111 (+1463679us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b5400442d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:18.385236 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759540992}
I20260709 13:02:18.387192  4495 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.388363  4574 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:18.388772  4574 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:18.389711  4574 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.390973  4574 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:18.393977  4296 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.394469  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.395754  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.396520  4406 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.396895  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.397287  4406 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.398002  4574 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.366606  4383 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.387626  1593 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:18.401278  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:18.407387  4619 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:02:18.409430  4616 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
I20260709 13:02:18.411046  4616 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:18.411513  4616 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.413620  4617 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
W20260709 13:02:18.417667  4599 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
I20260709 13:02:18.420850  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1565) took 1242 ms. Trace:
W20260709 13:02:18.422250  4615 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
I20260709 13:02:18.423940  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:18.377035  4409 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.421545  1663 rpcz_store.cc:276] 0709 13:02:17.178019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.666039 (+488020us) service_pool.cc:224] Handling call
0709 13:02:17.666411 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.934417 (+268006us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:17.934462 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.935054 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.089819 (+154765us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:18.089876 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.420328 (+330452us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:18.420397 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.420466 (+    69us) raft_consensus.cc:1685] finished
0709 13:02:18.420486 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.420768 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":615607680,"thread_start_us":154296,"threads_started":1,"wal-append.queue_time_us":298927}
I20260709 13:02:18.425468  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2132) took 1232 ms. Trace:
I20260709 13:02:18.432408  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2142) took 1042 ms. Trace:
W20260709 13:02:18.432770  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136594us
I20260709 13:02:18.430285  1741 rpcz_store.cc:276] 0709 13:02:17.193577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.200795 (+  7218us) service_pool.cc:224] Handling call
0709 13:02:17.222132 (+ 21337us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:17.243792 (+ 21660us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:17.243824 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:17.297741 (+ 53917us) log.cc:844] Serialized 78 byte log entry
0709 13:02:17.541859 (+244118us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:17.714319 (+172460us) log.cc:844] Serialized 14 byte log entry
0709 13:02:17.717142 (+  2823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.717212 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.176930 (+459718us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:02:18.317387 (+140457us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b180020a520. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:18.425160 (+107773us) raft_consensus.cc:1685] finished
0709 13:02:18.425194 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.425388 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11915,"prepare.queue_time_us":192514,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":19883,"raft.run_cpu_time_us":6561,"raft.run_wall_time_us":519248,"spinlock_wait_cycles":18928384,"thread_start_us":193660,"threads_started":1,"wal-append.queue_time_us":419938}
I20260709 13:02:18.434491  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 388) took 1015 ms. Trace:
I20260709 13:02:18.432999  1734 rpcz_store.cc:276] 0709 13:02:17.390087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.422981 (+ 32894us) service_pool.cc:224] Handling call
0709 13:02:17.423369 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.437355 (+ 13986us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:17.437395 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.443237 (+  5842us) log.cc:844] Serialized 26 byte log entry
0709 13:02:17.693996 (+250759us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:17.694062 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:17.699461 (+  5399us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.416772 (+717311us) spinlock_profiling.cc:243] Waited 2.13 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:18.427006 (+ 10234us) raft_consensus.cc:1685] finished
0709 13:02:18.427038 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.432192 (+  5154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44152,"raft.queue_time_us":11815,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":11270,"spinlock_wait_cycles":4891008,"thread_start_us":221035,"threads_started":1,"wal-append.queue_time_us":540650}
I20260709 13:02:18.434984  1664 rpcz_store.cc:276] 0709 13:02:17.419135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.759775 (+340640us) service_pool.cc:224] Handling call
0709 13:02:17.760105 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:17.937490 (+177385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:17.937548 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:17.938053 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.114535 (+176482us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:18.114576 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.186327 (+ 71751us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:18.186416 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.424905 (+238489us) raft_consensus.cc:1685] finished
0709 13:02:18.424932 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.434401 (+  9469us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407418752,"thread_start_us":176108,"threads_started":1,"wal-append.queue_time_us":317065}
I20260709 13:02:18.399870  4096 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.442549  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:18.428881  4610 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
W20260709 13:02:18.450899  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 160041us
I20260709 13:02:18.451701  4281 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:18.382321  4602 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.159s
I20260709 13:02:18.433353  4487 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:18.412681  4616 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.470292  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2243) took 1089 ms. Trace:
I20260709 13:02:18.470939  1590 rpcz_store.cc:276] 0709 13:02:17.381203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.342565 (+961362us) service_pool.cc:224] Handling call
0709 13:02:18.355240 (+ 12675us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.373749 (+ 18509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.373784 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.382715 (+  8931us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.460392 (+ 77677us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:18.460441 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.460509 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.469972 (+  9463us) raft_consensus.cc:1685] finished
0709 13:02:18.470006 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.470205 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19811,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":86,"thread_start_us":44905,"threads_started":1,"wal-append.queue_time_us":45320}
W20260709 13:02:18.476102  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.484299  4621 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
W20260709 13:02:18.486011  4609 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 13:02:18.490024  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.490944  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2236) took 1277 ms. Trace:
I20260709 13:02:18.491088  4543 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:18.491361  1584 rpcz_store.cc:276] 0709 13:02:17.213173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.277040 (+1063867us) service_pool.cc:224] Handling call
0709 13:02:18.287202 (+ 10162us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.367779 (+ 80577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.490628 (+122849us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b540031f454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:18.490848 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183739136}
I20260709 13:02:18.492168  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.493844  4515 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:18.494927  4515 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:18.496865  4522 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.489151  1586 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:18.507623  1590 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 13:02:18.496466  4515 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.498193  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.460929  4487 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.519989  1584 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:46322
I20260709 13:02:18.520519  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:18.525978  4606 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 13:02:18.529991  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2240) took 1247 ms. Trace:
I20260709 13:02:18.532130  1584 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:02:18.533776  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 393) took 1087 ms. Trace:
I20260709 13:02:18.493031  4383 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.533149  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.535086  1740 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:18.534413  1584 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:18.534302  1587 rpcz_store.cc:276] 0709 13:02:17.282849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.293379 (+1010530us) service_pool.cc:224] Handling call
0709 13:02:18.293847 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:18.316684 (+ 22837us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.316737 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:18.317445 (+   708us) log.cc:844] Serialized 52 byte log entry
0709 13:02:18.440180 (+122735us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:18.440247 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.440351 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.529597 (+ 89246us) raft_consensus.cc:1685] finished
0709 13:02:18.529651 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.529891 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2237,"raft.queue_time_us":7802,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":269,"thread_start_us":121843,"threads_started":1,"wal-append.queue_time_us":208672}
I20260709 13:02:18.534428  1737 rpcz_store.cc:276] 0709 13:02:17.446023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.967865 (+521842us) service_pool.cc:224] Handling call
0709 13:02:17.997592 (+ 29727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.997847 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:18.350502 (+352655us) log.cc:844] Serialized 14 byte log entry
0709 13:02:18.533345 (+182843us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:18.533411 (+    66us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:18.533466 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.533574 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.533714 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":314,"prepare.queue_time_us":289438,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":1102,"thread_start_us":534320,"threads_started":2}
I20260709 13:02:18.500447  4582 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:18.541152  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.550438  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:18.541889  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.551669  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.552227  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:18.541949  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 72819.4us
I20260709 13:02:18.542915  1740 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:18.466049  4096 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.460960  4521 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 13:02:18.555830  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:18.565238  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:18.512217  4464 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.512516  4486 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.517395  4578 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:18.539494  4616 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.566810  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.576152  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:18.577590  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.568513  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:18.579166  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.498209  4184 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.068s
I20260709 13:02:18.569665  4582 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:18.571561  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:18.571834  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:18.572098  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.553843  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:18.587795  2310 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:18.574050  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:18.588912  2314 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:18.568292  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:18.583702  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.564929  4281 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.572429  4506 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:18.578436  4574 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:18.573112  4578 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.602576  1584 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.586550  4495 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:18.584872  4296 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.586630  4517 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:18.581533  4304 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.585302  4486 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.590363  4409 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:18.599325  4406 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:18.601107  4096 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.627377  1584 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:18.627790  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 56044.4us
I20260709 13:02:18.620748  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:18.631233  4641 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.614539  4578 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:18.605456  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 63749.4us
I20260709 13:02:18.621481  4495 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.649880  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.626284  4281 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:18.634045  4582 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:18.656054  1734 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:18.655859  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:18.656983  2317 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:18.650246  4406 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:18.634297  4506 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 13:02:18.653721  4184 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:18.679207  4304 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:18.658762  4641 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.680621  4406 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:18.660465  4495 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:18.712146  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:18.713238  1664 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:02:18.688866  4184 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:18.689038  4506 consensus_queue.cc:1243] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 13:02:18.722790  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:18.723691  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:18.726423  1665 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:18.690593  4582 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:18.727763  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:18.731364  1663 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:18.750684  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:18.764292  1742 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:18.765105  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1543) took 1257 ms. Trace:
I20260709 13:02:18.765547  1742 rpcz_store.cc:276] 0709 13:02:17.508165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.728749 (+1220584us) service_pool.cc:224] Handling call
0709 13:02:18.765013 (+ 36264us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:18.765681  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 102032us
W20260709 13:02:18.771526  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 161465us
W20260709 13:02:18.774721  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 191155us
I20260709 13:02:18.716527  4557 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.750277  4582 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:18.776034  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:18.770113  4631 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:02:18.781957  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.782141  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:18.770435  4584 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.194s	user 0.000s	sys 0.013s
I20260709 13:02:18.786266  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:18.787731  2318 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:18.788072  2318 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:18.778246  4626 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.175s	user 0.001s	sys 0.001s
W20260709 13:02:18.796454  4622 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.225s	user 0.000s	sys 0.002s
I20260709 13:02:18.777871  4304 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 13:02:18.809998  2313 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:18.788862  2318 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.827842  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:18.849988  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:18.782425  4649 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:18.855293  2318 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:18.860083  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.812107  4304 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:18.798532  4650 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:18.882956  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1559) took 1718 ms. Trace:
I20260709 13:02:18.810986  4641 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:18.859148  4574 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:18.934921  1660 rpcz_store.cc:276] 0709 13:02:17.164983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.634939 (+469956us) service_pool.cc:224] Handling call
0709 13:02:17.635276 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:17.953231 (+317955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.882647 (+929416us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b5400075dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:18.882880 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729239296}
W20260709 13:02:18.934742  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 353409us
W20260709 13:02:18.891163  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97151.8us
I20260709 13:02:18.848387  4296 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260709 13:02:18.902263  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 216890us
I20260709 13:02:18.887295  4486 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:18.902505  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2237) took 1689 ms. Trace:
I20260709 13:02:18.944420  1585 rpcz_store.cc:276] 0709 13:02:17.218570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.284053 (+1065483us) service_pool.cc:224] Handling call
0709 13:02:18.300631 (+ 16578us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.324573 (+ 23942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.324611 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.337826 (+ 13215us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.459480 (+121654us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:18.459541 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.459647 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.902187 (+442540us) raft_consensus.cc:1685] finished
0709 13:02:18.902219 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.902417 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9259,"raft.queue_time_us":11816,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":7068,"thread_start_us":100873,"threads_started":1,"wal-append.queue_time_us":214034}
W20260709 13:02:18.906577  4635 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.250s	user 0.002s	sys 0.000s
W20260709 13:02:18.906993  4632 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
W20260709 13:02:18.907317  4637 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 13:02:18.949579  4486 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:18.908842  4644 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 13:02:18.965255  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2247) took 1500 ms. Trace:
I20260709 13:02:18.900631  4556 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.965799  1593 rpcz_store.cc:276] 0709 13:02:17.464962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.403223 (+938261us) service_pool.cc:224] Handling call
0709 13:02:18.403713 (+   490us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:18.443443 (+ 39730us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.443492 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:18.444387 (+   895us) log.cc:844] Serialized 52 byte log entry
0709 13:02:18.562659 (+118272us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:18.562716 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.569532 (+  6816us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.964960 (+395428us) raft_consensus.cc:1685] finished
0709 13:02:18.964984 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.965181 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30866,"raft.queue_time_us":39214,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":7190,"thread_start_us":42437,"threads_started":1,"wal-append.queue_time_us":98735}
W20260709 13:02:18.914047  4638 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 13:02:18.914657  4647 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
W20260709 13:02:18.968461  4647 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:02:18.968989  4647 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:18.900852  4582 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:18.915441  4636 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
W20260709 13:02:18.902011  4601 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.543s	user 0.012s	sys 0.000s
I20260709 13:02:18.972219  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2120) took 1921 ms. Trace:
I20260709 13:02:18.919881  4495 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:18.863008  4649 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.923442  4578 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.925287  4406 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:18.907599  4643 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.114s
I20260709 13:02:18.914630  4557 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:18.933039  4655 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:18.933110  4281 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:02:18.939767  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:18.943192  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:18.935333  4304 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:18.937362  4574 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:18.951510  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2091) took 2513 ms. Trace:
I20260709 13:02:19.006564  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.017668  1666 rpcz_store.cc:276] 0709 13:02:16.438047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.607929 (+1169882us) service_pool.cc:224] Handling call
0709 13:02:17.717179 (+109250us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:17.728592 (+ 11413us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.951280 (+1222688us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b54002f3350. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:18.951442 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250196736}
I20260709 13:02:18.962677  4416 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:18.947575  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:18.963694  4428 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.590s	user 0.011s	sys 0.001s
I20260709 13:02:18.972610  1588 rpcz_store.cc:276] 0709 13:02:17.051142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.053157 (+  2015us) service_pool.cc:224] Handling call
0709 13:02:17.164057 (+110900us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:17.222832 (+ 58775us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:17.222863 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:18.378026 (+1155163us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b2c00032118. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:18.382442 (+  4416us) log.cc:844] Serialized 52 byte log entry
0709 13:02:18.586998 (+204556us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:18.587111 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.587303 (+   192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:18.971847 (+384544us) raft_consensus.cc:1685] finished
0709 13:02:18.971873 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:18.972135 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13349,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":11247,"spinlock_wait_cycles":51352320,"thread_start_us":153603,"threads_started":1,"wal-append.queue_time_us":231231}
I20260709 13:02:18.994652  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:18.995415  2318 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:18.994757  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:19.003783  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 153943us
I20260709 13:02:19.004026  1741 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 13:02:19.004382  4663 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:19.005064  4601 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:19.011065  4664 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 13:02:18.943747  4650 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.011605  4654 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 13:02:18.996078  4655 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.012676  4487 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.011052  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2242) took 1646 ms. Trace:
I20260709 13:02:19.064234  1591 rpcz_store.cc:276] 0709 13:02:17.364497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.300306 (+935809us) service_pool.cc:224] Handling call
0709 13:02:18.313710 (+ 13404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:18.367007 (+ 53297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.367053 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:18.380664 (+ 13611us) log.cc:844] Serialized 26 byte log entry
0709 13:02:18.510241 (+129577us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:18.510300 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.510399 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.010758 (+500359us) raft_consensus.cc:1685] finished
0709 13:02:19.010781 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.010971 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19515,"raft.queue_time_us":12981,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":8263,"thread_start_us":72711,"threads_started":1,"wal-append.queue_time_us":167974}
W20260709 13:02:19.036260  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.044564  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97099us
I20260709 13:02:19.069114  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:19.051015  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:19.043463  4556 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:19.062119  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55673.8us
I20260709 13:02:19.071134  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:19.043251  4574 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.056380  4663 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:19.052771  4428 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:19.076292  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 121896us
I20260709 13:02:19.050771  4557 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:19.051643  4286 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:19.080245  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.078639  4486 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:19.083779  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:19.090960  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:19.084903  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 146416us
I20260709 13:02:19.091071  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:19.091586  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.090975  1591 raft_consensus.cc:2376] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6. Re-sending same reply.
I20260709 13:02:19.092114  1588 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:19.094314  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.095559  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.097736  1588 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.063746  4655 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:19.078840  4517 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:19.077358  4582 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:19.083384  4487 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.082973  4557 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.061596  4601 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.103994  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.097971  4416 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:19.108947  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:19.122052  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.103477  4517 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:19.086365  4556 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.109956  4557 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.141911  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:19.104885  4582 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:19.106257  4670 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:19.139881  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.140365  1588 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:19.140502  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55297.8us
W20260709 13:02:19.146660  4661 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 13:02:19.147775  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:19.148003  4662 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:19.148590  4662 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.135972  4096 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:19.124809  4428 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:19.114619  4659 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:19.151525  1588 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:19.156443  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.139590  4517 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:19.160532  1589 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:19.162438  4616 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.574s	user 0.006s	sys 0.006s
I20260709 13:02:19.139384  4641 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:19.095543  4304 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.163615  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:19.178409  1736 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:19.179668  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:19.183787  1735 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:19.185936  1739 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:02:19.145051  4670 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:19.160063  4643 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.170095  4496 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 13:02:19.163915  4096 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:19.209396  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.184958  4406 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.229457  4542 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.231079  4542 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:19.231796  4655 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.240134  4674 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.240885  4674 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.244118  4674 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:19.196882  4495 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:19.187920  4665 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:19.196133  4578 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:19.200754  4464 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:19.213155  4662 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.226153  4616 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:19.355604  1739 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:19.361466  1742 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:19.257695  4623 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:19.366500  1736 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260709 13:02:19.181493  4486 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.333334  4291 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:19.375823  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.213809  4643 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.292233  4428 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:19.378984  4291 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:19.390791  4668 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.226s
I20260709 13:02:19.391992  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:19.422734  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:19.385368  4653 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.330s	user 0.002s	sys 0.000s
I20260709 13:02:19.469635  1588 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:19.353603  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:19.471426  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.471611  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:19.474164  1588 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.368403  4659 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:19.473719  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:19.479693  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 337869us
I20260709 13:02:19.483865  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:19.479936  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:19.460004  4666 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:19.491586  1739 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:19.293678  4464 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.473436  4383 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.338346  4665 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.515044  4650 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.483114  4495 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:19.486056  4640 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:02:19.545203  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 463576us
I20260709 13:02:19.519665  1739 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:19.525185  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.559564  1736 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:19.523705  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.522222  2309 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 13:02:19.540439  4605 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.199s	user 0.003s	sys 0.002s
I20260709 13:02:19.415544  4096 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.523396  4601 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.392204  4670 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.566543  4294 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:19.536581  4655 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:19.524536  4556 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.525101  4649 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:19.526141  4643 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.528450  4572 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.525588  4520 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:19.516043  4682 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.579357  1665 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 13:02:19.518350  4645 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:02:19.047250   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:19.593084  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2124) took 2520 ms. Trace:
I20260709 13:02:19.578547  4294 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:19.577913  4568 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:19.562326  4495 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.607086  4682 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.523792  4428 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.557354  4383 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:19.615954  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 387) took 2239 ms (client timeout 1646 ms). Trace:
W20260709 13:02:19.630209  1662 rpcz_store.cc:269] 0709 13:02:17.376496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:17.569608 (+193112us) service_pool.cc:224] Handling call
0709 13:02:19.615693 (+2046085us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:19.615852 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1855008000}
I20260709 13:02:19.617050  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 674) took 1073 ms. Trace:
I20260709 13:02:19.614140  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:02:19.624850  1588 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:19.633562  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.633944  1740 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [2.4-3.5]
I20260709 13:02:19.635118  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:19.635663  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.636107  1588 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:19.543406  4666 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.631623  2317 rpcz_store.cc:276] 0709 13:02:18.543656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.543810 (+   154us) service_pool.cc:224] Handling call
0709 13:02:19.616841 (+1073031us) spinlock_profiling.cc:243] Waited 67.4 ms on lock 0x7b5400573ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:19.616986 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154962688}
I20260709 13:02:19.598774  4520 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:19.641808  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.604669  1592 rpcz_store.cc:276] 0709 13:02:17.072366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.041220 (+968854us) service_pool.cc:224] Handling call
0709 13:02:18.061474 (+ 20254us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:18.062151 (+   677us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.062185 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:18.690445 (+628260us) spinlock_profiling.cc:243] Waited 1.06 ms on lock 0x7b2c00032118. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:18.715864 (+ 25419us) log.cc:844] Serialized 52 byte log entry
0709 13:02:18.856504 (+140640us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:18.856544 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.856620 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.587068 (+730448us) raft_consensus.cc:1685] finished
0709 13:02:19.587090 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.592982 (+  5892us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4211,"raft.queue_time_us":147,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":18239,"spinlock_wait_cycles":2438912,"thread_start_us":88433,"threads_started":1,"wal-append.queue_time_us":239179}
I20260709 13:02:19.521903  4659 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.657544  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.612282  4670 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.617193  4495 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.617653  4645 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:19.600944  4572 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:19.652029  4416 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.002s
W20260709 13:02:19.680334  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 219063us
I20260709 13:02:19.682466  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2090) took 3237 ms. Trace:
W20260709 13:02:19.665485  4678 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.301s	user 0.002s	sys 0.000s
W20260709 13:02:19.686230  4487 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.001s
I20260709 13:02:19.707674  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:19.680194  4291 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.721818  1586 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:19.718351  1589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2256) took 1299 ms (client timeout 1678 ms). Trace:
I20260709 13:02:19.721062  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 605) took 1504 ms. Trace:
I20260709 13:02:19.686679  1661 rpcz_store.cc:276] 0709 13:02:16.434605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:16.448894 (+ 14289us) service_pool.cc:224] Handling call
0709 13:02:16.475623 (+ 26729us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:16.547146 (+ 71523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:16.547178 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:17.642534 (+1095356us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:17.643189 (+   655us) log.cc:844] Serialized 52 byte log entry
0709 13:02:18.021141 (+377952us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:18.021195 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:18.021311 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:19.539069 (+1517758us) spinlock_profiling.cc:243] Waited 7.39 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:19.671450 (+132381us) raft_consensus.cc:1685] finished
0709 13:02:19.671490 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:19.671779 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45963,"raft.queue_time_us":47806,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":15601,"spinlock_wait_cycles":25395328,"thread_start_us":377525,"threads_started":1,"wal-append.queue_time_us":654510}
W20260709 13:02:19.725595  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.036s	sys 0.058s
I20260709 13:02:19.692175  4416 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:19.724318  2313 rpcz_store.cc:276] 0709 13:02:18.222535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.222972 (+   437us) service_pool.cc:224] Handling call
0709 13:02:18.223393 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:18.848784 (+625391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.720675 (+871891us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:19.720970 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1325532928}
I20260709 13:02:19.692734  4464 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.700218  4600 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:19.726071  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.036s	sys 0.059s
W20260709 13:02:19.723290  1589 rpcz_store.cc:269] 0709 13:02:18.419347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.423512 (+  4165us) service_pool.cc:224] Handling call
0709 13:02:19.718040 (+1294528us) spinlock_profiling.cc:243] Waited 674 ms on lock 0x7b54001c4154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:19.718258 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1550765952}
I20260709 13:02:19.723392  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:19.730239  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:19.741021  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 269478us
I20260709 13:02:19.725147  4364 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:19.753080  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.754038  1592 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:19.735410  4531 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:02:19.759475  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.766911  4184 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.768360  4184 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.721787  4294 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.770095  4294 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:19.773986  4364 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:19.774832  4364 consensus_queue.cc:1243] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:02:19.776677  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.785296  4672 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 13:02:19.788514  4286 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.698s	user 0.010s	sys 0.003s
I20260709 13:02:19.789083  4286 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:19.792383  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 326944us
I20260709 13:02:19.793923  4496 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.794351  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:19.794531  4496 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.795857  4668 consensus_queue.cc:1243] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:19.796547  4639 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:19.796742  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2149) took 1243 ms. Trace:
I20260709 13:02:19.797569  4667 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:19.797139  4639 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.797986  4667 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:19.797904  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 314162us
I20260709 13:02:19.799101  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:19.799511  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 240330us
I20260709 13:02:19.799878  4639 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:19.800536  4639 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:19.797168  1741 rpcz_store.cc:276] 0709 13:02:18.553175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.553339 (+   164us) service_pool.cc:224] Handling call
0709 13:02:18.570637 (+ 17298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:19.054420 (+483783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.796454 (+742034us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:19.796674 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982394624}
I20260709 13:02:19.801733  4680 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:19.797897  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 398) took 1810 ms. Trace:
I20260709 13:02:19.802985  1586 rpcz_store.cc:276] 0709 13:02:17.987434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.507868 (+520434us) service_pool.cc:224] Handling call
0709 13:02:19.721614 (+1213746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.725973 (+  4359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.797698 (+ 71725us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54001c4150. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:19.797833 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2760699648}
I20260709 13:02:19.804478  4296 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.805019  4296 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.806468  4687 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:19.807500  4296 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.810716  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.812883  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.822722  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.826627  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2145) took 1306 ms. Trace:
I20260709 13:02:19.826840  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.827057  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.827739  1586 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:19.826315  1489 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:19.827077  1734 rpcz_store.cc:276] 0709 13:02:18.519942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.520150 (+   208us) service_pool.cc:224] Handling call
0709 13:02:19.826408 (+1306258us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54000e1d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:19.826556 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284341632}
I20260709 13:02:19.828812  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.743456  4291 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:19.827610  1590 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:19.742233  4665 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.840410  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:19.797084  4496 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.797260  4668 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:19.753230  4682 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:19.846716  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:19.823561  4517 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.002s
I20260709 13:02:19.835886  4617 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.833082  4623 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:02:19.838038  4291 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:19.842792  4601 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.856799  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:19.856957  4568 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:19.824393  4531 consensus_queue.cc:1175] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df6a18af06d543b2931837f5e6bab95f" responder_term: 6 status { last_received { term: 5 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 13:02:19.862164  4689 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 13:02:19.867492  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112903us
I20260709 13:02:19.878738  1586 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 13:02:19.888940  2314 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:19.728246  4520 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.748939  4600 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:19.937410  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 92038.2us
I20260709 13:02:19.938982  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:19.975607  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.976532  1739 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:19.976826  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:19.946471  1587 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:19.954043  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:19.927487  4291 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:19.960628  1741 pending_rounds.cc:85] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 1
I20260709 13:02:19.892520  4655 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:19.966454  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:19.986927  1741 pending_rounds.cc:107] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 13:02:19.987442  1741 raft_consensus.cc:2889] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:19.930576  4688 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:19.928440  4647 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:19.950206  4568 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.013393  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:20.000348  1590 pending_rounds.cc:85] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 2
I20260709 13:02:20.016011  1590 pending_rounds.cc:107] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:20.016393  1590 raft_consensus.cc:2889] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:20.000914  4383 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.002260  4687 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:20.001638  1489 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 6, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:20.007340  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:19.998057  4666 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:19.893065  4616 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.080s
I20260709 13:02:19.799114  4667 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.011430  4464 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:20.021126  4688 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.965895  4676 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:19.925364  4617 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.967042  4639 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:19.887744  4645 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:19.973613  4572 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:19.948932  4601 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:19.903155  4517 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:20.025663  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.056285  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2137) took 1113 ms. Trace:
I20260709 13:02:20.154711  1591 rpcz_store.cc:276] 0709 13:02:18.942893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.139190 (+196297us) service_pool.cc:224] Handling call
0709 13:02:19.149956 (+ 10766us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:19.154858 (+  4902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.154894 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:19.162300 (+  7406us) log.cc:844] Serialized 52 byte log entry
0709 13:02:19.736852 (+574552us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:19.736892 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:19.736960 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.055980 (+319020us) raft_consensus.cc:1685] finished
0709 13:02:20.056008 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.056195 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10138,"raft.queue_time_us":15791,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":12908,"thread_start_us":574081,"threads_started":1,"wal-append.queue_time_us":739362}
I20260709 13:02:19.960013  4682 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:20.114539  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:20.116057  1584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.061s	sys 0.080s
W20260709 13:02:20.162887  1584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.062s	sys 0.080s
W20260709 13:02:20.146716  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.151118  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:20.076921  4689 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:02:20.074824  4665 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.162935  1667 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:02:20.166167  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.055735  4520 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:20.052431  4553 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:20.075491  4600 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.108325  4556 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:20.184950  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.176998  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:20.194186  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:20.170835  4641 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.988s	user 0.016s	sys 0.006s
I20260709 13:02:20.118979  4649 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:20.203603  2314 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:20.023267  4655 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.175668  4664 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.163211  4687 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:20.204191  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:20.180989  4553 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.145295  4685 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 13:02:20.150189  4639 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.263890  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:20.209343  4692 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 13:02:20.266387  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.206328  4506 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.207433  4623 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:20.214093  4325 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.183271  4464 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.230154  4383 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:20.022369  4647 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.235306  4553 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.284255  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:20.261716  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.014s	sys 0.030s
I20260709 13:02:20.210807  4649 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.212620  4641 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:20.192379  4665 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.276311  4542 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.044s	user 0.017s	sys 0.000s
I20260709 13:02:20.206662  4556 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.182538  4682 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:02:20.230583  4486 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.017s	sys 0.030s
W20260709 13:02:20.294912  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.014s	sys 0.031s
W20260709 13:02:20.279362  4557 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.125s	user 0.012s	sys 0.004s
I20260709 13:02:20.150758  4676 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.303001  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 569007us
I20260709 13:02:20.289455  4294 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:20.296205  4677 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.021s	sys 0.039s
I20260709 13:02:20.294405  4649 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:20.287050  4623 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:20.235885  4664 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.302387  4486 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.017s	sys 0.030s
W20260709 13:02:20.317530  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 495856us
I20260709 13:02:20.311444  4428 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.286942  4506 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.315212  4677 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.021s	sys 0.039s
I20260709 13:02:20.312680  4294 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:20.334565  4294 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.313836  4675 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.284818  4325 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.325309  4574 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.240s	user 0.016s	sys 0.001s
W20260709 13:02:20.347266  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.314108  4542 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260709 13:02:20.310987  4557 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.335441  4688 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:20.370432  4664 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.374676  4464 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:20.379165  4695 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:02:20.380916  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2133) took 1440 ms. Trace:
I20260709 13:02:20.381320  1584 rpcz_store.cc:276] 0709 13:02:18.940359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:18.940514 (+   155us) service_pool.cc:224] Handling call
0709 13:02:18.940902 (+   388us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:18.984193 (+ 43291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:18.984224 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:18.989786 (+  5562us) log.cc:844] Serialized 52 byte log entry
0709 13:02:19.531463 (+541677us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b18002646a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:20.163409 (+631946us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:20.163475 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.163575 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.380543 (+216968us) raft_consensus.cc:1685] finished
0709 13:02:20.380565 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.380843 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":107528,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":9848,"spinlock_wait_cycles":4062464,"thread_start_us":674432,"threads_started":2,"wal-append.queue_time_us":1249072}
I20260709 13:02:20.383098  4647 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.391065  4623 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.395083  4506 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.395481  4325 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.395833  4517 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.396507  4601 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.340734  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:20.330799  4428 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.415689  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.003s	sys 0.011s
W20260709 13:02:20.416261  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.003s	sys 0.012s
I20260709 13:02:20.312822  4687 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:20.417937  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.408744  4506 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:20.420802  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 435439us
I20260709 13:02:20.408613  4286 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 13:02:20.355901  4574 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.344571  4675 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.426766  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.424106  4506 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.442042  1592 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
W20260709 13:02:20.444206  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 481816us
I20260709 13:02:20.408836  4601 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 13:02:20.453994  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.465062  1668 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:20.467123  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.467404  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.468226  1663 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:20.468590  1664 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:20.418908  4641 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 13:02:20.475533  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.476372  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.482224  1666 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:20.477619  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.475306  4291 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.444937  4676 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.486479  4291 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.490506  4294 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.494092  4428 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.477018  4677 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.460604  4576 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:20.449471  4670 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.777s	user 0.015s	sys 0.003s
I20260709 13:02:20.493537  4568 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.491892  4506 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.498594  1662 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:20.472940  4665 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.418303  4582 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.501055  4096 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.013s	sys 0.032s
W20260709 13:02:20.504817  4640 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.000s	sys 0.020s
W20260709 13:02:20.543653  4699 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 13:02:20.500991  4557 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.502580  4639 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:20.518558  4667 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:20.547225  4096 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.013s	sys 0.032s
I20260709 13:02:20.598855  2313 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 13:02:20.542135  4640 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.000s	sys 0.021s
I20260709 13:02:20.600512  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.608273  1738 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
W20260709 13:02:20.603211  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.620292  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:20.509766  4677 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.511826  4576 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.554703  4682 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.616053  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:20.575966  4685 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.620337  1659 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:20.637355  2310 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:20.604267  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.627985  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:20.630550  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.637392  2313 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:20.596824  4670 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.638731  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:20.642923  1667 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:02:20.610754  4665 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.648268  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.651924  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.653398  1662 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:20.656299  1735 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:20.659406  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.621021  4464 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.618758  4325 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.673677  1737 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:20.634003  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:20.663842  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.664655  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.675621  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.666162  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:20.677459  1742 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:20.679386  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:20.682557  4708 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:02:20.682725  4675 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:20.682898  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.683615  1660 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:20.683441  4675 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:20.658411  4663 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:02:20.660696  4685 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.690407  4685 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:20.674016  4716 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.691561  4716 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.642961  4676 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
W20260709 13:02:20.693928  4705 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 13:02:20.694566  4677 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:20.696496  4702 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 13:02:20.633601  4506 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.619016  4688 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.711259  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.659137  4574 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.701138  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.715404  1662 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:20.666121  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 315509us
I20260709 13:02:20.702270  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.717077  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.718035  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.718923  1735 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:20.719020  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:20.648504  4682 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.707645  1660 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:20.723323  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:20.663201  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 217948us
I20260709 13:02:20.726272  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:20.735004  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.732796  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:20.743435  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.746197  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:20.714644  4673 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:20.715622  4557 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:20.657860  4670 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.762576  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:20.771493  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.670931  4665 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:20.772481  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:20.743131  4184 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.974s	user 0.021s	sys 0.000s
I20260709 13:02:20.677147  4464 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.710353  4542 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 13:02:20.729604  4578 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:20.744597  4574 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:20.701469  4325 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.779034  1734 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 3.
I20260709 13:02:20.748421  4668 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:20.767592  4557 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:20.803339  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:20.771615  4568 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:20.776549  4685 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:20.804014  1666 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:20.775125  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:20.769990  4673 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.719892  4506 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.723690  4688 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.811512  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.803321  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:20.836525  1584 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.737200  4682 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.823225  1668 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:20.774469  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.008s	sys 0.017s
W20260709 13:02:20.841188  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.008s	sys 0.017s
I20260709 13:02:20.803434  4294 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.807116  4718 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.817219  4542 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:20.809882  4578 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.838902  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2259) took 1024 ms. Trace:
I20260709 13:02:20.836449  1659 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:20.859658  1586 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:02:20.832935  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 57955.5us
I20260709 13:02:20.850405  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:20.861828  1668 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.834486  4601 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 13:02:20.851694  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 185872us
I20260709 13:02:20.837877  4673 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:20.816622  4557 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:20.864341  1738 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:20.867080  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:20.859387  1585 rpcz_store.cc:276] 0709 13:02:19.814062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.814222 (+   160us) service_pool.cc:224] Handling call
0709 13:02:19.836365 (+ 22143us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:19.843609 (+  7244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.843662 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:19.860455 (+ 16793us) log.cc:844] Serialized 52 byte log entry
0709 13:02:20.326345 (+465890us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:20.364136 (+ 37791us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.787851 (+423715us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.838326 (+ 50475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.838436 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:20.838496 (+    60us) raft_consensus.cc:1685] finished
0709 13:02:20.838515 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:20.838821 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99416,"prepare.queue_time_us":40742,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":3696,"raft.queue_time_us":163977,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":34405,"spinlock_wait_cycles":6782336,"thread_start_us":850491,"threads_started":2,"wal-append.queue_time_us":712547}
I20260709 13:02:20.869812  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.852389  4641 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:20.844707  4685 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:20.886515  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:20.819145  4668 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.876046  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:20.853040  4574 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:20.871120  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.858583  4487 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:20.841485  4722 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:20.852839  4296 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:20.859114  4718 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.862022  4716 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.893957  1585 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:20.933486  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:20.886623  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.956581  1739 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:20.957016  1668 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
W20260709 13:02:20.887342  4714 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.217s	user 0.000s	sys 0.001s
I20260709 13:02:20.924103  1584 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:20.870610  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:20.889313  4520 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.696s	user 0.012s	sys 0.000s
I20260709 13:02:20.877910  4542 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:20.888801  4464 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:20.986475  1663 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:20.966302  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:20.963883  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:20.969301  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:20.868279  4733 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:02:20.844578  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118428us
I20260709 13:02:20.931747  4601 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:20.979717  1741 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:20.929662  4487 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:20.992296  1668 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:20.993391  4496 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:20.993528  4685 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.995165  4601 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.994258  4496 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:20.957185  4722 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:20.963860  4719 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:20.980557  4542 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.047966  4542 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.962878  4294 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:21.052464  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.054939  2314 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:21.055074  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:21.033546  4702 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:02:21.009809  4722 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:20.925971  4574 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:20.925105  4641 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:20.930786  4296 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.032107  4670 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.009112  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.076345  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.077672  1668 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:20.883422  4720 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20260709 13:02:21.034312  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:20.992956  4487 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:21.005434  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:21.095314  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 96522.2us
I20260709 13:02:21.042853  1662 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:21.045408  4668 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.060515  4325 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.061098  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.063894  4718 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.061542  4304 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.060518  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.003252  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.082908  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.097483  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:21.103724  1668 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:21.079433  4506 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.109102  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.068259  4665 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.118542  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.111824  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.115279  1667 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:21.115478  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.106606  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:21.116734  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147771us
I20260709 13:02:21.104635  4542 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.100826  4716 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:21.144110  1667 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:21.146763  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.146863  1662 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:21.153210  4741 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:02:21.155323  1662 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 13:02:21.159065  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.164577  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.110177  4735 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 13:02:21.166016  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 419938us
I20260709 13:02:21.168146  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.175410  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:21.171834  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:21.164881  1632 connection.cc:256] RPC call timeout handler was delayed by -1.2695884930000148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
I20260709 13:02:21.142097  4682 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.146324  4616 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:21.179847  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.181248  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.112396  4734 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260709 13:02:21.113422  4383 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:21.112632  4720 consensus_queue.cc:1243] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:21.147715  4716 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.151083  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.177912  1739 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:21.150553  4688 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.181407  4616 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:21.211119  4616 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:21.212215  4734 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:21.212674  4734 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.181144  4733 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:21.213991  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.214228  4733 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.215291  4733 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.193151  4748 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:21.167325  4578 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.163147  4325 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:21.201256  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.115926  4496 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.079938  4719 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.208024  1634 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:21.116999  4673 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.223573  1662 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:21.203622  4716 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:21.224067  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 407) took 1481 ms. Trace:
I20260709 13:02:21.204777  4735 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:21.204741  4383 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:21.231721  1587 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260709 13:02:21.352885  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147559us
I20260709 13:02:21.354748  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:21.343341  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:21.269032  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.223497  2309 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 3.
I20260709 13:02:21.316908  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:21.290277  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0715884929999788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
I20260709 13:02:21.231273  1589 rpcz_store.cc:276] 0709 13:02:19.742605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.748132 (+  5527us) service_pool.cc:224] Handling call
0709 13:02:19.748614 (+   482us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:19.749074 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:19.749112 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:19.750162 (+  1050us) log.cc:844] Serialized 52 byte log entry
0709 13:02:20.316313 (+566151us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:20.341859 (+ 25546us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.363376 (+ 21517us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.139690 (+776314us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b180026ec00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:21.194740 (+ 55050us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.194863 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.223791 (+ 28928us) raft_consensus.cc:1685] finished
0709 13:02:21.223822 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.224004 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21027,"prepare.queue_time_us":90316,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":4182,"raft.queue_time_us":152040,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":93403,"spinlock_wait_cycles":16156288,"thread_start_us":513262,"threads_started":2,"wal-append.queue_time_us":859375}
W20260709 13:02:21.251382  4553 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.949s	user 0.013s	sys 0.003s
I20260709 13:02:21.339820  4751 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.391031  4553 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.391858  4719 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.391772  4751 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.393838  4553 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.394115  4751 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.394766  1736 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:21.314721  4568 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:21.409240  4568 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.230046  4734 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.410899  4557 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.002s
I20260709 13:02:21.412475  4557 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.253099  4649 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.926s	user 0.015s	sys 0.001s
I20260709 13:02:21.352785  4735 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.228655  4325 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.222651  4748 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.289793  4688 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.414512  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:21.436990  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:21.422122  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:21.339843  4640 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:21.336939  4617 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.287889  4574 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.412034  4383 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.441314  4688 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.464300  4688 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.413399  4568 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.432065  4649 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.445948  4735 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.446965  4557 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.447896  2313 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:21.471850  4649 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:21.431051  4667 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.824s	user 0.015s	sys 0.001s
I20260709 13:02:21.447324  4748 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.418907  4286 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:21.497895  1733 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 13:02:21.498412  1733 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 2
I20260709 13:02:21.498868  1733 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 13:02:21.499207  1733 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:21.451212  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.415050  4737 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.447764  4749 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:21.456686  4184 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:21.445073  4751 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:21.508622  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:21.506489  4294 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.519351  4364 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:21.527225  4364 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.527779  4364 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.505929  4184 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.532948  4184 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.503961  4737 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.542095  4294 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.544525  4725 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.496s	user 0.001s	sys 0.002s
I20260709 13:02:21.544762  4294 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.551039  4325 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.517299  2309 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:21.555466  4677 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:21.556212  4641 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.510210  4748 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.559053  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1575) took 1603 ms. Trace:
I20260709 13:02:21.559888  4677 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.570878  4677 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.560582  4748 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:21.572441  4746 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.530s	user 0.001s	sys 0.000s
W20260709 13:02:21.574839  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:21.575479  1705 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.739s (SENT) [suppressed 15 similar messages]
I20260709 13:02:21.576025  4556 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.576670  4677 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.577960  4737 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.565774  4487 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.596426  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.504531  4749 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:21.570662  1733 rpcz_store.cc:276] 0709 13:02:19.956139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.956282 (+   143us) service_pool.cc:224] Handling call
0709 13:02:19.960270 (+  3988us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.499841 (+1539571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.558742 (+ 58901us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54001095d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:21.558975 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3535446144}
I20260709 13:02:21.579895  4748 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.561936  4673 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:21.610102  4755 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:02:21.610631  4487 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.613842  1660 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:21.536947  4668 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:21.619805  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.602799  4749 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:21.622723  4184 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.625008  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49554.5us
I20260709 13:02:21.628075  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:21.628731  2314 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:21.624027  4668 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:21.643362  2314 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:21.631105  4649 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.644520  4649 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.645351  1489 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:21.648207  4702 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.542s	user 0.001s	sys 0.000s
I20260709 13:02:21.648492  4649 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:21.650496  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.653764  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.654701  1664 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:21.623858  4364 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.660347  4722 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.001s
I20260709 13:02:21.662443  4364 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.668957  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:21.670236  4487 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:21.670809  4487 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:21.643792  4304 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.534s	user 0.014s	sys 0.004s
I20260709 13:02:21.672952  1593 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 13:02:21.673357  1593 pending_rounds.cc:85] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:02:21.673628  1593 pending_rounds.cc:107] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:02:21.631034  2309 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:21.673858  1593 raft_consensus.cc:2889] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:21.674232  1593 pending_rounds.cc:107] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 13:02:21.674680  1593 raft_consensus.cc:2889] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:21.638069  4668 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.680022  4668 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:21.681023  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.681751  1663 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.683230  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1578) took 1054 ms. Trace:
I20260709 13:02:21.683714  1738 rpcz_store.cc:276] 0709 13:02:20.628954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.630151 (+  1197us) service_pool.cc:224] Handling call
0709 13:02:21.682983 (+1052832us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b5400435154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:21.683157 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":518022912}
W20260709 13:02:21.687291  4729 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.637s	user 0.005s	sys 0.000s
I20260709 13:02:21.689400  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2261) took 1874 ms. Trace:
I20260709 13:02:21.648147  1489 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 6 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:21.646230  4184 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.675580  4428 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.691021  4428 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.684757  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:21.692955  2309 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:21.675289  4726 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.633s	user 0.000s	sys 0.002s
I20260709 13:02:21.634524  4716 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.689903  1592 rpcz_store.cc:276] 0709 13:02:19.815103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.815236 (+   133us) service_pool.cc:224] Handling call
0709 13:02:19.834228 (+ 18992us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.442738 (+608510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.689071 (+1246333us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:21.689326 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397229440}
I20260709 13:02:21.678817  1659 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:21.677079  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:21.671769  4487 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.706712  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2262) took 1873 ms. Trace:
I20260709 13:02:21.707218  1590 rpcz_store.cc:276] 0709 13:02:19.832879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.977004 (+144125us) service_pool.cc:224] Handling call
0709 13:02:19.999727 (+ 22723us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:20.017304 (+ 17577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.017354 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:20.113772 (+ 96418us) log.cc:844] Serialized 616 byte log entry
0709 13:02:20.619144 (+505372us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:20.619189 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.619264 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.674421 (+1055157us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:21.695704 (+ 21283us) raft_consensus.cc:1685] finished
0709 13:02:21.695733 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.706634 (+ 10901us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3750,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":4147,"spinlock_wait_cycles":59016448,"thread_start_us":412974,"threads_started":1,"wal-append.queue_time_us":820993}
I20260709 13:02:21.711695  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.712383  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.712647  1667 raft_consensus.cc:2368] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:21.713135  1662 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:21.713418  1659 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:21.713726  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.714736  4576 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:21.715323  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.712664  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:21.716341  4745 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.667s	user 0.001s	sys 0.001s
I20260709 13:02:21.717808  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 689) took 1009 ms. Trace:
I20260709 13:02:21.718117  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.718959  1664 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:21.718758  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.719518  1665 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:21.721680  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.718159  2310 rpcz_store.cc:276] 0709 13:02:20.708838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.708977 (+   139us) service_pool.cc:224] Handling call
0709 13:02:20.709376 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.714718 (+  5342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.714765 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:20.722262 (+  7497us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.872394 (+150132us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:20.872477 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.872654 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:21.715697 (+843043us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:21.717487 (+  1790us) raft_consensus.cc:1685] finished
0709 13:02:21.717531 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.717741 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21978,"raft.queue_time_us":7180,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2772,"spinlock_wait_cycles":37030784,"thread_start_us":108787,"threads_started":1,"wal-append.queue_time_us":252502}
I20260709 13:02:21.726449  4296 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.727370  4119 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:21.724669  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.727963  4649 raft_consensus.cc:1064] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:02:21.731429  4096 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:21.732756  4680 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:21.733570  4680 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:21.735076  4680 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.732206  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.730270  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.732296  4096 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.745695  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:21.746910  4711 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:21.747367  4711 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.748119  1660 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:21.759047  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.676506  4616 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.744378  4096 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.762835  4096 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:21.742939  4655 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:21.748596  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.740897  4184 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:21.777091  1663 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:21.778193  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1577) took 1156 ms. Trace:
I20260709 13:02:21.781385  1737 rpcz_store.cc:276] 0709 13:02:20.621821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.636977 (+ 15156us) service_pool.cc:224] Handling call
0709 13:02:21.773781 (+1136804us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:21.778097 (+  4316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25093504}
I20260709 13:02:21.780272  1664 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:21.784010  1664 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:21.780308  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.789098  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:21.748899  4711 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.802240  4711 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:21.660951  4722 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.785857  1664 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.807204  1664 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:21.783663  4654 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.778131  4184 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:21.783964  4576 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.783751  4748 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:21.791047  4680 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.825057  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:21.829787  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.832391  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:21.833316  4119 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:21.835803  4616 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:21.838677  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.843675  1664 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 13:02:21.844131  1664 pending_rounds.cc:85] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 6
W20260709 13:02:21.819720  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.844586  1664 pending_rounds.cc:107] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 5.7
W20260709 13:02:21.844782  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.844969  1664 raft_consensus.cc:2889] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:21.845247  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.802616  4623 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:02:21.800827  4687 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.005s	sys 0.007s
W20260709 13:02:21.883800  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:21.883481  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 406) took 2492 ms. Trace:
I20260709 13:02:21.886976  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1581) took 1251 ms. Trace:
I20260709 13:02:21.890316  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1587) took 1232 ms. Trace:
I20260709 13:02:21.892176  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 418) took 1304 ms. Trace:
I20260709 13:02:21.846930  4325 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.896525  1734 rpcz_store.cc:276] 0709 13:02:20.587908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.589362 (+  1454us) service_pool.cc:224] Handling call
0709 13:02:21.892016 (+1302654us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b5400284e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:21.892117 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":323793664}
I20260709 13:02:21.892763  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:21.893625  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:21.897897  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.896109  1666 rpcz_store.cc:276] 0709 13:02:20.660542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.672827 (+ 12285us) service_pool.cc:224] Handling call
0709 13:02:21.848278 (+1175451us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x7b540044e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:21.890254 (+ 41976us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211559552}
I20260709 13:02:21.826418  4184 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:21.900727  1736 rpcz_store.cc:276] 0709 13:02:20.638429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.639810 (+  1381us) service_pool.cc:224] Handling call
0709 13:02:21.847877 (+1208067us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:21.886876 (+ 38999us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648024320}
I20260709 13:02:21.903640  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.904513  1735 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:21.899869  1742 pending_rounds.cc:85] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:02:21.905536  1742 pending_rounds.cc:107] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 13:02:21.905586  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.905942  1742 raft_consensus.cc:2889] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:21.906589  1664 raft_consensus.cc:2410] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 13:02:21.886950  4687 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.929s	user 0.006s	sys 0.007s
I20260709 13:02:21.895815  1588 rpcz_store.cc:276] 0709 13:02:19.390793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.642164 (+251371us) service_pool.cc:224] Handling call
0709 13:02:19.642453 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:19.670841 (+ 28388us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:19.674505 (+  3664us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.145955 (+471450us) log.cc:844] Serialized 14 byte log entry
0709 13:02:20.146188 (+   233us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:20.146258 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:20.146324 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.699987 (+553663us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b18000e94a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:21.573114 (+873127us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b18000e94a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:02:21.882457 (+309343us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b540025d354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:21.883284 (+   827us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:21.883421 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44678,"prepare.queue_time_us":3994,"prepare.run_cpu_time_us":142,"prepare.run_wall_time_us":4534,"raft.queue_time_us":34094,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":14039,"spinlock_wait_cycles":139295232,"thread_start_us":441028,"threads_started":1,"wal-append.queue_time_us":638333,"wal-append.run_cpu_time_us":4046,"wal-append.run_wall_time_us":1116948}
I20260709 13:02:21.848807  4716 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:21.918440  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:21.918325  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.919410  1735 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:21.919797  1666 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:21.921209  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:21.839270  4654 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.926800  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.838002  4748 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:21.929131  1736 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:21.898339  4623 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:21.900555  4383 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 13:02:21.948167  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.951923  2310 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 13:02:21.910614  4677 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:21.962999  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2263) took 2129 ms. Trace:
I20260709 13:02:21.882727  4576 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:21.919896  4768 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.002s
I20260709 13:02:21.963953  1587 rpcz_store.cc:276] 0709 13:02:19.833284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:19.978737 (+145453us) service_pool.cc:224] Handling call
0709 13:02:20.087387 (+108650us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:21.352594 (+1265207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.962644 (+610050us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:21.962918 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2630983552}
I20260709 13:02:21.973822  1734 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:21.978359  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:21.979164  1666 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:21.948163  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 431272us
I20260709 13:02:21.982108  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:21.970290  1736 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:21.991187  1734 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:21.910480  4325 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.000221  1733 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:22.009855  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.005925  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.014930  1736 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:21.973371  4736 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 13:02:22.030323  1735 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:21.973002  4677 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.046023  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.052685  4716 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.058239  2313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 307) took 1182 ms (client timeout 1554 ms). Trace:
I20260709 13:02:22.059609  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.059242  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.060549  1668 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:22.060863  1659 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:22.061108  4654 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:22.061977  4770 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:02:22.062668  1665 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:02:22.061879  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:22.058740  2313 rpcz_store.cc:269] 0709 13:02:20.875624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.875784 (+   160us) service_pool.cc:224] Handling call
0709 13:02:22.058008 (+1182224us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54002e8454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:22.058167 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1193392512}
I20260709 13:02:22.067360  4677 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:22.068388  4749 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.070233  4748 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:22.045384  4722 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:22.073591  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.074339  1735 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:22.074477  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.075374  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
W20260709 13:02:22.076329  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.078250  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.053932  4600 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.082245  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.082381  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1584) took 1212 ms. Trace:
I20260709 13:02:22.082933  1741 rpcz_store.cc:276] 0709 13:02:20.869198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.869370 (+   172us) service_pool.cc:224] Handling call
0709 13:02:22.081325 (+1211955us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54003fa7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:22.081465 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233950464}
I20260709 13:02:22.084640  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:22.085122  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.088505  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:22.090124  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.092608  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.104125  1666 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:22.107154  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.107795  1741 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:22.111030  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.089457  4487 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:22.096385  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.094288  4531 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:22.103881  4616 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:22.122625  2309 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:22.114944  4487 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.126006  1737 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:02:22.127413  4325 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:22.042497  4623 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.128487  4576 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:22.113364  4766 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.227s	user 0.002s	sys 0.000s
I20260709 13:02:22.078330  4716 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.133366  4730 raft_consensus.cc:1027] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: attempt to promote peer e6adef9da7da44f6af0af5695cf9b704: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 13:02:22.109268  4737 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:22.090581  4640 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:22.140579  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44326.5us
I20260709 13:02:22.141449  1735 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:22.138412  4616 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.160552  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.165020  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.165251  4286 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:22.165870  4286 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:22.166466  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.167097  4286 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.167886  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.171547  2309 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:22.175365  4734 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:22.149899  4487 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:22.179167  1735 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:22.088762  4600 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.139365  4531 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.159647  4616 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:22.186241  4774 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.091s	user 0.001s	sys 0.001s
W20260709 13:02:22.192080  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.152737  4749 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:22.150403  4751 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:22.184648  4738 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.120s	user 0.006s	sys 0.000s
I20260709 13:02:22.203401  1663 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:22.162343  4640 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:22.243535  4600 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.243738  4184 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:22.247908  4649 raft_consensus.cc:1068] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Unable to promote non-voter e6adef9da7da44f6af0af5695cf9b704: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
  Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }.
  Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.249174  1739 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:22.249711  4685 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.250224  4685 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.254989  4735 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:22.255465  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.33044e+06us
I20260709 13:02:22.259366  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.260200  1737 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:22.261696  4718 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:22.262430  4718 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.265064  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.266479  1588 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:22.281296  1736 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:22.279795  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:22.282375  2313 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:22.281333  1733 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:22.212699  4531 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:22.285272  1735 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:22.287972  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.288866  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:22.290061  2310 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:22.216933  4751 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.291942  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.292791  1588 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:22.295136  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.295912  1592 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.297876  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 408) took 2198 ms. Trace:
I20260709 13:02:22.298309  1661 rpcz_store.cc:276] 0709 13:02:20.098998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.099225 (+   227us) service_pool.cc:224] Handling call
0709 13:02:20.152325 (+ 53100us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.316289 (+163964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.316339 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:21.571669 (+1255330us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:21.572138 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 13:02:21.606093 (+ 33955us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:21.606120 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:21.750867 (+144747us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54001a81d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:21.751185 (+   318us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.297621 (+546436us) raft_consensus.cc:1685] finished
0709 13:02:22.297642 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.297791 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2474,"raft.queue_time_us":84665,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":9789,"spinlock_wait_cycles":376041216,"thread_start_us":33542,"threads_started":1,"wal-append.queue_time_us":212039}
I20260709 13:02:22.192842  4542 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 13:02:22.304091  4600 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:22.312276  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.266695  4712 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:02:22.317761  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.318912  1589 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:22.320508  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.284471  4723 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.301393  4751 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:22.314482  4738 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:02:22.303821  2313 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: []
W20260709 13:02:22.330703  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.229954  4640 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:22.308255  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.312456  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:22.306193  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 163694us
I20260709 13:02:22.338828  1584 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.340309  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:22.341074  2310 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:22.323071  1592 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:22.341564  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.342795  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2266) took 2508 ms. Trace:
I20260709 13:02:22.343539  1593 rpcz_store.cc:276] 0709 13:02:19.834232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.006297 (+172065us) service_pool.cc:224] Handling call
0709 13:02:20.006743 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:21.675287 (+1668544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.342554 (+667267us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54002701d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:22.342737 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3831147392}
I20260709 13:02:22.311755  4600 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.298144  4682 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:22.333747  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.321672  4753 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:22.335695  4304 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.335860  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.345031  2313 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:22.344904  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.346446  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.321673  4712 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:22.372102  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:22.334198  1588 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:22.374645  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.362720  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:22.363574  2317 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 13:02:22.303748  1663 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:22.376442  1706 leader_election.cc:400] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:22.324048  4659 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:22.385426  1590 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.381304  1589 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 13:02:22.380621  1584 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:22.388401  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 87074.9us
I20260709 13:02:22.385752  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:22.390481  1593 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:22.386754  1706 leader_election.cc:403] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:02:22.336506  4296 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:22.342406  4383 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.397228  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.399277  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.349970  4600 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:22.428016  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69293.3us
W20260709 13:02:22.386446  4096 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.003s
I20260709 13:02:22.349179  4685 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:22.363446  4515 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:22.434482  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.442243  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:22.450103  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 60142us
I20260709 13:02:22.328985  4723 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.452174  1660 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
W20260709 13:02:22.453282  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.456086  1736 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:22.455950  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:22.455962  1593 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:22.459228  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.461759  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.463359  1593 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:22.464534  1589 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:22.466154  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:22.470381  1587 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:22.474586  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.474788  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:22.480935  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:22.362468  4682 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.395309  4304 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:22.449313  4711 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.639s	user 0.013s	sys 0.002s
I20260709 13:02:22.487478  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:22.482358  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.507007  2310 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:22.507220  1593 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:22.440127  4582 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:22.481949  1589 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:22.531196  1736 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:22.489764  4774 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:02:22.534724  1736 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 13:02:22.538898  4776 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 13:02:22.539767  4716 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.541420  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:22.455178  4184 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:22.488415  4666 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:22.541904  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.490128  4506 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.528196  4428 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:22.543540  1584 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:22.544663  1659 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:22.491474  4574 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.562188  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.542730  1665 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:22.507447  4464 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:22.547919  1593 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:22.549407  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:22.546540  1736 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:22.402478  4659 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.389317  4753 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.520237  4781 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:22.481834  4721 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 13:02:22.647908  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.454468  4296 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.451879  4383 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:22.568707  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.603268  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:22.668376  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.478240  4119 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 13:02:22.566936  4718 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.699281  1590 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:22.565150  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.570363  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:22.460433  4515 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.433728  4739 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.144s
I20260709 13:02:22.663226  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:22.560595  4742 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.511s	user 0.000s	sys 0.008s
I20260709 13:02:22.560657  4294 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:22.564807  1587 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:22.602788  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.773120  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2155) took 1016 ms. Trace:
I20260709 13:02:22.773854  1660 rpcz_store.cc:276] 0709 13:02:21.779409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:21.779733 (+   324us) service_pool.cc:224] Handling call
0709 13:02:22.772801 (+993068us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b54003c8f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:22.773013 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1522715008}
W20260709 13:02:22.673442  1556 connection.cc:256] RPC call timeout handler was delayed by -1.4363927450000347s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2885s for a response
W20260709 13:02:22.783845  1556 connection.cc:256] RPC call timeout handler was delayed by -1.3151629840000396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260709 13:02:22.676978  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.691776  1592 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:22.468176  4649 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 13:02:22.702601  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.655108  4184 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.521584  4712 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.715354  4542 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.717331  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.566633  4506 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.722002  1590 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:22.722433  2314 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:02:22.723356  4765 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.822s	user 0.004s	sys 0.000s
W20260709 13:02:22.724881  4773 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.670s	user 0.003s	sys 0.000s
I20260709 13:02:22.727077  4364 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:22.726317  1489 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:22.734077  4737 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:22.739868  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.841027  1664 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:22.745918  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.748018  1733 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:02:22.842451  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.757436  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:22.759155  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.759824  4296 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.649091  1736 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:02:22.542986  4673 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.126s
I20260709 13:02:22.795864  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.857153  1660 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:22.795742  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2270) took 2855 ms. Trace:
I20260709 13:02:22.799652  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.800369  1738 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
I20260709 13:02:22.659797  4667 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.108s
I20260709 13:02:22.711184  1589 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:22.804981  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.706240  4711 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:22.807709  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 144618us
I20260709 13:02:22.866323  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:22.646375  4574 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.868394  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.869441  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:22.871598  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.834476  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.646203  4464 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:22.756593  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.882376  1667 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:22.846262  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:22.884919  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.840519  4325 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:22.858160  1587 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.668283  4781 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.831455  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.874264  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.890466  1733 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:22.891553  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.894833  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.897104  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.850140  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:22.860297  4749 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:22.899712  1584 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:22.899822  1590 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:22.923440  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.924360  1590 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:22.901983  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:22.927332  1585 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:22.903918  1741 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.796s	user 0.012s	sys 0.008s
I20260709 13:02:22.928154  1741 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:22.899509  1593 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:22.857813  1591 rpcz_store.cc:276] 0709 13:02:19.940133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.931959 (+991826us) service_pool.cc:224] Handling call
0709 13:02:20.982426 (+ 50467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:20.982676 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:21.022128 (+ 39452us) log.cc:844] Serialized 14 byte log entry
0709 13:02:21.764509 (+742381us) spinlock_profiling.cc:243] Waited 5.86 ms on lock 0x7b180008a0a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:02:22.719468 (+954959us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b4800080358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:02:22.792072 (+ 72604us) log.cc:844] Serialized 14 byte log entry
0709 13:02:22.795315 (+  3243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:22.795381 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:22.795433 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.795522 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.795670 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":515960,"prepare.queue_time_us":400460,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":4868,"raft.queue_time_us":568382,"raft.run_cpu_time_us":2847,"raft.run_wall_time_us":287726,"spinlock_wait_cycles":39794176,"thread_start_us":474554,"threads_started":2,"wal-append.queue_time_us":935972}
I20260709 13:02:22.801391  4184 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:22.940549  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 391262us
I20260709 13:02:22.942662  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:22.937393  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.945848  1735 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:22.798266  4783 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.100s
I20260709 13:02:22.953603  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.955964  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.958313  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.959328  1587 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:22.841122  4716 consensus_queue.cc:1175] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df6a18af06d543b2931837f5e6bab95f" responder_term: 7 status { last_received { term: 7 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 7 index: 6 } }
I20260709 13:02:22.961766  1593 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 13:02:22.963311  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:22.845067  1489 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:22.999450  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:22.966519  4753 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.914635  4737 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:22.839862  4286 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.094s
I20260709 13:02:23.000366  1593 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:23.006312  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.007337  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:23.008077  1738 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:22.971072  4506 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:22.973008  4728 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:02:22.975683  4487 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.765s	user 0.010s	sys 0.010s
I20260709 13:02:22.976884  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:22.979607  4464 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:22.979947  4496 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 13:02:22.983292  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:22.814527  4785 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.120s
I20260709 13:02:22.985774  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 415) took 2406 ms. Trace:
I20260709 13:02:22.988049  1586 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: []
W20260709 13:02:23.023433  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46682.5us
I20260709 13:02:22.989884  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:22.971211  1589 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:22.992913  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.001199  1667 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:23.019085  1734 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:23.003437  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 476289us
W20260709 13:02:22.920742  4773 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:02:23.009305  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:23.012526  2310 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.505s	user 0.012s	sys 0.008s
I20260709 13:02:23.032573  2310 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:22.897805  4325 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:22.828804  4542 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:23.002699  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:23.041008  1662 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.006s	sys 0.017s
W20260709 13:02:23.041383  1662 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.006s	sys 0.017s
I20260709 13:02:23.041324  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.018628  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:23.019623  1591 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
I20260709 13:02:23.019335  1664 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:23.021466  1489 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:22.940517  4680 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
W20260709 13:02:23.029482  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46276.3us
I20260709 13:02:23.029806  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:23.017858  4718 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 13:02:23.035547  1587 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:23.020160  4600 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.563s	user 0.014s	sys 0.002s
I20260709 13:02:23.037124  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:23.086275  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49298.8us
I20260709 13:02:23.018920  4728 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:23.016718  4734 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.840s	user 0.023s	sys 0.028s
I20260709 13:02:23.038648  1665 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:23.036875  4723 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.001125  4739 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:22.889590  4711 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.108747  1489 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 6, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:23.056617  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:23.106460  1666 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:23.084470  4685 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:23.088148  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.022117  4487 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.034659  4294 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 13:02:23.094648  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.036966  4325 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:23.096645  1488 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 253) took 24213 ms (client timeout 15000 ms). Trace:
I20260709 13:02:23.100397  1587 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:23.040465  4576 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:22.996629  4749 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.082553  4673 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.082911  4506 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:23.059863  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.112844  1661 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:23.023483  1740 rpcz_store.cc:276] 0709 13:02:20.579675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:20.582260 (+  2585us) service_pool.cc:224] Handling call
0709 13:02:20.608731 (+ 26471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:20.623984 (+ 15253us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:20.624039 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:20.637898 (+ 13859us) log.cc:844] Serialized 26 byte log entry
0709 13:02:20.820660 (+182762us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:20.820725 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:20.820833 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:22.327632 (+1506799us) spinlock_profiling.cc:243] Waited 9.86 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:22.919421 (+591789us) raft_consensus.cc:1685] finished
0709 13:02:22.919471 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:22.985688 (+ 66217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5817,"raft.queue_time_us":6526,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":9102,"spinlock_wait_cycles":22679296,"thread_start_us":115930,"threads_started":1,"wal-append.queue_time_us":336737}
I20260709 13:02:23.089066  4304 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.093029  1738 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:23.087556  4782 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.036s
W20260709 13:02:23.117583  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61065.5us
I20260709 13:02:23.103083  4682 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:23.156526  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 306531us
I20260709 13:02:23.164016  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.111270  4096 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.116074  1489 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:23.171352  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 111614us
I20260709 13:02:23.164935  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.085654  4600 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.121281  4659 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:23.193233  1587 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.194600  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.142315  4496 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:23.142012  4286 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.202070  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2171) took 1352 ms. Trace:
I20260709 13:02:23.201486  1735 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:23.202993  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.204404  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.145325  4649 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:23.145273  4576 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.209671  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.212317  1593 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:23.240229  1584 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:23.242205  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.162577  4673 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:23.243050  1593 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:23.242367  1489 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 3 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:02:23.233825  1742 rpcz_store.cc:276] 0709 13:02:21.849519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.401148 (+551629us) service_pool.cc:224] Handling call
0709 13:02:22.450434 (+ 49286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.450664 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:22.488810 (+ 38146us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.201411 (+712601us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:23.201475 (+    64us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:23.201554 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.201814 (+   260us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.201993 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11454,"prepare.queue_time_us":69209,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":108030,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":51308,"thread_start_us":697623,"threads_started":2,"wal-append.queue_time_us":634004}
I20260709 13:02:23.238938  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:23.247288  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 423) took 1187 ms. Trace:
I20260709 13:02:23.247735  1662 rpcz_store.cc:276] 0709 13:02:22.059916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.060087 (+   171us) service_pool.cc:224] Handling call
0709 13:02:22.060422 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.063855 (+  3433us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:22.281572 (+217717us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.041951 (+760379us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.089356 (+ 47405us) log.cc:844] Serialized 26 byte log entry
0709 13:02:23.089869 (+   513us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:23.089931 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.090038 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.247037 (+156999us) raft_consensus.cc:1685] finished
0709 13:02:23.247065 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.247217 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2969,"prepare.queue_time_us":263968,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":49907,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":222,"thread_start_us":977143,"threads_started":3,"wal-append.queue_time_us":432144}
I20260709 13:02:23.249307  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:23.187456  4734 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:23.187219  4782 consensus_queue.cc:1243] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:23.115957  4739 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.186280  4464 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:02:23.233424  4582 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.701s	user 0.012s	sys 0.000s
I20260709 13:02:23.331156  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.275669  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.234246  4718 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:23.334587  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.375604  1593 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:23.334929  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59479.7us
I20260709 13:02:23.331022  1489 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:23.378844  1489 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 1 to 2, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:23.301340  1587 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:23.377300  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.244529  4515 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:23.385107  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 518959us
I20260709 13:02:23.245476  4576 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.394412  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:23.251919  4574 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.388757  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.252355  4673 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:23.383338  4786 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 13:02:23.406061  1489 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:23.271287  4659 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:23.270459  4383 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.278966  4781 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.333074  4649 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:23.122771  1488 rpcz_store.cc:269] 0709 13:01:58.841883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:01:58.907553 (+ 65670us) service_pool.cc:224] Handling call
0709 13:02:23.054680 (+24147127us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:14.188872 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:14.222430 (+ 33558us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:14.222500 (+    70us) write_op.cc:435] Acquired schema lock
0709 13:02:14.222527 (+    27us) tablet.cc:662] Decoding operations
0709 13:02:14.287603 (+ 65076us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:14.287701 (+    98us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:14.287730 (+    29us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:02:14.289184 (+  1454us) tablet.cc:701] Row locks acquired
0709 13:02:14.289233 (+    49us) write_op.cc:260] PREPARE: finished
0709 13:02:14.289433 (+   200us) write_op.cc:270] Start()
0709 13:02:14.289707 (+   274us) write_op.cc:276] Timestamp: P: 1783602134289361 usec, L: 0
0709 13:02:14.289736 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 13:02:14.290541 (+   805us) log.cc:844] Serialized 6357 byte log entry
0709 13:02:15.190429 (+899888us) op_driver.cc:464] REPLICATION: finished
0709 13:02:15.417884 (+227455us) write_op.cc:301] APPLY: starting
0709 13:02:15.418059 (+   175us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:15.418562 (+   503us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:15.418772 (+   210us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:15.421386 (+  2614us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:15.458912 (+ 37526us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 13:02:15.459004 (+    92us) write_op.cc:312] APPLY: finished
0709 13:02:15.459759 (+   755us) log.cc:844] Serialized 113 byte log entry
0709 13:02:15.461719 (+  1960us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:15.462247 (+   528us) write_op.cc:454] Released schema lock
0709 13:02:15.462335 (+    88us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:15.463030 (+   695us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:15.463068 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22704,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":226747,"apply.run_cpu_time_us":6891,"apply.run_wall_time_us":45208,"num_ops":16,"prepare.queue_time_us":2107210,"prepare.run_cpu_time_us":26991,"prepare.run_wall_time_us":426750,"raft.queue_time_us":240666,"raft.run_cpu_time_us":7480,"raft.run_wall_time_us":153802,"replication_time_us":899974,"thread_start_us":2368031,"threads_started":4,"wal-append.queue_time_us":463760,"wal-append.run_cpu_time_us":12792,"wal-append.run_wall_time_us":1714409}]]}
I20260709 13:02:23.423247  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:23.415462  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.371855  4734 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:23.383872  4676 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.210299  4096 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.445619   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:23.421494  4718 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:23.384878  4723 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:23.412748  4640 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.034s	user 0.013s	sys 0.006s
I20260709 13:02:23.279999  4304 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.141s
W20260709 13:02:23.455714  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:23.478719  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:23.255146  4286 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.416756  4783 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:23.329870  4682 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.111s
I20260709 13:02:23.277185  4667 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.150s
I20260709 13:02:23.465291  1489 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:23.416275  4798 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:23.468497  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45364.5us
I20260709 13:02:23.499557  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:23.496100  1489 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:23.445137  4531 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.137s	user 0.015s	sys 0.008s
I20260709 13:02:23.443496  4184 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:23.506322  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.858s	user 0.032s	sys 0.047s
I20260709 13:02:23.474674  4800 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:23.475535  4640 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:23.476281  4793 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 13:02:23.476732  4737 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.486426  4649 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:23.444258  4515 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:23.447642  4542 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.528035  4496 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:23.414310  4582 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 13:02:23.483318  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 420) took 1086 ms. Trace:
I20260709 13:02:23.491588  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:23.486268  4783 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:23.487052  4787 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 13:02:23.505218  4799 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260709 13:02:23.474962  4789 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
W20260709 13:02:23.507793  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.033s	sys 0.047s
I20260709 13:02:23.525946  4184 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:02:23.513113  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 425059us
W20260709 13:02:23.488586  4792 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.391s	user 0.000s	sys 0.002s
W20260709 13:02:23.477393  4794 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.355s	user 0.000s	sys 0.001s
W20260709 13:02:23.529234  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 282564us
I20260709 13:02:23.536254  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:23.539582  2315 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:23.541853  1592 pending_rounds.cc:85] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:02:23.583701  1592 pending_rounds.cc:107] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 13:02:23.584149  1592 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:23.489656  4711 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.467511  4676 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.550158  1588 rpcz_store.cc:276] 0709 13:02:22.397329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.397573 (+   244us) service_pool.cc:224] Handling call
0709 13:02:22.397931 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:22.457574 (+ 59643us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:22.721413 (+263839us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b18003d4620. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:02:23.142512 (+421099us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.482884 (+340372us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:23.482952 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:23.483014 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.483102 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.483251 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71102,"prepare.queue_time_us":303938,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":73213,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":29060,"spinlock_wait_cycles":13315840,"thread_start_us":718173,"threads_started":2}
I20260709 13:02:23.510164  4531 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.490124  4506 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:23.500895  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:23.597087  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 96362.2us
I20260709 13:02:23.505445  4798 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.579206  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.579067  1589 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:23.606997  1586 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:23.607816  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.530224  4805 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:23.502086  4810 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 13:02:23.608645  1589 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:23.608649  4805 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:23.580511  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.558010  4802 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:23.602957  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:23.570232  4487 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.607966  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.610574  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.623483  1589 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.612552  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:23.575055  4811 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.586642  1590 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:23.572309  4600 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.575614  4542 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.618968  2317 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:23.625154  1588 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:23.633132  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 217886us
I20260709 13:02:23.634758  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:23.638809  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:23.606814  4506 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:23.605201  4711 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.575217  4649 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 13:02:23.613986  4723 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:23.610680  4781 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:23.615339  4805 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.659133  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:23.618251  4802 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:23.666911  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2167) took 1827 ms. Trace:
I20260709 13:02:23.670336  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.407246  4730 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Committing config change with OpId 4.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:23.617702  4737 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:02:23.646134  4806 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.586000  4734 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:23.709559  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 140639us
I20260709 13:02:23.637494  4801 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:02:23.712890  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:23.615079  4783 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.645838  4749 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:23.669231  1737 rpcz_store.cc:276] 0709 13:02:21.839764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.287140 (+447376us) service_pool.cc:224] Handling call
0709 13:02:22.308896 (+ 21756us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:22.309340 (+   444us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:22.309376 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:22.351522 (+ 42146us) log.cc:844] Serialized 26 byte log entry
0709 13:02:22.818808 (+467286us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:22.818879 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:22.818993 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:23.666595 (+847602us) raft_consensus.cc:1685] finished
0709 13:02:23.666630 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:23.666821 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1548,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4292,"thread_start_us":367265,"threads_started":1,"wal-append.queue_time_us":750245}
I20260709 13:02:23.626823  4364 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:02:23.661901  4804 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:23.667734  4718 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:23.660575  4723 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:23.668597  2309 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 13:02:23.677934  4682 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:23.645949  4496 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.674175  4802 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:23.869285  4364 consensus_queue.cc:1243] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:02:23.675395  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 166394us
I20260709 13:02:23.645406  4811 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.814080  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
W20260709 13:02:23.725273  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 329521us
I20260709 13:02:23.720064  4487 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:23.868748  1708 leader_election.cc:336] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.529s (SENT)
I20260709 13:02:23.896840  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:23.664883  4711 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.686522  4667 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:23.875260  4723 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.645721  4542 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.711319  4659 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:23.665016  4816 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 13:02:23.719719  4600 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.724247  1588 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:23.678248  4739 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:23.892251  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 308926us
W20260709 13:02:23.871874  4809 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 13:02:23.676550  4781 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.723155  4286 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.872506  4749 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.867417  4718 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:23.938942  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50801us
I20260709 13:02:23.940160  4383 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:02:23.948113  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51368.9us
I20260709 13:02:23.726286  4506 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.876368  4682 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:23.884320  4783 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:23.961171  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 236559us
I20260709 13:02:23.714716  4806 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:23.930024  4640 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:23.878424  4804 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:23.960225  4735 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.704s	user 0.015s	sys 0.004s
I20260709 13:02:23.963410  4824 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:23.984591  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:23.990716  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:24.032683  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48247.2us
W20260709 13:02:24.034389  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 43775.3us
I20260709 13:02:23.968616  4600 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.999711  1586 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:23.971923  4659 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:24.005095  4286 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:23.989655  4531 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:24.006351  4817 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 13:02:24.006675  4781 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.006511  4716 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:02:24.007129  4820 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 13:02:23.988387  4749 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:23.982852  4718 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:23.992465  4682 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:24.009234  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2152) took 1201 ms. Trace:
I20260709 13:02:24.073832  1590 rpcz_store.cc:276] 0709 13:02:22.808121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.925189 (+117068us) service_pool.cc:224] Handling call
0709 13:02:23.483255 (+558066us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.625790 (+142535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.008041 (+382251us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540025c454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:24.008989 (+   948us) spinlock_profiling.cc:243] Waited 553 ms on lock 0x7b540025c450. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.009153 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1509275264}
I20260709 13:02:23.978762  4816 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:23.976815  4184 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:23.967283  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 386906us
W20260709 13:02:24.086021  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.086779  1705 leader_election.cc:400] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
W20260709 13:02:24.017086  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 184829us
W20260709 13:02:24.087060  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:24.087937  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.018978  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:24.016393  4616 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.795s	user 0.019s	sys 0.000s
I20260709 13:02:23.932133  4487 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.002135  4828 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:02:24.095193  2317 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:24.087286  1705 leader_election.cc:403] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260709 13:02:24.049978  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80824.7us
I20260709 13:02:24.025645  4798 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.058595  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:24.063443  1663 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:02:24.102367  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 43861.8us
I20260709 13:02:24.036109  4294 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:24.042214  4810 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.040479  4830 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:24.078814  4814 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.351s	user 0.002s	sys 0.000s
I20260709 13:02:24.045490  4640 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
W20260709 13:02:24.107178  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.111965  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.116351  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2162) took 1361 ms. Trace:
I20260709 13:02:24.051364  4737 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 13:02:24.095261  4496 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:24.152467  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:24.116621  4810 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.044358  4824 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.094686  4828 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:24.137290  4487 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.148249  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.148193  1661 rpcz_store.cc:276] 0709 13:02:22.755244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.757103 (+  1859us) service_pool.cc:224] Handling call
0709 13:02:24.116151 (+1359048us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.116287 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":620196608}
I20260709 13:02:24.053946  4659 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.065263  4735 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.052s
W20260709 13:02:24.117092  4815 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 13:02:24.131682  4830 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:24.156230  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.156437  4542 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.134845  4806 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:24.143519  4325 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.007s	user 0.013s	sys 0.000s
I20260709 13:02:24.172188  4736 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:24.147974  4831 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 13:02:24.116994  4798 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:24.189046  1590 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 13:02:24.195604  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47446.7us
I20260709 13:02:24.246291  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.198556  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1593) took 1314 ms. Trace:
I20260709 13:02:24.145743  4826 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:24.137303  4576 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.000s
I20260709 13:02:24.201047  4680 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:24.179612  4828 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.149950  4827 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:24.195113  4801 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.249814  1738 rpcz_store.cc:276] 0709 13:02:22.884592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.009049 (+124457us) service_pool.cc:224] Handling call
0709 13:02:24.198244 (+1189195us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.198468 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129236224}
I20260709 13:02:24.221324  4811 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.301131  4096 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.255463  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.195046  4711 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.106593  4616 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.262234  4325 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.262748  4576 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:24.271062  4842 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:02:24.277844  4184 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.278638  4804 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.304796  4810 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.182229  4812 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.105s
I20260709 13:02:24.280623  4623 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:24.281739  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49478 (request call id 76) took 1096 ms. Trace:
I20260709 13:02:24.348955  1664 rpcz_store.cc:276] 0709 13:02:23.237932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.238157 (+   225us) service_pool.cc:224] Handling call
0709 13:02:24.273314 (+1035157us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b540030b954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe3735 00007fcdedfdb9fa 00007fcdf2fa34c9 00007fcdec51fc54 00007fcdec51fbe2 00007fcdec51fb11 00007fcdec51fa8c 00007fcdec51ed42 00007fcdec31848c 00007fcdec3178c6
0709 13:02:24.281618 (+  8304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":461863808}
W20260709 13:02:24.283420  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1594) took 1398 ms (client timeout 1770 ms). Trace:
I20260709 13:02:24.287624  4749 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.288385  4119 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:24.290726  4824 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.363261  4506 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:24.292964  4515 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:24.294425  1593 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:24.400714  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:24.351032  4751 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.981s	user 0.012s	sys 0.018s
W20260709 13:02:24.319495  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64126.1us
I20260709 13:02:24.405782  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:24.408169  4600 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.410482  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.411556  1587 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:24.412822  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.166594  4496 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.416121  4496 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.417025  4749 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:24.417460  4749 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.417824  4749 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:24.321875  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 216899us
I20260709 13:02:24.424042  4616 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:24.426041  4576 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:24.243770  4736 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:24.309618  4531 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:24.349946  1735 rpcz_store.cc:269] 0709 13:02:22.884858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.997193 (+112335us) service_pool.cc:224] Handling call
0709 13:02:24.283203 (+1286010us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003516d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.283354 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":333769472}
I20260709 13:02:24.355412  1733 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:24.354261  4119 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:24.388224  4325 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:24.327004  4680 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:24.291366  4800 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:24.340682  4673 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.921s	user 0.006s	sys 0.011s
I20260709 13:02:24.295157  4785 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:24.346880  4828 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:24.449944  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.407968  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:24.312209  4814 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:02:24.432546  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:24.432845  1475 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ff44a4643e01482cb6b6a79b099fa766 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:24.439816  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.455448  2313 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:24.442183  1587 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:24.314507  4801 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:24.413372  4805 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.741s	user 0.009s	sys 0.002s
I20260709 13:02:24.399019  4515 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:24.444399  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.340562  4682 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:02:24.451853  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43976.2us
I20260709 13:02:24.454394  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.457051  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.434887  4751 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.460170  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.567582  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.552104  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:24.617916  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.570091  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.614691  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:24.616261  1589 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:24.611018  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
W20260709 13:02:24.621830  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.620627  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.620152  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:24.622714  1593 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:24.623992  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.624616  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.626531  4800 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 13:02:24.621163  4364 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:24.627004  4800 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:24.627256  4802 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.694s	user 0.010s	sys 0.003s
I20260709 13:02:24.627560  4800 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:24.627712  4802 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.631419  4680 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:24.632050  4680 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.632468  4680 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:24.638237  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1608) took 1980 ms (client timeout 1842 ms). Trace:
W20260709 13:02:24.639550  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:24.645818  4849 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
W20260709 13:02:24.646535  4730 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.676s	user 0.012s	sys 0.000s
I20260709 13:02:24.596470  4685 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:24.650533  4685 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:24.627581  4364 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:24.658209  4723 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.699s	user 0.003s	sys 0.011s
I20260709 13:02:24.658795  4723 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:24.638829  1665 rpcz_store.cc:269] 0709 13:02:22.657613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.701856 (+ 44243us) service_pool.cc:224] Handling call
0709 13:02:24.637311 (+1935455us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b5400012a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.637448 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":581587840}
I20260709 13:02:24.663128  1584 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:24.657861  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2160) took 1672 ms. Trace:
W20260709 13:02:24.691416  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:24.723187  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 99263.5us
I20260709 13:02:24.450894  4785 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:24.664135  1585 rpcz_store.cc:276] 0709 13:02:22.985269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.016450 (+ 31181us) service_pool.cc:224] Handling call
0709 13:02:23.408441 (+391991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:23.408772 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:23.441919 (+ 33147us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.679664 (+237745us) log.cc:844] Serialized 14 byte log entry
0709 13:02:23.720230 (+ 40566us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:23.720299 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:23.720346 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:23.720445 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.650863 (+930418us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b1800312700. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1
0709 13:02:24.651989 (+  1126us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b1800312700. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:24.657676 (+  5687us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54001fa250. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.657793 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40257,"prepare.queue_time_us":50346,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":4630,"raft.queue_time_us":287196,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":22350,"spinlock_wait_cycles":1013829760,"thread_start_us":236803,"threads_started":1,"wal-append.queue_time_us":273621}
I20260709 13:02:24.738938  1668 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:24.750402  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1604) took 2170 ms (client timeout 1770 ms). Trace:
I20260709 13:02:24.453368  4325 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:24.751544  1666 rpcz_store.cc:269] 0709 13:02:22.563638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.570257 (+  6619us) service_pool.cc:224] Handling call
0709 13:02:24.734209 (+2163952us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b540030b954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:24.734423 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":739500800}
I20260709 13:02:24.443367  4531 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.758725  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 427) took 1218 ms. Trace:
W20260709 13:02:24.759405  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 740628us
I20260709 13:02:24.764190  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:24.766381  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 214240us
I20260709 13:02:24.795179  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.454514  4673 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.767582  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.737809  4739 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:24.763290  4802 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:24.740252  4827 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:24.786271  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:24.468501  4805 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.789295  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:24.575519  4718 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.472970  4728 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:24.810719  2313 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 13:02:24.754243  4723 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:24.815083  1738 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:24.788215  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:24.820555  1736 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:02:24.822300  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.545925  4119 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.765192  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:24.856894  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91289.2us
I20260709 13:02:24.824559  1735 raft_consensus.cc:2376] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8. Re-sending same reply.
W20260709 13:02:24.833865  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:24.837617  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74045.8us
I20260709 13:02:24.816715  4739 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:24.875409  4739 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:24.878293  4812 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:24.844515  1737 raft_consensus.cc:2376] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6. Re-sending same reply.
I20260709 13:02:24.851421  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:24.832046  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:24.823637  1634 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.024s	sys 0.034s
W20260709 13:02:24.883625  1634 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.024s	sys 0.034s
W20260709 13:02:24.884112  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 569950us
I20260709 13:02:24.762683  1592 rpcz_store.cc:276] 0709 13:02:23.540774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.541203 (+   429us) service_pool.cc:224] Handling call
0709 13:02:23.541592 (+   389us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:23.611722 (+ 70130us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.611764 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:23.616282 (+  4518us) log.cc:844] Serialized 78 byte log entry
0709 13:02:23.893917 (+277635us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:23.949090 (+ 55173us) log.cc:844] Serialized 14 byte log entry
0709 13:02:24.033526 (+ 84436us) log.cc:844] Serialized 14 byte log entry
0709 13:02:24.275585 (+242059us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b180030bda0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:24.656550 (+380965us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b180030bda0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:24.656760 (+   210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:24.656835 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:24.656877 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:24.656893 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:24.758655 (+101762us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":504403,"prepare.queue_time_us":55113,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":654791,"raft.run_cpu_time_us":656,"raft.run_wall_time_us":11034,"spinlock_wait_cycles":28899456,"thread_start_us":310298,"threads_started":2,"wal-append.queue_time_us":438285}
I20260709 13:02:24.800688  4304 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 13:02:24.868542  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.828006  4847 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:24.867895  2314 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:24.885959  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:24.905081  1587 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:24.910266  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:24.911466  1588 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:24.912874  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.914315  1592 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 3.
I20260709 13:02:24.916669  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:24.918666  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50230.5us
I20260709 13:02:24.843210  4813 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:24.865664  4851 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:02:24.920475  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:24.860504  4666 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:24.921654  1737 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 4: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:24.922856  1735 pending_rounds.cc:85] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:02:24.928435  1735 pending_rounds.cc:107] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 13:02:24.928893  1735 raft_consensus.cc:2889] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:24.923444  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:24.934489  1742 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:24.935552  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:24.936316  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:24.936894  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:24.938351  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:24.940987  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:24.933183  4732 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 13:02:24.886482  4827 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:24.948738  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:24.950673  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:24.930586  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:24.953332  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:24.955389  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:24.975935  1663 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:24.969101  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:24.977823  1592 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:24.820595  4515 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:24.920369  4783 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:24.980252  1662 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:24.983644  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:24.981554  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51097.5us
I20260709 13:02:24.985034  1737 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:24.928376  4812 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:24.921783  4781 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:24.926980  4723 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:24.830665  4785 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.041173  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:24.932962  4666 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:24.935847  4487 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:25.039789  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:25.043433  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:24.861167  4834 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.120s
I20260709 13:02:25.062141  1666 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 13:02:24.955813  4751 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.011735  1588 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:25.058981  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:24.984957  4304 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:25.072052  1662 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:25.107681  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:24.933337  4813 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:25.050555  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:24.922451  4847 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:25.080581  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.038522  4812 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.055841  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:25.126639  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71527.1us
I20260709 13:02:25.114184  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.043855  4781 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:25.115672  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.117640  1663 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:25.127678  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.118894  1588 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:24.985000  4824 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:25.136776  1664 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [4.4-5.5]
I20260709 13:02:25.107849  4732 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.109413  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:25.167303  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.169414  1734 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.169471  1739 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:25.078896  1585 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:25.058102  4666 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:25.181953  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.182209  4659 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:25.187489  4712 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:25.135309  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.192054  4824 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:25.192915  4824 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:25.194042  4824 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.194794  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 434) took 1484 ms. Trace:
W20260709 13:02:25.199224  4749 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.781s	user 0.010s	sys 0.003s
I20260709 13:02:25.200613  4749 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:25.191936  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56725.1us
I20260709 13:02:25.056509  4515 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.110597  4304 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:25.145934  4531 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.149668  4835 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:25.156468  4847 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:25.111366  4682 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.158051  4673 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.160317  4781 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:25.111214  4723 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:25.133533  4812 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.162478  4364 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.164820  4813 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:25.107738  4487 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:25.166115  4829 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:25.167289  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 442) took 1125 ms. Trace:
I20260709 13:02:25.051458  4783 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.204950  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:25.191875  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1603) took 2630 ms (client timeout 3268 ms). Trace:
W20260709 13:02:25.278366  1668 rpcz_store.cc:269] 0709 13:02:22.561747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.561938 (+   191us) service_pool.cc:224] Handling call
0709 13:02:25.191382 (+2629444us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:25.191730 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5004083584}
I20260709 13:02:25.106654  1740 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:25.281131  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.195434  1667 rpcz_store.cc:276] 0709 13:02:23.710291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.710426 (+   135us) service_pool.cc:224] Handling call
0709 13:02:23.710815 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.989253 (+278438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:23.989323 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:23.990402 (+  1079us) log.cc:844] Serialized 26 byte log entry
0709 13:02:24.041533 (+ 51131us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:24.043758 (+  2225us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.194101 (+1150343us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400260f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:25.194183 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.194261 (+    78us) raft_consensus.cc:1685] finished
0709 13:02:25.194285 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.194695 (+   410us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15798,"raft.queue_time_us":32968,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":96,"spinlock_wait_cycles":602734848,"thread_start_us":31531,"threads_started":1,"wal-append.queue_time_us":161318,"wal-append.run_cpu_time_us":1718,"wal-append.run_wall_time_us":1025285}
I20260709 13:02:25.079911  4785 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:25.170753  4809 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:02:25.221671  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.286888  1734 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:25.222211  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.139621  4737 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:25.071245  4600 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.145s
I20260709 13:02:25.239233  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:25.292325  2318 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:25.217726  4304 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:25.269313  1661 rpcz_store.cc:276] 0709 13:02:24.042336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.194648 (+152312us) service_pool.cc:224] Handling call
0709 13:02:24.581419 (+386771us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:24.581940 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.581985 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:24.734565 (+152580us) log.cc:844] Serialized 26 byte log entry
0709 13:02:25.013049 (+278484us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:25.013131 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.013254 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.166900 (+153646us) raft_consensus.cc:1685] finished
0709 13:02:25.166942 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.167200 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1140,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":5791,"thread_start_us":197074,"threads_started":1,"wal-append.queue_time_us":391176}
W20260709 13:02:25.272795  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 303883us
I20260709 13:02:25.285918  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.299154  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:25.266896  4828 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.814s	user 0.009s	sys 0.007s
I20260709 13:02:25.289247  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:25.301046  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 96217.6us
I20260709 13:02:25.304175  1740 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:25.309155  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.309914  1740 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
W20260709 13:02:25.260869  4506 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.858s	user 0.015s	sys 0.001s
W20260709 13:02:25.290678  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 340005us
I20260709 13:02:25.328258  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.269130  4829 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.306334  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.311363  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:25.316298  2281 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.832s (SENT)
I20260709 13:02:25.264472  4718 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.309630  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.347870  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.275558  4805 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:25.329806  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:25.356463  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.356690  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.234120  4801 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:25.339605  1734 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:25.339937  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:25.318620  4184 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:25.292815  4737 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.293962  4781 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:25.228873  4835 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:25.362264  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46045.1us
I20260709 13:02:25.317695  4749 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:25.364041  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.396216  1740 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 4: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:25.347091  4829 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.341837  4723 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:25.417621  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69868.9us
I20260709 13:02:25.412245  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:25.360131  4828 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:25.421073  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.364991  4682 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:25.427585  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.429004  1740 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.334110  4847 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.086s
I20260709 13:02:25.421736  1665 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:25.374572  4813 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:02:25.436364  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.367437  4801 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:25.422953  4751 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:25.420163  4184 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:25.436528  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:25.448076  1556 connection.cc:256] RPC call timeout handler was delayed by -1.4260370099999591s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20260709 13:02:25.420531  4723 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:25.424691  4828 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.479820  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.480826  1740 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.461086  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:25.413445  4865 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:25.476490  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.373687  4737 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:25.481750  1742 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:25.483484  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.484958  1588 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:25.436981  4824 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:25.490801  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:25.496290  1588 raft_consensus.cc:2376] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7. Re-sending same reply.
I20260709 13:02:25.500900  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:25.503312  1588 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:25.484686  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:25.523524  1740 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.524878  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:25.528743  1592 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.544s	user 0.011s	sys 0.003s
I20260709 13:02:25.529338  1592 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:25.530954  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.532024  1592 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:25.534412  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.469969  4751 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.526593  1740 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:25.542979  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.544118  1740 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.562259  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.567687  4835 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:25.482944  4865 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:25.573984  4865 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:25.489030  4723 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:25.506901  4739 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.618s	user 0.015s	sys 0.000s
I20260709 13:02:25.579448  4739 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.483548  4828 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:25.593132  4857 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 13:02:25.536062  4574 raft_consensus.cc:2764] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:25.458112  4682 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:25.598843  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2191) took 1283 ms. Trace:
I20260709 13:02:25.601473  4682 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.602538  4783 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.606277  4739 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:25.607968  4487 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:25.609953  4861 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.187s	user 0.004s	sys 0.000s
I20260709 13:02:25.611028  4184 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.612330  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.617141  4751 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:25.620594  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.495697  4824 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:25.623202  4824 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.627928  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 616) took 3448 ms. Trace:
I20260709 13:02:25.612442  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.630725  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.631583  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
W20260709 13:02:25.631883  4863 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
W20260709 13:02:25.634382  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1596) took 2737 ms (client timeout 2153 ms). Trace:
I20260709 13:02:25.605288  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:25.636588  1592 raft_consensus.cc:2376] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8. Re-sending same reply.
I20260709 13:02:25.579792  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.613418  1593 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:25.660113  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.632688  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:25.634980  1733 rpcz_store.cc:269] 0709 13:02:22.896560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.989305 (+ 92745us) service_pool.cc:224] Handling call
0709 13:02:25.634014 (+2644709us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540039dfd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:25.634237 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3060252416}
I20260709 13:02:25.673125  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.674515  1733 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:25.675818  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.676679  1733 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:25.563961  1740 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:25.680184  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.681079  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.599318  1589 rpcz_store.cc:276] 0709 13:02:24.315064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.625110 (+310046us) service_pool.cc:224] Handling call
0709 13:02:24.667235 (+ 42125us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:24.790163 (+122928us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.790220 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:24.791001 (+   781us) log.cc:844] Serialized 52 byte log entry
0709 13:02:25.114053 (+323052us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:25.114102 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.114178 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.592380 (+478202us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:25.598436 (+  6056us) raft_consensus.cc:1685] finished
0709 13:02:25.598480 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:25.598756 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":163044,"raft.queue_time_us":133982,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":10141,"spinlock_wait_cycles":336433408,"thread_start_us":290281,"threads_started":1,"wal-append.queue_time_us":494034}
I20260709 13:02:25.681876  1736 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:25.628508  2309 rpcz_store.cc:276] 0709 13:02:22.179596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.181900 (+  2304us) service_pool.cc:224] Handling call
0709 13:02:22.265923 (+ 84023us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:23.881282 (+1615359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.627346 (+1746064us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54000c49d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:25.627834 (+   488us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3155609344}
I20260709 13:02:25.683269  1735 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:25.686502  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 265521us
I20260709 13:02:25.688181  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.688592  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.689428  1589 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:25.689850  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.690203  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:25.693133  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 292930us
W20260709 13:02:25.734818  2281 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.990s (SENT)
I20260709 13:02:25.708820  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.740010  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:25.710587  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:25.724542  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.742019  1589 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.746248  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:25.723853  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.755677  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.756670  1735 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:25.758231  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.690726  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.760110  1593 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:25.779953  4734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.579s	user 0.021s	sys 0.060s
I20260709 13:02:25.780470  4881 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:02:25.782240  4734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.587s	user 0.023s	sys 0.060s
I20260709 13:02:25.782811  4878 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 13:02:25.783458  4881 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:25.783877  4734 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:25.784713  4734 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:25.785218  4878 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:25.785682  4878 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:25.786157  4734 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:25.786573  4878 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:25.788331  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 499213us
I20260709 13:02:25.789319  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:25.792327  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:26.089686  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2175) took 1117 ms. Trace:
I20260709 13:02:25.537752  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.558017  1664 rpcz_store.cc:276] 0709 13:02:24.972633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.973910 (+  1277us) service_pool.cc:224] Handling call
0709 13:02:25.045592 (+ 71682us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:25.137341 (+ 91749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.089257 (+951916us) spinlock_profiling.cc:243] Waited 90.8 ms on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:26.089582 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":208752384}
I20260709 13:02:26.559311  1588 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 4: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:26.560149  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2317) took 1082 ms. Trace:
I20260709 13:02:26.560257  4734 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.560617  1588 rpcz_store.cc:276] 0709 13:02:25.478947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.537213 (+ 58266us) service_pool.cc:224] Handling call
0709 13:02:26.560038 (+1022825us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.000366  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2180) took 1008 ms. Trace:
I20260709 13:02:25.772609  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:26.562013  4834 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.563114  1592 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:25.834903  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45691us
W20260709 13:02:26.132212  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 77475.3us
I20260709 13:02:26.563740  4889 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.564844  4889 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.567229  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:26.562071  1666 rpcz_store.cc:276] 0709 13:02:24.991895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.110484 (+118589us) service_pool.cc:224] Handling call
0709 13:02:25.138402 (+ 27918us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:25.313841 (+175439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:25.313896 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:25.352807 (+ 38911us) log.cc:844] Serialized 26 byte log entry
0709 13:02:25.684108 (+331301us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:25.684156 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.999653 (+315497us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:25.999743 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:25.999845 (+   102us) raft_consensus.cc:1685] finished
0709 13:02:25.999866 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.000261 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46728,"raft.queue_time_us":185,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":15637,"spinlock_wait_cycles":401957760,"thread_start_us":330058,"threads_started":1,"wal-append.queue_time_us":433245}
I20260709 13:02:26.568790  4325 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:26.016372  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.563125  4834 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.569633  4325 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.570144  4325 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:26.572300  4834 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:26.572971  1589 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.830s	user 0.006s	sys 0.018s
I20260709 13:02:26.563715  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2325) took 1082 ms. Trace:
W20260709 13:02:26.430263  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.574180  1592 rpcz_store.cc:276] 0709 13:02:25.481596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.749899 (+268303us) service_pool.cc:224] Handling call
0709 13:02:26.563638 (+813739us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.576016  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:26.576407  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:25.814807  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:26.019132  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2190) took 1036 ms. Trace:
I20260709 13:02:26.003759  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:26.578886  1733 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:26.005306  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:26.579766  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1604) took 1463 ms. Trace:
I20260709 13:02:25.795423  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:26.577174  1592 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:02:26.580561  1733 rpcz_store.cc:276] 0709 13:02:25.115863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.814452 (+698589us) service_pool.cc:224] Handling call
0709 13:02:26.579540 (+765088us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.581430  1735 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:26.582221  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2203) took 1405 ms. Trace:
I20260709 13:02:25.857697  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:26.356251  4785 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.063s	user 0.016s	sys 0.005s
W20260709 13:02:25.883854  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 135648us
I20260709 13:02:26.582638  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:25.834017  4286 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:26.579934  1737 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.586463  4286 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:26.586544  1660 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:25.805006  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:26.588308  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2170) took 1650 ms (client timeout 2118 ms). Trace:
I20260709 13:02:26.588094  4286 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:26.182934  4304 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.851s	user 0.013s	sys 0.002s
W20260709 13:02:26.348023  4782 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.854s	user 0.000s	sys 0.013s
I20260709 13:02:25.988304  4119 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:26.282226  4496 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.017s	sys 0.064s
I20260709 13:02:25.722469  4676 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:26.591300  4496 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.017s	sys 0.067s
W20260709 13:02:26.310870  4666 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.137s	user 0.017s	sys 0.000s
W20260709 13:02:26.591758  4751 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.973s	user 0.011s	sys 0.008s
I20260709 13:02:26.592941  4184 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.578397  1736 rpcz_store.cc:276] 0709 13:02:24.983000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.983241 (+   241us) service_pool.cc:224] Handling call
0709 13:02:25.636119 (+652878us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:25.687188 (+ 51069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.018803 (+331615us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b540039dfd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.019040 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359681408}
I20260709 13:02:25.715097  4640 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:26.593860  4184 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.594796  4640 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.596350  4819 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:26.573732  1589 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:26.590441  4304 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.598531  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2319) took 1118 ms. Trace:
I20260709 13:02:26.599080  1589 rpcz_store.cc:276] 0709 13:02:25.479668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.696667 (+216999us) service_pool.cc:224] Handling call
0709 13:02:26.598419 (+901752us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:25.979580  4685 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.328s	user 0.014s	sys 0.011s
I20260709 13:02:26.601382  4304 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.200889  4728 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:26.602483  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:25.601241  4736 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:26.024737  4800 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.396s	user 0.015s	sys 0.004s
I20260709 13:02:26.619028  4800 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.620632  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:26.175671  4812 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.905s	user 0.007s	sys 0.007s
I20260709 13:02:26.621793  1733 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:26.622359  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2208) took 1392 ms. Trace:
I20260709 13:02:26.622220  4812 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.623942  1741 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:26.624902  4812 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:26.625967  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2194) took 1554 ms (client timeout 1980 ms). Trace:
I20260709 13:02:26.625942  4880 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:02:26.626523  1741 rpcz_store.cc:269] 0709 13:02:25.071643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.071836 (+   193us) service_pool.cc:224] Handling call
0709 13:02:26.625688 (+1553852us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.625888 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3426386304}
W20260709 13:02:26.513302  4487 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.001s	sys 0.001s
I20260709 13:02:26.629571  1740 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:26.631367  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1616) took 1499 ms. Trace:
I20260709 13:02:26.632097  1740 rpcz_store.cc:276] 0709 13:02:25.132473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.757735 (+625262us) service_pool.cc:224] Handling call
0709 13:02:26.631107 (+873372us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.631287 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2001204992}
W20260709 13:02:26.635432  1556 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.571s (SENT) [suppressed 38 similar messages]
I20260709 13:02:26.636926  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.639971  4800 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:26.640229  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:26.127339  4821 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.098s	user 0.002s	sys 0.000s
W20260709 13:02:26.177752  4826 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.921s	user 0.024s	sys 0.000s
I20260709 13:02:26.642354  4826 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:25.715986  4830 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:25.793560  4847 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 13:02:26.002189  4855 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.667s	user 0.006s	sys 0.000s
I20260709 13:02:26.700218  4847 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.700904  4847 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:02:26.198659  4860 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.776s	user 0.009s	sys 0.002s
I20260709 13:02:25.997287  4865 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:25.983601  4866 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.498s	user 0.001s	sys 0.003s
I20260709 13:02:26.751569  4325 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:26.130522  4867 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.118s	user 0.000s	sys 0.002s
W20260709 13:02:26.756889  4737 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.236s	user 0.009s	sys 0.017s
W20260709 13:02:25.984114  4871 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 13:02:26.202178  4874 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.366s	user 0.002s	sys 0.000s
I20260709 13:02:26.087162  4878 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:25.833961  4883 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:26.760998  4824 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.137s	user 0.018s	sys 0.004s
I20260709 13:02:26.514127  4903 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.547314  4787 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.434844  1587 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:26.456369  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2194) took 1555 ms. Trace:
I20260709 13:02:26.433310  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 429) took 2247 ms. Trace:
I20260709 13:02:26.448113  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.578435  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:26.769116  1584 rpcz_store.cc:276] 0709 13:02:24.185382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.202652 (+ 17270us) service_pool.cc:224] Handling call
0709 13:02:24.203022 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:24.663706 (+460684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.431990 (+1768284us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:26.432658 (+   668us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057576704}
I20260709 13:02:26.770257  4800 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:02:26.770431  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 192385us
I20260709 13:02:26.771792  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:26.580078  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.774375  1590 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 13:02:26.775080  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2309) took 1300 ms. Trace:
I20260709 13:02:26.768090  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2174) took 1794 ms. Trace:
I20260709 13:02:26.775480  1590 rpcz_store.cc:276] 0709 13:02:25.474710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.447039 (+972329us) service_pool.cc:224] Handling call
0709 13:02:26.774965 (+327926us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:26.441803  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.000s	sys 0.007s
W20260709 13:02:26.777750  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.002s	sys 0.013s
I20260709 13:02:26.775950  1665 rpcz_store.cc:276] 0709 13:02:24.972168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.972416 (+   248us) service_pool.cc:224] Handling call
0709 13:02:24.973332 (+   916us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:24.974073 (+   741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:24.974118 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:25.055578 (+ 81460us) log.cc:844] Serialized 590 byte log entry
0709 13:02:25.421430 (+365852us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:25.515533 (+ 94103us) log.cc:844] Serialized 14 byte log entry
0709 13:02:25.678814 (+163281us) log.cc:844] Serialized 14 byte log entry
0709 13:02:25.679291 (+   477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:25.679429 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.012031 (+332602us) spinlock_profiling.cc:243] Waited 37.1 ms on lock 0x7b1800139000. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:26.029582 (+ 17551us) spinlock_profiling.cc:243] Waited 4.93 ms on lock 0x7b1800139000. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:26.765982 (+736400us) raft_consensus.cc:1685] finished
0709 13:02:26.766037 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.766336 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":165297,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":2070,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":52486,"spinlock_wait_cycles":96703104,"thread_start_us":294289,"threads_started":1,"wal-append.queue_time_us":477338}
W20260709 13:02:26.778177  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.03674e+06us
I20260709 13:02:26.779130  4325 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:26.780304  4325 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:26.780928  4325 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:26.465052  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2184) took 1425 ms. Trace:
W20260709 13:02:26.458563  1708 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.823s	user 0.002s	sys 0.009s
I20260709 13:02:26.766523  4737 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.582635  1735 rpcz_store.cc:276] 0709 13:02:25.176345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.795060 (+618715us) service_pool.cc:224] Handling call
0709 13:02:26.582112 (+787052us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.585575  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1603) took 1471 ms. Trace:
W20260709 13:02:26.047339  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.794332  4865 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:26.590421  4782 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.116s	user 0.000s	sys 0.014s
I20260709 13:02:26.525792  4600 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:26.177227  4712 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.989s	user 0.017s	sys 0.000s
I20260709 13:02:26.781998  4903 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.602134  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.601293  4685 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.809091  1659 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:26.809379  4600 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:26.810645  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1606) took 4208 ms (client timeout 1571 ms). Trace:
I20260709 13:02:26.810850  4903 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:26.811347  4685 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:26.811113  1659 rpcz_store.cc:269] 0709 13:02:22.602234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:22.602452 (+   218us) service_pool.cc:224] Handling call
0709 13:02:26.810354 (+4207902us) spinlock_profiling.cc:243] Waited 4.04 s on lock 0x7b54002535d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.810543 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9291227904}
I20260709 13:02:26.592844  4666 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.812891  4600 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:26.178257  4680 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.545s	user 0.014s	sys 0.014s
I20260709 13:02:26.814049  2309 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:26.814355  4680 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.782346  4787 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.604775  4184 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.822230  4680 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:26.320451  4732 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.113s	user 0.023s	sys 0.003s
I20260709 13:02:26.823136  4787 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:26.823688  4732 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:26.601900  4286 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:26.605372  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107951us
W20260709 13:02:26.826123  1557 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.003s (SENT)
I20260709 13:02:26.826689  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:26.827104  4808 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.592594  4751 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.827924  4808 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.606254  1592 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:26.830924  4364 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:26.831125  4782 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.588846  1660 rpcz_store.cc:269] 0709 13:02:24.937974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.938118 (+   144us) service_pool.cc:224] Handling call
0709 13:02:26.587998 (+1649880us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.588173 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3788242176}
I20260709 13:02:26.831717  4364 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.595355  4640 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:26.831954  4782 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:26.349176  4781 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.953s	user 0.015s	sys 0.002s
I20260709 13:02:26.836100  4781 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:26.836876  4781 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.614753  4304 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.837699  4304 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:26.365340  4892 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:02:26.622779  1733 rpcz_store.cc:276] 0709 13:02:25.230042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.620221 (+1390179us) service_pool.cc:224] Handling call
0709 13:02:26.622272 (+  2051us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.653388  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1607) took 1526 ms. Trace:
I20260709 13:02:26.841779  1736 rpcz_store.cc:276] 0709 13:02:25.117552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.595282 (+1477730us) service_pool.cc:224] Handling call
0709 13:02:26.643890 (+ 48608us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.568717  4889 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:26.385474  4829 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.959s	user 0.014s	sys 0.006s
I20260709 13:02:26.851490  4829 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.798597  4824 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.689671  1593 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:26.853991  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2326) took 1372 ms. Trace:
I20260709 13:02:26.854398  1593 rpcz_store.cc:276] 0709 13:02:25.481861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.761147 (+279286us) service_pool.cc:224] Handling call
0709 13:02:26.853894 (+1092747us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.854945  4824 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.855286  4829 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:26.851507  4723 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.273s	user 0.015s	sys 0.008s
I20260709 13:02:26.855831  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:26.856370  4723 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:26.690680  4828 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.098s	user 0.013s	sys 0.006s
I20260709 13:02:26.858001  4828 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.858875  4723 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.860303  4828 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.695250  1740 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.3-6.3]
W20260709 13:02:26.696940  1742 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.211s	user 0.018s	sys 0.011s
I20260709 13:02:26.861256  1742 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:26.861452  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1611) took 1742 ms. Trace:
W20260709 13:02:26.861917  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2199) took 1687 ms (client timeout 1950 ms). Trace:
W20260709 13:02:26.862360  1742 rpcz_store.cc:269] 0709 13:02:25.174845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.460536 (+285691us) service_pool.cc:224] Handling call
0709 13:02:26.861837 (+1401301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.864959  4712 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.737776  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.007s	sys 0.026s
W20260709 13:02:26.865700  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.048s	user 0.007s	sys 0.026s
I20260709 13:02:26.865882  4184 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.866470  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.866873  4782 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:26.866663  4184 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.867492  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:26.868530  4364 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.869019  4184 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:26.738278  4798 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.739025  4830 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:26.740872  1737 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:26.451558  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2176) took 2873 ms. Trace:
I20260709 13:02:26.761991  1739 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:26.784063  1663 rpcz_store.cc:276] 0709 13:02:25.039185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.145530 (+106345us) service_pool.cc:224] Handling call
0709 13:02:25.146676 (+  1146us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:25.307286 (+160610us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:25.307340 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:25.358629 (+ 51289us) log.cc:844] Serialized 564 byte log entry
0709 13:02:25.848240 (+489611us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:25.848309 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.463437 (+615128us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b540013d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:26.463597 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:26.463742 (+   145us) raft_consensus.cc:1685] finished
0709 13:02:26.463765 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.464632 (+   867us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368183680,"thread_start_us":489103,"threads_started":1,"wal-append.queue_time_us":575956}
W20260709 13:02:26.784322  1708 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.149s	user 0.008s	sys 0.015s
W20260709 13:02:26.785104  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.782691  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2321) took 1302 ms. Trace:
I20260709 13:02:26.869892  4798 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.790874  1738 rpcz_store.cc:276] 0709 13:02:25.114397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.816544 (+702147us) service_pool.cc:224] Handling call
0709 13:02:26.585268 (+768724us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.790485  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:26.879756  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89417.2us
I20260709 13:02:26.880715  4902 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.880502  4515 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.662s	user 0.007s	sys 0.023s
I20260709 13:02:26.808456  4736 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.881671  4902 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.561219  4734 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:26.882580  4904 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:02:26.836753  4883 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.884423  4902 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.884246  4883 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.838168  1589 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:26.615453  4834 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:26.878934  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1605) took 1762 ms. Trace:
I20260709 13:02:26.811821  1735 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 13:02:26.886269  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.888636  4834 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.865721  4712 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.878072  4600 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:26.891248  4600 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.891402  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.891814  4600 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.892490  1589 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:26.850275  1591 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:26.893198  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:26.893936  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 161236us
I20260709 13:02:26.894989  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:26.698493  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.895778  4839 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.895640  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:26.896530  4839 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.887491  1735 pending_rounds.cc:85] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:02:26.897265  1734 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:26.897732  1735 pending_rounds.cc:107] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 13:02:26.898229  1735 raft_consensus.cc:2889] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:26.898386  4876 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:26.898839  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2206) took 1721 ms (client timeout 1843 ms). Trace:
I20260709 13:02:26.899118  4876 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:26.899310  1734 rpcz_store.cc:269] 0709 13:02:25.209341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.360876 (+151535us) service_pool.cc:224] Handling call
0709 13:02:26.898601 (+1537725us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400284954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.898778 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3404424960}
I20260709 13:02:26.901070  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:26.739953  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.871176  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:26.861943  1740 rpcz_store.cc:276] 0709 13:02:25.119021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.633482 (+1514461us) service_pool.cc:224] Handling call
0709 13:02:26.694804 (+ 61322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.861094 (+166290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.861353 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.782706  1590 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260709 13:02:26.878212  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2196) took 1764 ms (client timeout 1882 ms). Trace:
I20260709 13:02:26.873588  1586 rpcz_store.cc:276] 0709 13:02:23.577641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.638469 (+ 60828us) service_pool.cc:224] Handling call
0709 13:02:26.450876 (+2812407us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54000fa854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.451239 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":819912064}
I20260709 13:02:26.875454  1587 rpcz_store.cc:276] 0709 13:02:25.480272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.620179 (+139907us) service_pool.cc:224] Handling call
0709 13:02:26.782562 (+1162383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.881340  4808 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.880472  4119 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:26.781463  4325 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.937965  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:26.831975  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:26.830960  4751 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.768797  1585 rpcz_store.cc:276] 0709 13:02:24.901237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:24.922710 (+ 21473us) service_pool.cc:224] Handling call
0709 13:02:26.455843 (+1533133us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.456140 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233955584}
I20260709 13:02:26.943514  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:26.892302  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.944269  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2311) took 1460 ms. Trace:
W20260709 13:02:26.895104  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2166) took 3777 ms (client timeout 1953 ms). Trace:
I20260709 13:02:26.847939  1733 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:26.902024  4890 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:26.891755  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.895361  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.901868  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.946316  1663 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:26.903733  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.944842  1590 rpcz_store.cc:276] 0709 13:02:25.475859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.776409 (+1300550us) service_pool.cc:224] Handling call
0709 13:02:26.777179 (+   770us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.935719 (+158540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.936022 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.948314  1590 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:26.947099  1665 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:26.950067  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2198) took 1821 ms. Trace:
I20260709 13:02:26.950480  1733 rpcz_store.cc:276] 0709 13:02:25.128712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.847256 (+1718544us) service_pool.cc:224] Handling call
0709 13:02:26.847717 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.949807 (+102090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.950006 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.865800  4781 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:26.906108  4666 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.899173  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1614) took 1779 ms. Trace:
I20260709 13:02:26.905314  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:26.911038  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.27566e+06us
I20260709 13:02:26.871541  4800 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:26.865175  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.888155  1593 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
W20260709 13:02:26.926334  1737 rpcz_store.cc:269] 0709 13:02:25.113796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.123684 (+  9888us) service_pool.cc:224] Handling call
0709 13:02:26.003481 (+879797us) spinlock_profiling.cc:243] Waited 2.58 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:26.873431 (+869950us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5933952}
I20260709 13:02:26.788385  4737 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.939677  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 627) took 1098 ms. Trace:
I20260709 13:02:26.940864  1587 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:02:26.875322  4830 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:26.942001  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:26.942315  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.942795  1592 raft_consensus.cc:2376] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7. Re-sending same reply.
W20260709 13:02:26.945051  1591 rpcz_store.cc:269] 0709 13:02:23.118138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:23.118414 (+   276us) service_pool.cc:224] Handling call
0709 13:02:26.894736 (+3776322us) spinlock_profiling.cc:243] Waited 3.66 s on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:26.894999 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8424294656}
W20260709 13:02:26.937107  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:26.955417  1735 rpcz_store.cc:276] 0709 13:02:25.120066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.790171 (+1670105us) service_pool.cc:224] Handling call
0709 13:02:26.790636 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.898874 (+108238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.899100 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.948961  4304 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.960614  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.887995  1741 rpcz_store.cc:276] 0709 13:02:25.116684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.627619 (+1510935us) service_pool.cc:224] Handling call
0709 13:02:26.628181 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.628763 (+   582us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:26.727654 (+ 98891us) log.cc:844] Serialized 14 byte log entry
0709 13:02:26.878268 (+150614us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:26.878361 (+    93us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:26.878450 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.878593 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.878847 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155792,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":3754,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":221,"thread_start_us":173882,"threads_started":2,"wal-append.queue_time_us":180808}
I20260709 13:02:26.967676  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:26.936569  4902 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:26.964113  1586 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:26.976998  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2315) took 1489 ms. Trace:
I20260709 13:02:26.978220  1587 rpcz_store.cc:276] 0709 13:02:25.477610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.940009 (+1462399us) service_pool.cc:224] Handling call
0709 13:02:26.940577 (+   568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.966283 (+ 25706us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.966548 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.977326  1738 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:26.985414  1659 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:26.986251  1735 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:26.987088  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2215) took 1616 ms. Trace:
I20260709 13:02:26.987421  4712 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.989431  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2213) took 1684 ms. Trace:
I20260709 13:02:26.961529  2309 rpcz_store.cc:276] 0709 13:02:25.814129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.818372 (+  4243us) service_pool.cc:224] Handling call
0709 13:02:25.819033 (+   661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.814738 (+995705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.878641 (+ 63903us) spinlock_profiling.cc:243] Waited 994 ms on lock 0x7b54003ef154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:26.912067 (+ 33426us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2287231232}
I20260709 13:02:26.991671  4728 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.992416  4728 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:26.997697  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:26.999320  1593 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:27.003029  4680 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:27.004027  4680 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:27.007128  1557 leader_election.cc:336] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.552s (SENT)
I20260709 13:02:27.007695  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:26.998015  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.008749  1665 raft_consensus.cc:2368] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:27.009652  4800 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.012095  4781 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.012617  4800 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.012897  4781 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:26.957226  4732 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:26.987453  1735 rpcz_store.cc:276] 0709 13:02:25.370563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.980786 (+1610223us) service_pool.cc:224] Handling call
0709 13:02:26.985978 (+  5192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.986758 (+   780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.986998 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:27.013935  4732 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.016744  4732 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:27.009176  1557 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.571s (SENT)
I20260709 13:02:26.990098  1740 rpcz_store.cc:276] 0709 13:02:25.305971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.955318 (+1649347us) service_pool.cc:224] Handling call
0709 13:02:26.988522 (+ 33204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.988985 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.989024 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:26.989075 (+    51us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:26.989143 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:26.989206 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:26.989335 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:26.962608  4723 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:26.954844  4734 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:26.995286  4728 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.030891  1668 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:27.038784  4600 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:26.950251  4847 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.054728  1739 raft_consensus.cc:2410] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:27.044816  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.059626  1665 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 13:02:27.073334  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54763.3us
W20260709 13:02:27.062526  1557 leader_election.cc:336] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.675s (SENT)
W20260709 13:02:27.075980  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 206601us
W20260709 13:02:27.081171  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62018.5us
I20260709 13:02:27.082594  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.044607  4723 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.091356  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:27.100395  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1620) took 1965 ms (client timeout 1552 ms). Trace:
W20260709 13:02:27.100917  1739 rpcz_store.cc:269] 0709 13:02:25.135253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:25.170663 (+ 35410us) service_pool.cc:224] Handling call
0709 13:02:26.005071 (+834408us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:27.100042 (+1094971us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540012e3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:27.100292 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456930944}
I20260709 13:02:27.104135  1734 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:27.106101  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:27.107084  2314 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:27.109864  4876 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:27.110711  4737 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:27.115145  4913 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:02:27.078105  4640 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.150084  4737 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:27.153939  4640 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.156364  4680 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.156693  4723 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.161056  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2212) took 1865 ms. Trace:
W20260709 13:02:27.149725  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91939.3us
I20260709 13:02:27.168497  1733 rpcz_store.cc:276] 0709 13:02:25.295574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.955051 (+1659477us) service_pool.cc:224] Handling call
0709 13:02:27.000930 (+ 45879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.001561 (+   631us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:27.002135 (+   574us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.060235 (+ 58100us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.160415 (+100180us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b18004a5b40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:27.160641 (+   226us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:27.160694 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:27.160789 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.160876 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.160994 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34282,"prepare.queue_time_us":1487,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":88,"raft.queue_time_us":112236,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":32573696,"thread_start_us":43041,"threads_started":1}
I20260709 13:02:27.057731  1584 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Committing config change with OpId 4.4: config changed from index 2 to 4, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:27.182224  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2214) took 1841 ms. Trace:
I20260709 13:02:27.185395  1737 rpcz_store.cc:276] 0709 13:02:25.341064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.966668 (+1625604us) service_pool.cc:224] Handling call
0709 13:02:27.053809 (+ 87141us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.181802 (+127993us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 13:02:27.181843 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:27.181889 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:27.181919 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.181995 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.182152 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96354,"raft.queue_time_us":109265}
I20260709 13:02:27.149822  4828 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.190438  1738 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:27.153898  4737 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:27.194242  1557 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.770s (SENT)
I20260709 13:02:27.186992  4666 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:27.203401  1659 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:27.191646  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.203357  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.213649  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.218292  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.205917  1666 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.213708  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.224797  1585 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.231521  4781 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:02:27.234207  4325 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.236557  4325 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:27.191712  4828 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:27.241865  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50311.9us
I20260709 13:02:27.241492  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.243674  4736 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:27.253961  4902 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:27.256819  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2200) took 2073 ms. Trace:
I20260709 13:02:27.258474  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.261101  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.262001  1661 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:27.262277  1659 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:27.257366  1736 rpcz_store.cc:276] 0709 13:02:25.175443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.848350 (+1672907us) service_pool.cc:224] Handling call
0709 13:02:26.848838 (+   488us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:26.849418 (+   580us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.849447 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:26.850171 (+   724us) log.cc:844] Serialized 52 byte log entry
0709 13:02:27.021000 (+170829us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:27.021040 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.021108 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:27.248820 (+227712us) raft_consensus.cc:1685] finished
0709 13:02:27.248842 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.248995 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4723,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":170291,"threads_started":1,"wal-append.queue_time_us":344523}
I20260709 13:02:27.277611  2314 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:27.289052  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.295933  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.296880  1666 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
W20260709 13:02:27.297668  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.298161  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.306070  1666 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.310110  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.311249  1587 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:27.313997  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.322204  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:27.329699  4919 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.181s	user 0.000s	sys 0.002s
I20260709 13:02:27.338338  2313 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:27.339190  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:27.341627  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 151224us
I20260709 13:02:27.345934  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.347239  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.347462  1590 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:27.342865  1592 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260709 13:02:27.337644  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.357901  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.343253  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:27.363056  1659 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
W20260709 13:02:27.343461  4924 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:02:27.380774  1666 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:27.382143  1661 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:02:27.384721  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2312) took 1908 ms. Trace:
I20260709 13:02:27.385133  1589 rpcz_store.cc:276] 0709 13:02:25.476643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.938184 (+1461541us) service_pool.cc:224] Handling call
0709 13:02:27.018039 (+ 79855us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:27.018489 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:27.018526 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:27.019155 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.184625 (+165470us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:27.184667 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.184748 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:27.384496 (+199748us) raft_consensus.cc:1685] finished
0709 13:02:27.384517 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.384655 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":113122,"threads_started":1,"wal-append.queue_time_us":205913}
I20260709 13:02:27.386940  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.358457  4685 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:02:27.392037  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2306) took 1909 ms. Trace:
I20260709 13:02:27.391988  4830 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.393524  1665 raft_consensus.cc:2368] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:27.381371  4915 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.392652  1584 rpcz_store.cc:276] 0709 13:02:25.473262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.771002 (+1297740us) service_pool.cc:224] Handling call
0709 13:02:26.771723 (+   721us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:26.772147 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:26.846199 (+ 74052us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.379025 (+532826us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.380697 (+  1672us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.381883 (+  1186us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:27.381945 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:27.381995 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.382090 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.382238 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":516,"prepare.queue_time_us":131587,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":370,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":30590,"spinlock_wait_cycles":15818112,"thread_start_us":144197,"threads_started":2,"wal-append.queue_time_us":126606}
I20260709 13:02:27.394322  4915 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.342361  4847 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.397660  1587 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.401408  1585 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:27.347467  4798 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.403867  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.407990  1592 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:27.414666  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.423606  1593 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 13:02:27.418736  4819 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.431370  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2204) took 2253 ms. Trace:
I20260709 13:02:27.431107  1587 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:27.433943  1742 rpcz_store.cc:276] 0709 13:02:25.176593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.863176 (+1686583us) service_pool.cc:224] Handling call
0709 13:02:26.863788 (+   612us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:26.864436 (+   648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:26.864468 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:26.865212 (+   744us) log.cc:844] Serialized 52 byte log entry
0709 13:02:26.883582 (+ 18370us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b18004a5e40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:27.181998 (+298416us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:27.182052 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.182203 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:27.429538 (+247335us) raft_consensus.cc:1685] finished
0709 13:02:27.429563 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.429820 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25901,"raft.queue_time_us":6844,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"spinlock_wait_cycles":5235456,"thread_start_us":308962,"threads_started":1,"wal-append.queue_time_us":469733}
I20260709 13:02:27.427955  4890 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:27.440421  1593 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 13:02:27.440757  1593 pending_rounds.cc:85] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 7
I20260709 13:02:27.442327  1593 pending_rounds.cc:107] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 13:02:27.442592  1593 raft_consensus.cc:2889] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:27.439852  4839 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.441560  4890 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:27.444041  4890 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.445987  4883 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.452594  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.453542  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.453825  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.454303  1585 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:27.454634  1584 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.457218  4915 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.459471  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2310) took 1984 ms. Trace:
I20260709 13:02:27.452780  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.461861  1590 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.464893  4781 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:02:27.453493  1587 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:27.466144  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.469883  4834 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.459959  1588 rpcz_store.cc:276] 0709 13:02:25.475269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.774939 (+1299670us) service_pool.cc:224] Handling call
0709 13:02:26.837323 (+ 62384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:26.837962 (+   639us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:26.968899 (+130937us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.146906 (+178007us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:27.184629 (+ 37723us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.185896 (+  1267us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:27.186172 (+   276us) log.cc:844] Serialized 14 byte log entry
0709 13:02:27.186372 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.186453 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:27.387413 (+200960us) raft_consensus.cc:1685] finished
0709 13:02:27.387435 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:27.459253 (+ 71818us) spinlock_profiling.cc:243] Waited 61.8 ms on lock 0x7b54002701d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:27.459397 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":67267,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":5640,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":43666,"spinlock_wait_cycles":142182784,"thread_start_us":219747,"threads_started":2,"wal-append.queue_time_us":319109}
I20260709 13:02:27.476809  1584 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:27.477734  1592 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:27.483148  1590 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:27.483202  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.480686  4830 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.485013  1584 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.435302  4819 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.489390  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.488086  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:27.491369  4685 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:27.492087  1664 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:27.492209  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.492928  1592 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.494172  4685 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:27.494696  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.495047  4685 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:27.495393  1593 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:27.495512  4685 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:27.496919  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2211) took 2206 ms. Trace:
I20260709 13:02:27.497334  1734 rpcz_store.cc:276] 0709 13:02:25.290400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:26.956166 (+1665766us) service_pool.cc:224] Handling call
0709 13:02:27.103892 (+147726us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:27.104722 (+   830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.496732 (+392010us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540012e3d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:27.496864 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266077440}
I20260709 13:02:27.497812  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.498719  4364 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:27.499109  4364 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.499460  4364 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:27.500402  4929 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 13:02:27.496701  4685 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.503187  4685 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.490043  4781 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.503101  1592 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
W20260709 13:02:27.511548  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 314322us
I20260709 13:02:27.511940  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.512562  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:27.513114  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.513298  1590 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:27.515007  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:27.517102  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.531054  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.531589  4364 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:27.535776  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 192746us
I20260709 13:02:27.540318  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.541239  1588 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.548236  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.516304  4890 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.506069  4781 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:27.551913  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.551820  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.552762  1587 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:27.553035  1586 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:27.554250  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.525465  4325 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:27.563470  1739 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
W20260709 13:02:27.563966  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.564234  1584 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
W20260709 13:02:27.551473  4931 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.182s	user 0.000s	sys 0.004s
I20260709 13:02:27.565729  1585 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:27.542807  4515 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:27.566990  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67435.8us
I20260709 13:02:27.543772  4723 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:27.538686  4800 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.570195  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.571017  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:27.571946  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 57016.1us
I20260709 13:02:27.578202  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.582500  4834 raft_consensus.cc:2764] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:27.565941  4826 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.563750  4906 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:27.569347  4666 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.569923  4723 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.585140  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.592742  1742 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.592597  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.584546  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.569589  4325 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:27.569836  4515 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:27.593261  1586 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.594405  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.606002  1584 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.610184  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.611152  1557 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.613s (SENT)
I20260709 13:02:27.611660  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:27.612921  1734 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:27.612908  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.613937  1742 raft_consensus.cc:2368] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:27.594966  4723 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.606104  4515 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:27.623946  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.624239  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.630144  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:27.632510  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64168.1us
I20260709 13:02:27.577253  4800 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.629954  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.637048  1585 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:27.638554  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.585668  4883 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:02:27.642342  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:27.639742  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75903.3us
I20260709 13:02:27.635566  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.642951  1661 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.640179  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.643680  1659 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:27.616254  4736 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:27.645473  4878 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:27.646215  4781 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.592006  4826 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.646436  4819 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.642189  4834 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:27.648146  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.649850  4936 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:27.650117  1739 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:27.633314  4934 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260709 13:02:27.650501  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:27.645516  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 387174us
I20260709 13:02:27.650820  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.688351  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.688601  1668 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:27.652560  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.594319  4906 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.651137  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.630362  4364 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:27.682303  1584 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:27.682369  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.624387  4666 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.690989  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.673485  1661 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:27.699697  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.706724  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.653638  4878 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:27.712180  2317 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.733167  1736 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.712921  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.660379  4737 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:27.714330  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:27.715068  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.715634  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 104536us
I20260709 13:02:27.716256  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:27.718274  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87036.2us
I20260709 13:02:27.687625  4781 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.720593  4906 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.660130  4808 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.690524  4839 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:27.659794  4902 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:27.752633  4902 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.754501  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.755499  1742 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.756558  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.757663  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.758263  1557 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.842s (SENT)
I20260709 13:02:27.746718  4839 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:27.759168  4839 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.759289  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.760139  1739 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.761023  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.762404  1668 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:27.724056  4826 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:27.724381  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:27.763237  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52501.5us
I20260709 13:02:27.763962  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.764070  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:27.764871  1738 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 13:02:27.766144  1738 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260709 13:02:27.723130  4890 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.766474  1738 pending_rounds.cc:85] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:02:27.766814  1738 pending_rounds.cc:107] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260709 13:02:27.766907  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.767544  1663 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:27.767858  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.768500  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.755667  1734 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:27.770170  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.725368  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.770841  1742 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 4.
I20260709 13:02:27.773213  4834 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:27.777155  4834 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:27.729465  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.777594  4834 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:27.707082  4325 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.778594  4737 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:27.779743  4737 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:27.779853  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.780141  4737 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:27.780731  1664 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:27.731458  1556 leader_election.cc:400] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:27.734396  1661 raft_consensus.cc:2368] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:27.781121  4737 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.781715  4728 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:27.782333  4737 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.782596  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.783800  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.784826  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.743321  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.786144  1741 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.709968  1667 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:02:27.745538  4781 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.786931  1667 raft_consensus.cc:606] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.781765  1556 leader_election.cc:403] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:27.788082  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.788463  4781 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:27.741978  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91554.5us
I20260709 13:02:27.789915  4834 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.750504  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.798226  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:27.798606  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48325.9us
I20260709 13:02:27.799338  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.799989  4736 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:27.800447  4736 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:27.800829  4736 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:27.751375  4808 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.755626  4902 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:27.801870  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.802093  4902 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:27.802470  4902 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:27.802850  4902 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.803844  4808 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:27.804049  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.804348  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.804817  2315 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:27.805558  4487 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:27.805862  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:27.806533  2317 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:27.807765  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.808723  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:27.754637  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.809762  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:27.810037  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.810660  1661 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:27.811160  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118930us
I20260709 13:02:27.767072  1738 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:27.725081  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.781630  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 127545us
I20260709 13:02:27.681560  4938 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:02:27.812702  1733 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:27.746258  4915 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:27.815156  1586 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:27.794571  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.797013  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:27.816579  1590 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:27.817703  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.818305  1590 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.823462  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:27.819983  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:27.823854  1705 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.118s (SENT)
I20260709 13:02:27.824394  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:27.827747  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.789290  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.762549  4847 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:27.765923  4937 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:27.788800  4640 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:27.813862  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.830569  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:27.794094  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:27.816690  4734 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:27.831545  1589 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:27.833309  1665 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:27.799404  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.834543  1659 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:27.835229  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:27.835682  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.737087  4878 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:27.813004  4666 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:27.825923  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55053.6us
I20260709 13:02:27.852210  1588 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:27.747217  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.828051  1668 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:27.864332  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:27.816723  4487 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:27.867075  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 43738.4us
I20260709 13:02:27.873253  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:27.873988  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:27.874204  1738 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:27.824669  4834 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.869540  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:27.877539  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:27.831034  4781 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:27.829612  4847 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.801404  4839 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.829923  4883 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:27.837965  4734 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:27.838222  4640 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:27.891144  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:27.893671  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 129728us
I20260709 13:02:27.898527  1667 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:27.897708  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112278us
I20260709 13:02:27.901878  1590 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:27.879184  4487 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.872419  4915 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:27.883128  1659 raft_consensus.cc:2368] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:27.878693  4286 raft_consensus.cc:2764] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:27.904356  1664 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:27.904901  1661 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:27.892964  4847 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.883494  4798 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:27.904291  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:27.942571  4915 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.903563  4734 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:27.948398  4941 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:27.949019  4941 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:27.949481  4941 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.886525  4830 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 13:02:27.893174  4939 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:27.893472  4685 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:27.931813  4487 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:27.973884  1708 leader_election.cc:400] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:27.986291  4286 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:27.987519  4685 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:27.990201  4685 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:27.991995  4940 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 13:02:27.991639  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112537us
I20260709 13:02:27.993125  1632 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:27.990971  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51719.3us
I20260709 13:02:27.899200  4723 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:27.897580  4666 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 13:02:27.939441  4883 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.013652  1662 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
W20260709 13:02:28.013688  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 238693us
I20260709 13:02:28.021064  1589 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:28.023545  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.029479  1737 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 4.
I20260709 13:02:28.032083  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.051159  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:28.062063  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 194880us
I20260709 13:02:28.067022  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.041169  4325 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.047771  4781 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:28.079880  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54529.5us
I20260709 13:02:28.079030  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.082525  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.071352  4943 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:28.087085  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:28.087976  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.087126  1708 leader_election.cc:400] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
W20260709 13:02:28.095603  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 283043us
I20260709 13:02:28.080060  4830 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:28.075898  4781 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:28.049896  4941 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.104671  1662 raft_consensus.cc:606] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:28.086473  4943 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.043787  4640 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.112553  1662 raft_consensus.cc:614] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:28.093683  4828 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.100664  4954 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:28.130360  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 271182us
I20260709 13:02:28.132110  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:28.131270  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55505.9us
I20260709 13:02:28.132616  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:28.133821  1661 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:28.135084  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:28.148531  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61575.1us
I20260709 13:02:28.139824  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.151154  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:28.122918  4943 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.126036  4839 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.127501  4808 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:28.175851  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87976us
I20260709 13:02:28.176735  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.185067  1634 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.160564  4828 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:28.155867  4954 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:28.160822  4685 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:28.202494  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.207999  1557 leader_election.cc:400] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:28.208601  1557 leader_election.cc:403] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.137801  4734 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.214044  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.161638  4955 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:28.171667  4830 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:28.206262  4949 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 13:02:28.178097  4723 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.195528  4828 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:28.255329  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 115633us
I20260709 13:02:28.259582  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.260802  1663 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:28.241770  4515 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.597s	user 0.012s	sys 0.000s
I20260709 13:02:28.261956  4515 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.213755  4600 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:28.216362  4958 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:28.159186  4883 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:28.253089  4948 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 13:02:28.211263  4839 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
W20260709 13:02:28.282428  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 71282.8us
I20260709 13:02:28.245868  4685 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:28.259164  4828 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.258584  4965 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:28.255687  4955 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:28.313642  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:28.314405  1592 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.314546  4955 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.264528  4515 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:28.299129  4839 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.299757  4947 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.300236  4910 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:28.303401  4734 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:28.282666  4600 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.307494  4964 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:28.309417  4967 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:28.253144  4956 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:28.271706  4902 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:28.359660  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 227689us
W20260709 13:02:28.318213  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 132306us
I20260709 13:02:28.275260  4960 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:28.347847  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 145440us
I20260709 13:02:28.327579  4839 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:28.257535  4781 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:28.348078  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52714.3us
W20260709 13:02:28.321238  4736 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.520s	user 0.007s	sys 0.007s
I20260709 13:02:28.317278  4954 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:28.291747  4968 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:28.361060  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115470us
W20260709 13:02:28.370030  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 205134us
W20260709 13:02:28.345098  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82618.7us
I20260709 13:02:28.372160  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.372411  1592 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:28.373195  1556 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:28.373575  1556 leader_election.cc:403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.357422  4910 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:28.375349  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:28.377936  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.378482  4960 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:28.366554  4967 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.379509  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:28.379731  4967 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.380589  4963 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:28.381104  4963 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:28.269493  4325 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 2 to 6, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:28.382225  4963 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.365137  4830 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:28.375392  4737 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.593s	user 0.011s	sys 0.003s
I20260709 13:02:28.368669  4964 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:28.390935  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.369112  4954 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.402132  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.397864  4974 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:28.389989  4685 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.405331  4781 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:28.404940  4910 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:28.409114  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47174.2us
I20260709 13:02:28.390115  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.413466  1666 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:28.417948  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.371223  4736 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.398321  4968 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.430004  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.411532  4954 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.413580  4515 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:28.436339  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66150.5us
W20260709 13:02:28.432978  4834 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.010s
I20260709 13:02:28.419351  4960 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.413594  4956 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:28.444861  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.449465  4969 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:28.433208  4839 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.450755  4969 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.450930  4839 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:28.451861  4969 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:28.452247  4969 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:28.452824  4969 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.416462  4974 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.456213  4736 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:28.456827  4902 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:28.442731  4515 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.460151  4964 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.466272  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.466600  4515 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:28.467227  4515 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:02:28.467738  4515 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.467921  1556 leader_election.cc:400] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:28.468313  1556 leader_election.cc:403] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.469041  4834 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.469856  4834 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
W20260709 13:02:28.472594  4878 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.606s	user 0.013s	sys 0.001s
I20260709 13:02:28.476787  1661 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49478
I20260709 13:02:28.478062  1661 raft_consensus.cc:606] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Received request to transfer leadership
I20260709 13:02:28.478446  1661 raft_consensus.cc:614] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:02:28.478834  4847 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.005s
I20260709 13:02:28.479329  4847 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.473053  4878 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.480263  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.481190  2313 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:28.482273  1741 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:28.483120  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.484860  1738 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
W20260709 13:02:28.485643  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 111173us
I20260709 13:02:28.485874  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.486483  1740 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.486536  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.486888  4878 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.486685  4828 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.488174  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.489027  4828 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.468726  4515 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.431058  4962 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:28.496160  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78340.5us
W20260709 13:02:28.492434  4945 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.546s	user 0.001s	sys 0.003s
I20260709 13:02:28.500823  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:28.501641  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.502336  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.506188  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.545760  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.546516  1741 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:28.547679  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 100915us
I20260709 13:02:28.548198  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:28.548435  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103737us
I20260709 13:02:28.552012  1592 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.552205  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2225) took 1060 ms. Trace:
I20260709 13:02:28.552616  1593 rpcz_store.cc:276] 0709 13:02:27.492167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.496216 (+  4049us) service_pool.cc:224] Handling call
0709 13:02:27.496568 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:27.496833 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:27.496859 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:27.497450 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 13:02:27.909238 (+411788us) raft_consensus.cc:1645] Marking committed up to 1
0709 13:02:27.909296 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:27.909418 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:28.491744 (+582326us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:28.551845 (+ 60101us) raft_consensus.cc:1685] finished
0709 13:02:28.551873 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:28.552115 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54007680,"thread_start_us":366267,"threads_started":1,"wal-append.queue_time_us":421644}
I20260709 13:02:28.554952  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.555024  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.558980  1739 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:28.503643  4962 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.562951  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.563895  1733 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:28.544894  2309 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:28.566792  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.570533  1739 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.567716  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:28.542136  4979 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:28.573436  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:28.573675  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 170759us
I20260709 13:02:28.560818  4736 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:28.581570  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.581861  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.579370  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.584865  2317 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:28.585328  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.567657  4962 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.582257  1738 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 4.
I20260709 13:02:28.586102  1736 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:28.585331  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.577171  4847 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.599319  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.595245  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:28.593240  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103810us
I20260709 13:02:28.582672  4902 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:28.593670  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.611105  1556 leader_election.cc:400] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:28.611584  1556 leader_election.cc:403] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.586702  4839 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.631804  4943 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:28.633342  2314 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:28.614501  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.615075  4968 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.622184  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:28.622416  4902 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:28.623019  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:28.610286  4956 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.593000  4976 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:28.655290  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.593323  4979 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:28.612042  4960 raft_consensus.cc:2764] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:02:28.624257  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.622225  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:28.610054  4736 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:28.633895  1475 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica ff44a4643e01482cb6b6a79b099fa766 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:28.665282  4736 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.665889  4902 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.666208  4979 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.625975  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:28.669399  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:28.627310  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:28.628594  4487 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.630s	user 0.010s	sys 0.003s
I20260709 13:02:28.629285  1740 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:28.629930  4965 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:28.586339  4515 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.587246  4883 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.687621  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.612252  4983 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:28.638880  4974 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:28.639254  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.692237  1734 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:28.640735  4943 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.595021  4951 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:28.658999  1739 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:28.656149  4847 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:28.663941  4980 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:28.670665  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.672297  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.654582  4834 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:28.677093  1733 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:28.670706  1592 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:28.685489  1741 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:28.688529  1742 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:28.689275  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62153.1us
W20260709 13:02:28.689917  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:28.660660  4947 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:28.686518  4939 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.681s	user 0.012s	sys 0.001s
I20260709 13:02:28.718118  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:28.718741  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46527.7us
I20260709 13:02:28.697216  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.721163  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.722446  4976 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.727737  4839 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:28.733969  4984 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.734506  4965 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:28.717900  4947 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.734545  4972 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.696797  4951 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.737380  1739 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:28.676146  4487 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.738073  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.738297  1738 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:28.738886  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.739847  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.717846  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:28.714233  4960 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:28.731696  4954 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.745365  4906 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:28.834106  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:28.834894  1661 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:28.746501  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.758023  1736 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:28.771569  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 265476us
I20260709 13:02:28.836338  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:28.775075  1742 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:28.772584  4939 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:28.776690  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.840282  1705 leader_election.cc:400] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:28.840619  1733 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.840761  1705 leader_election.cc:403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.776508  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:28.778267  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:28.790769  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:28.843421  1660 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 13:02:28.844107  1664 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 5: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:28.820540  4951 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.785275  1735 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:28.824855  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.830475  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.838374  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.856963  1740 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.821794  1665 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 13:02:28.780555  4947 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.876531  1736 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:28.878121  1556 leader_election.cc:400] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:28.878562  1556 leader_election.cc:403] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 13:02:28.882064  4666 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:28.881937  4487 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:28.884972  1539 auto_rebalancer.cc:893] Removing replace marker failed: Replica ff44a4643e01482cb6b6a79b099fa766 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:02:28.885576  1539 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 13:02:28.885927  1539 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 13:02:28.887390  4666 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:28.893599  4983 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:28.894277  4983 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.834510  4906 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:28.850494  4984 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:28.900583  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 154202us
W20260709 13:02:28.896052  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58621.2us
I20260709 13:02:28.840526  4839 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:28.842815  4965 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:28.905467  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:28.923314  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202268us
I20260709 13:02:28.924721  1733 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:28.935961  1740 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:28.939296  4960 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:28.939015  4839 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.940475  4968 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:28.840327  4972 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.946887  4954 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:28.877920  4969 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.951085  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:28.952646  4962 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.895280  4983 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.900084  4812 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:28.995910  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.954906  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.955909  2315 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:28.957868  4965 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:28.907472  4984 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:28.960260  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.964244  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:29.007891  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47720.5us
I20260709 13:02:28.988108  4998 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.008503  1738 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:28.983609  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:29.009627  1662 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:29.009490  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:28.989933  1736 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:28.992713  1665 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:28.992579  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:29.011265  1661 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:28.993283  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:29.003443  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:28.979421  4640 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:28.987592  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:29.013434  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:29.016999  1557 leader_election.cc:336] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.574s (SENT)
I20260709 13:02:29.026459  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.027854  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.017207  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.012974  4728 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:29.040261  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.042065  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:29.042456  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87637.9us
I20260709 13:02:29.042869  1739 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:29.009636  4984 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:29.041991  4728 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.051422  4728 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:29.052161  4972 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.053725  4666 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:29.058705  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:29.059424  1660 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.066862  4962 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.068657  4969 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:28.967718  4906 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.013996  4960 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:29.083261  1736 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:29.013336  4883 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:29.041594  4826 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.089654  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.088642  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.094003  1742 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:29.013773  4998 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.100375  4800 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.104017  4941 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.108647  4951 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.110433  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.115638  1733 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
I20260709 13:02:29.115829  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.118194  4325 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:29.116844  4969 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.119050  4967 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:29.119428  4969 raft_consensus.cc:2764] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:29.120188  4969 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:29.120285  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.120573  4969 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:29.120931  1741 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.121915  4968 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:29.122293  4968 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:29.123196  4968 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.124632  4968 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:29.094529  1736 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:29.128626  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111679us
I20260709 13:02:29.113421  1738 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:29.098124  4883 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.131054  4839 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.131589  4906 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.135677  4600 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.139115  4974 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:29.140329  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.121413  4969 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.141291  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:29.128008  5005 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.058s	user 0.003s	sys 0.000s
I20260709 13:02:29.153920  1660 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 13:02:29.155380  1736 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 13:02:29.166226  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47449us
I20260709 13:02:29.170426  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:29.192992  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.188436  1741 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:29.210170  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.217100  1740 raft_consensus.cc:2410] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:29.155499  4812 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:29.218928  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74700us
I20260709 13:02:29.219514  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.219969  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.222548  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.205827  4969 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:29.143009  4947 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.174557  4728 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.174932  4974 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:29.236682  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:29.222636  4600 raft_consensus.cc:2764] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 13:02:29.260180  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.218621  4962 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 13:02:29.262473  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:29.222378  4798 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.276s	user 0.027s	sys 0.037s
I20260709 13:02:29.231364  4839 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:29.230623  4953 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.223496  4971 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:02:29.265763  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.294231  4325 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.295022  4325 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:29.296557  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.264043  5010 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.298012  5010 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.298421  5010 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.299371  4325 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:29.299865  4325 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:29.300302  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:29.300843  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.269665  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.222687  4812 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.272195  4728 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.275705  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:29.276696  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.277704  4286 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:29.283279  4984 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.285612  1667 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:29.292035  4966 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 13:02:29.297156  4937 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
W20260709 13:02:29.237499  4876 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.098s	user 0.007s	sys 0.030s
I20260709 13:02:29.307510  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:29.267928  4798 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.321s	user 0.027s	sys 0.037s
I20260709 13:02:29.264513  4960 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.317816  4984 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:29.309130  4286 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:29.319761  4286 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:29.322970  1587 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:29.324360  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.325222  1591 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:29.326359  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.327208  1590 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:29.327376  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.270135  4972 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:29.339183  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.340157  1587 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:29.342191  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:29.315965  4876 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.187s	user 0.008s	sys 0.030s
I20260709 13:02:29.347049  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.347747  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.348250  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.349726  4737 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:29.270416  4980 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.068s
I20260709 13:02:29.338135  4960 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.312055  4971 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.354291  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.355453  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.315464  4839 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.366325  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.311847  4640 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:02:29.315056  4937 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:29.368738  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.369608  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.395735  1590 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:29.371181  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.371835  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.374747  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.376597  1586 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:29.377192  1584 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:29.335363  4798 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.338546  4953 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:29.402900  1591 raft_consensus.cc:2368] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:29.353747  4808 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 13:02:29.376121  4954 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:29.418097  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.419130  1590 raft_consensus.cc:2368] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
W20260709 13:02:29.420372  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54465.1us
I20260709 13:02:29.372005  4965 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.376389  4971 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.355100  4972 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.427793  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.428903  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.431525  1588 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:29.433923  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.435104  1584 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:29.435565  1593 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:29.436342  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 718606us
I20260709 13:02:29.436741  1590 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:29.438417  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.438453  1585 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:29.440615  1587 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:29.326663  4974 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.442011  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.411074  4728 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.446676  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.418673  4954 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:29.458541  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52650.3us
I20260709 13:02:29.460801  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.474361  1736 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:29.480988  4972 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:29.482767  4985 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.856s	user 0.004s	sys 0.011s
I20260709 13:02:29.433562  4971 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.443531  1587 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:02:29.486171  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.493417  1584 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.491482  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.495604  1588 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:29.467655  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:29.390810  4960 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:29.501945  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69447.5us
I20260709 13:02:29.501998  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.502445  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.503084  1587 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:29.498971  1586 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:29.442088  4826 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.500356  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.500970  1592 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 13:02:29.524106  1585 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:02:29.456710  4939 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:29.507318  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.445771  4951 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.536013  1588 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 13:02:29.515383  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.487109  4954 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:29.519472  1591 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:29.504447  4974 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:29.522962  4965 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.533607  1584 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:29.537650  1587 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:29.550490  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:29.548736  4286 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.552924  4286 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:29.558821  5013 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:02:29.562467  4819 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:29.563128  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59835.4us
I20260709 13:02:29.563213  4819 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.565043  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:29.565954  2314 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 5: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:29.539830  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.542376  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.536369  4737 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.522228  1736 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:29.514948  4941 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.514405  4962 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:29.590834  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.562606  4812 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.567598  1593 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:29.551285  5016 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 13:02:29.529587  5021 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 13:02:29.605016  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63484.2us
I20260709 13:02:29.551267  2318 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:29.521109  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.606683  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.606909  1586 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:29.583637  1588 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:29.609706  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:29.548559  4839 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.584023  1585 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:29.588223  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.589825  2317 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:29.516033  4728 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260709 13:02:29.422936  4325 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.517549  4947 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:29.617949  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 196783us
I20260709 13:02:29.603744  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.589293  4953 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.523761  4972 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 13:02:29.549924  4939 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:29.619273  1589 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:29.485731  4937 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.613354  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.626091  4982 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:02:29.629738  4781 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.590649  4962 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.630394  4819 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.631309  4962 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.631536  4737 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.632627  4962 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:29.633726  5026 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:29.592451  4666 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.647578  1587 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:29.651553  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 728) took 2000 ms. Trace:
I20260709 13:02:29.652053  1585 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:29.604867  5021 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.652094  2313 rpcz_store.cc:276] 0709 13:02:27.651249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:27.654109 (+  2860us) service_pool.cc:224] Handling call
0709 13:02:27.659798 (+  5689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:28.481600 (+821802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:29.651261 (+1169661us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b54006e0fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:29.651477 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1888761088}
I20260709 13:02:29.538527  4798 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.593151  4966 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.047s
W20260709 13:02:29.660351  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94962.7us
I20260709 13:02:29.660976  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:29.592730  4941 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.664950  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.634773  2309 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:29.678278  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:29.682264  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63585.4us
I20260709 13:02:29.682898  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.681130  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:29.662124  4685 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 13:02:29.694867  4883 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.705142  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:29.705659  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:29.668442  5022 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.661451  5010 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.660046  4286 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 13:02:29.708348  4878 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:29.681793  4972 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:29.719400  4878 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:29.719702  5017 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 13:02:29.724332  1734 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:29.709306  4937 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:29.712451  4906 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.514s	user 0.012s	sys 0.000s
W20260709 13:02:29.729691  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68813.2us
I20260709 13:02:29.679407  4953 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.735549  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:29.736178  5030 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 13:02:29.736831  4808 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.697186  5021 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.739238  5023 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:29.716585  4939 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.681830  4910 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 13:02:29.755896  4966 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.756575  4966 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.758841  4966 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.759351  4966 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.759773  4966 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.761729  5023 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:29.754228  4808 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.737746  4972 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.781020  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:29.775409  4968 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.650s	user 0.011s	sys 0.004s
I20260709 13:02:29.794359  5022 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 13:02:29.790798  4966 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.804194  5029 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:29.804911  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:02:29.807101  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.766422  4960 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.808561  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.809957  1667 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:29.818291  1742 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:29.817935  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.814688  1741 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 13:02:29.813898  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.815790  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.820223  1737 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.817391  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:29.811314  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:29.819466  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:29.820463  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.812561  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.822898  1739 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:29.820530  1738 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:29.826970  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:29.828902  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:29.792088  5010 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.811671  5030 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:29.830377  4941 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.791823  4980 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:02:29.833055  5027 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.818580  5029 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.833902  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.835003  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.835683  5029 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.836254  5029 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.836126  4954 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.836944  5029 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:29.836550  1738 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:29.838681  4972 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:29.839391  4954 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:29.840934  5032 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:29.841341  5032 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:29.841768  4883 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 13:02:29.841951  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 164662us
I20260709 13:02:29.843334  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.844215  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.844551  1708 leader_election.cc:400] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 4 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5.
I20260709 13:02:29.847388  5019 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.845435  1708 leader_election.cc:400] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:29.852516  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:29.855856  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62007.4us
I20260709 13:02:29.856055  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.774071  5023 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.884801  4951 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.885761  4951 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.876377  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:29.887713  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 633) took 1287 ms. Trace:
I20260709 13:02:29.878942  2281 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.888172  2315 rpcz_store.cc:276] 0709 13:02:28.587476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:28.592644 (+  5168us) service_pool.cc:224] Handling call
0709 13:02:29.874959 (+1282315us) spinlock_profiling.cc:243] Waited 237 ms on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:29.875126 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":545367296}
I20260709 13:02:29.816685  4998 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:29.829938  4878 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:29.900942  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52222.5us
W20260709 13:02:29.901394  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48950.2us
I20260709 13:02:29.906139  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.900264  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:29.903648  5030 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:29.904181  4951 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:29.904862  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50499.7us
I20260709 13:02:29.918285  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.918989  1737 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:29.919099  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:29.921623  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:29.895184  4969 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:29.851022  4910 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 13:02:29.898564  4938 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:29.895387  4941 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:02:29.930138  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:29.938532  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:29.930140  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:29.908996  4983 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s	user 0.015s	sys 0.029s
W20260709 13:02:29.939836  4983 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.015s	sys 0.029s
I20260709 13:02:29.936383  4728 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.940690  4728 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.941097  4728 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.943272  5018 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:29.878671  4883 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:02:29.946693  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:29.947738  4666 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.948159  1584 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:29.948828  1590 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:29.949381  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.929617  5012 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.950016  1584 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:29.951691  4960 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:29.952323  4960 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:29.951910  4971 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:29.952832  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:29.929962  4969 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.954085  4969 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:29.955612  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49566.2us
I20260709 13:02:29.842182  5032 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:29.934881  4878 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:29.972059  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:29.953619  1588 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.951849  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.959203  1589 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 13:02:29.974459  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:29.958459  1666 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:29.894275  4286 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 13:02:29.978662  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:29.976286  1737 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:29.981829  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:29.984032  1593 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:29.898232  5010 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:02:29.988245  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:29.999140  1584 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:30.004594  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.007476  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.010408  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.012246  1737 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260709 13:02:29.948184  4666 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.950024  5012 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.012797  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.015928  4998 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.017554  4876 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:29.926586  4906 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 13:02:29.977953  4878 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:29.938388  5041 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:30.024369  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:29.990960  5027 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:29.937604  4951 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.075s
I20260709 13:02:30.005636  5018 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:29.983965  4941 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:30.013443  4974 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:30.065461  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1661) took 1060 ms. Trace:
I20260709 13:02:30.067057  1733 rpcz_store.cc:276] 0709 13:02:29.005045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.013524 (+  8479us) service_pool.cc:224] Handling call
0709 13:02:30.065245 (+1051721us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:30.065401 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171385216}
I20260709 13:02:30.034453  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.049453  5042 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 13:02:30.050158  5030 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.024457  4666 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:29.953491  4960 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.053068  4876 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:30.147418  4876 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:30.053737  4906 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:30.149720  4906 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:30.150812  4906 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.055311  4971 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.036182  4839 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:29.980870  4883 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:02:30.070422  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.156440  4839 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.070565  1587 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:30.156924  4839 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:30.157080  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86773.5us
I20260709 13:02:30.157996  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.036126  4734 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.160668  4734 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.162147  4966 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:30.162607  4966 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.163000  4966 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:30.163125  4734 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.163839  4734 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:29.937381  4808 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.062856  5040 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 13:02:30.164847  5040 consensus_queue.cc:1243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:30.027222  5012 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.166620  4941 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.169129  4734 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.071961  1588 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:30.072605  5038 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260709 13:02:30.087774  1733 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:30.088281  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:30.172212  2318 raft_consensus.cc:606] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:30.172046  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.047508  4947 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.066603  5027 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.048117  4878 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.091470  4969 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:30.095561  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.097515  1736 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:30.099546  4965 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:30.106091  1557 leader_election.cc:400] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 7. Current term is 8.
W20260709 13:02:30.182250  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76385.7us
W20260709 13:02:30.128896  5037 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.363s	user 0.003s	sys 0.000s
I20260709 13:02:30.134750  5021 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:30.065447  5041 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.078320  4953 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:30.193804  1660 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:30.192691  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.148711  4876 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.177714  4951 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.216233  1663 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:30.184393  4965 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:30.227787  1737 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:30.228300  1737 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:30.198931  4953 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.229180  1737 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.241922  1737 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:30.243615  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 414794us
I20260709 13:02:30.184026  4947 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.228631  5023 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.263691  1733 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:30.237291  4876 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.231858  4967 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.232540  4963 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:02:30.242770  4953 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.261991  4781 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:30.229171  4951 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.282641  1737 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:02:30.216120  4734 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.291661  1737 pending_rounds.cc:85] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 8
I20260709 13:02:30.292948  4883 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:30.294478  5047 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 13:02:30.295692  5039 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260709 13:02:30.298354  4972 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:30.321858  4972 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.322826  4972 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.324309  4972 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:30.299347  4970 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:30.326309  4970 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:30.327256  4970 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.299995  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1656) took 1298 ms. Trace:
I20260709 13:02:30.328248  1740 rpcz_store.cc:276] 0709 13:02:29.001471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.002458 (+   987us) service_pool.cc:224] Handling call
0709 13:02:30.299785 (+1297327us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:30.299926 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407622912}
I20260709 13:02:30.310408  1737 pending_rounds.cc:107] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:02:30.308580  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:30.331550  2318 raft_consensus.cc:606] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:30.297385  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.315235  4942 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:30.315434  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.283079  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.339664  1733 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:30.320792  4666 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.325762  4969 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:30.332450  1741 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.390100   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:02:30.290831  4965 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.394155  4965 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:30.398118  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.399194  1733 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:30.346880  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.365561  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.366354  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.371294  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.403847  1664 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:30.403949  1738 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:30.373664  4666 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.374711  1668 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:30.291062  4977 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.221s
I20260709 13:02:30.406786  1733 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:30.377948  5027 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.334498  4808 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.360008  4728 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.379150  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.315428  4883 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.380772  1660 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:30.382040  4942 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.279490  4967 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.389906  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.308984  5041 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.355163  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.332759  1737 raft_consensus.cc:2889] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:30.433247  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78267.4us
I20260709 13:02:30.434098  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:30.438591  4968 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:30.425843  4808 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.439607  4728 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.439481  4968 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.440538  1585 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:02:30.441180  4883 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.441709  5027 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:30.421074  4966 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.442436  5027 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:30.442936  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:30.443886  4966 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.445897  1665 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:02:30.446141  4942 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.441129  4808 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.446951  4968 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:30.446293  1665 pending_rounds.cc:85] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 3
I20260709 13:02:30.468472  1665 pending_rounds.cc:107] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:02:30.468923  1665 raft_consensus.cc:2889] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:30.468716  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.447525  1666 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:30.443964  5023 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:30.450529  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2229) took 1129 ms. Trace:
I20260709 13:02:30.472538  1592 rpcz_store.cc:276] 0709 13:02:29.321088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.338024 (+ 16936us) service_pool.cc:224] Handling call
0709 13:02:29.369770 (+ 31746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:29.524523 (+154753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.450255 (+925732us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540025dd54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:30.450460 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284970752}
I20260709 13:02:30.450616  4839 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.453236  5058 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:30.418417  4325 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 13:02:30.453948  4808 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:30.429983  4951 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.423290  1741 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:30.485608  5058 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 13:02:30.497457  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.458403  4947 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.507671  1664 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:30.472306  5023 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:30.510636  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:30.515136  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 342407us
I20260709 13:02:30.512276  4808 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.521231  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.482421  4883 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 13:02:30.525348  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.485873  5012 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.520325  4800 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:30.516434  5023 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:30.512776  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:30.524641  4808 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.540122  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.520284  4968 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:30.543509  1662 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:30.547181  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:30.547325  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.548844  1585 raft_consensus.cc:2368] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8.
I20260709 13:02:30.521116  4966 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.527920  4947 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.546972  4728 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.558050  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.581776  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.547114  4951 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:30.584672  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.590811  1665 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:30.586740  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.594946  1663 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.587787  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.576897  4640 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:30.589727  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.575666  4728 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.567585  4968 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:30.542843  4878 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.653720  5066 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.578265  5019 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 13:02:30.652742  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.652873  1586 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:02:30.578454  5020 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.589133  4666 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.661834  2310 raft_consensus.cc:2368] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:30.646615  1667 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 13:02:30.663405  4808 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.662920  4728 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.661126  4878 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.670534  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:30.659030  4734 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:30.565828  4966 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.678704  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.689530  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.667009  5012 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.659793  5026 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:30.685019  4951 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.651085  4947 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:30.691893  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.645084  4953 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.670934  4972 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 13:02:30.716514  1588 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 13:02:30.656651  4640 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.687381  4781 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.720404  1634 leader_election.cc:400] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8.
W20260709 13:02:30.694553  5065 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:02:30.732690  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.733489  1660 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:30.721208  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.660597  5066 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.702271  1584 pending_rounds.cc:85] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 2
I20260709 13:02:30.723505  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.724012  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.671309  4971 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 13:02:30.698248  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.732800  1634 leader_election.cc:403] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:30.720822  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.744467  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:30.725256  5021 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.000s
I20260709 13:02:30.735201  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.746260  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:30.746604  1584 pending_rounds.cc:107] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 13:02:30.747174  1584 raft_consensus.cc:2889] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:30.747820  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.681318  5020 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.748533  1588 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:30.749843  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:30.753758  1587 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:30.751631  1667 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 13:02:30.754424  1667 pending_rounds.cc:85] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:02:30.719444  4839 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.759016  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.755810  1667 pending_rounds.cc:107] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 13:02:30.755285  1589 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:30.753880  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.726243  4734 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.761782  1587 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:30.731833  4640 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:30.760357  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.760331  1667 raft_consensus.cc:2889] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:30.758836  1590 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:30.763989  1587 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:30.712337  4951 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.718326  5026 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.751359  5012 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.784890  1589 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:30.786968  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:30.787612  1587 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:30.752894  4967 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.775034  4968 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.788834   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:30.795065  4951 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.733191  4947 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.748751  4781 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.797020  5020 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:30.795097  1489 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 127) took 21858 ms (client timeout 15000 ms). Trace:
I20260709 13:02:30.798108  1586 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 13:02:30.798846  4781 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.768577  4808 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 13:02:30.771575  5030 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:30.804615  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:30.811985  1589 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:30.830547  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.805917  4963 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.835717  4963 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.825793  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.841725  4967 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.841985  5066 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.842824  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.843693  1667 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:30.845014  5072 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 13:02:30.820669  5030 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:30.845813  5030 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.848809  5021 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:30.788889  4972 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:02:30.849980  4734 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.851063  5027 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:30.851359  4966 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.800194  4808 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:30.853193  4808 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.854405  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.854175  4808 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.803045  5067 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260709 13:02:30.855659  4808 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:30.854583  4953 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.857461  4876 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:30.858286  4876 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:30.858486  5012 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:30.859836  4876 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:30.798887  4947 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.860198  4876 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:30.861888  5026 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.837747  4781 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:30.863018  4781 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:30.863415  4781 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.864436  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 478) took 1432 ms. Trace:
I20260709 13:02:30.864703  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.865594  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.867043  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.867892  1665 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:30.864913  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:30.869199  1660 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 4.
I20260709 13:02:30.869699  1667 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:30.870311  5040 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:02:30.797973  1489 rpcz_store.cc:269] 0709 13:02:08.936146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.936359 (+   213us) service_pool.cc:224] Handling call
0709 13:02:30.794826 (+21858467us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:24.397791 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:24.406830 (+  9039us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:24.406895 (+    65us) write_op.cc:435] Acquired schema lock
0709 13:02:24.406920 (+    25us) tablet.cc:662] Decoding operations
0709 13:02:24.630510 (+223590us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:24.630713 (+   203us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:24.630745 (+    32us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:02:24.632886 (+  2141us) tablet.cc:701] Row locks acquired
0709 13:02:24.632959 (+    73us) write_op.cc:260] PREPARE: finished
0709 13:02:24.633165 (+   206us) write_op.cc:270] Start()
0709 13:02:24.633342 (+   177us) write_op.cc:276] Timestamp: P: 1783602144633086 usec, L: 0
0709 13:02:24.633366 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:02:24.656156 (+ 22790us) log.cc:844] Serialized 6021 byte log entry
0709 13:02:25.996436 (+1340280us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:27.080235 (+1083799us) op_driver.cc:464] REPLICATION: finished
0709 13:02:27.214055 (+133820us) write_op.cc:301] APPLY: starting
0709 13:02:27.214189 (+   134us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:27.214605 (+   416us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:27.214640 (+    35us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:27.317444 (+102804us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:27.344413 (+ 26969us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=15
0709 13:02:27.344499 (+    86us) write_op.cc:312] APPLY: finished
0709 13:02:27.345323 (+   824us) log.cc:844] Serialized 107 byte log entry
0709 13:02:27.347465 (+  2142us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:27.347963 (+   498us) write_op.cc:454] Released schema lock
0709 13:02:27.348085 (+   122us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:27.348746 (+   661us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:27.348786 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2733718,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":133401,"apply.run_cpu_time_us":7701,"apply.run_wall_time_us":134647,"num_ops":15,"prepare.queue_time_us":228433,"prepare.run_cpu_time_us":103769,"prepare.run_wall_time_us":631887,"raft.queue_time_us":359977,"raft.run_cpu_time_us":20938,"raft.run_wall_time_us":109713,"replication_time_us":2446524,"spinlock_wait_cycles":384774912,"thread_start_us":712403,"threads_started":4,"wal-append.queue_time_us":614070,"wal-append.run_cpu_time_us":101313,"wal-append.run_wall_time_us":3069127}]]}
I20260709 13:02:30.870745  5040 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:30.871744  5040 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.874549  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1665) took 1066 ms. Trace:
I20260709 13:02:30.875041  1736 rpcz_store.cc:276] 0709 13:02:29.808090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.811014 (+  2924us) service_pool.cc:224] Handling call
0709 13:02:30.874327 (+1063313us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b540012e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:30.874470 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538534144}
I20260709 13:02:30.842901  1589 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 13:02:30.879374  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.880052  1742 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.864854  1735 rpcz_store.cc:276] 0709 13:02:29.432227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.432978 (+   751us) service_pool.cc:224] Handling call
0709 13:02:30.302845 (+869867us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.303083 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:30.303389 (+   306us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.491388 (+187999us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:30.491479 (+    91us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:30.491547 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.491667 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:30.864231 (+372564us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b5400202bd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:30.864347 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12743,"prepare.queue_time_us":746,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"raft.queue_time_us":128,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":21856,"spinlock_wait_cycles":1999818112,"thread_start_us":187560,"threads_started":1,"wal-append.queue_time_us":270154}
I20260709 13:02:30.881891  4939 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:30.833317  4965 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:30.882403  4939 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:30.882558  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.882771  4965 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.882962  4781 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.883275  1741 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:30.885318  4965 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:30.886114  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.883447  4939 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:30.888206  4781 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.888437  4939 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:30.890820  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.891613  1736 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:30.892664  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.894168  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:30.893929  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.895046  1588 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:30.897866  1738 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:02:30.898882  5028 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.214s	user 0.000s	sys 0.005s
I20260709 13:02:30.900861  1738 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:30.906183  1741 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:30.913774  1666 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:30.917030  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.918589  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:30.924264  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67601.1us
I20260709 13:02:30.927733  1738 pending_rounds.cc:85] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 2
I20260709 13:02:30.928306  1738 pending_rounds.cc:107] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 13:02:30.928685  1738 raft_consensus.cc:2889] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:30.929999  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:30.840863  5065 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:02:30.946605  1742 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:30.931262  1663 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:30.933833  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.873355  5040 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:30.952042  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.953006  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:30.957875  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:30.958176  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:30.965042  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:30.971324  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.974819  4977 raft_consensus.cc:993] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:02:30.974992  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2261) took 1151 ms. Trace:
I20260709 13:02:30.975490  4977 raft_consensus.cc:1081] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:02:30.980113  4963 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:30.972643  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.982790  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:30.975419  1739 rpcz_store.cc:276] 0709 13:02:29.823475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.823592 (+   117us) service_pool.cc:224] Handling call
0709 13:02:29.824006 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:29.900659 (+ 76653us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:30.160719 (+260060us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b18002bd620. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:02:30.270944 (+110225us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.515941 (+244997us) log.cc:844] Serialized 14 byte log entry
0709 13:02:30.517167 (+  1226us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:30.518986 (+  1819us) log.cc:844] Serialized 26 byte log entry
0709 13:02:30.519171 (+   185us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:30.519214 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:30.519298 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:30.974747 (+455449us) raft_consensus.cc:1685] finished
0709 13:02:30.974772 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:30.974921 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86667,"prepare.queue_time_us":211517,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":71,"raft.queue_time_us":83926,"raft.run_cpu_time_us":543,"raft.run_wall_time_us":37288,"spinlock_wait_cycles":19917056,"thread_start_us":319230,"threads_started":2,"wal-append.queue_time_us":246011}
W20260709 13:02:30.984234  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50545.3us
I20260709 13:02:30.984786  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.932749  4781 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:30.986356  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:30.987596  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:30.992652  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:30.992571  1666 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:30.950582  4974 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 13:02:30.996335  5020 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:30.989393  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:31.074105  1742 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:30.996795  5054 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:30.970938  4600 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8.
I20260709 13:02:30.993887  4883 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 13:02:31.011281  4781 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.009999  5023 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:30.965983  4876 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.039858  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:31.007004  5074 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.264s	user 0.001s	sys 0.000s
I20260709 13:02:31.009629  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 482) took 1180 ms. Trace:
I20260709 13:02:30.950059  4812 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
I20260709 13:02:30.950425  4967 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 13:02:30.991894  4947 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:30.994179  1592 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:02:31.011659  4515 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:30.991514  4325 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:31.087769  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.112316  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 310098us
I20260709 13:02:30.997426  4640 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.120679  1593 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:31.126987  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.128029  1737 rpcz_store.cc:276] 0709 13:02:29.829006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.014308 (+185302us) service_pool.cc:224] Handling call
0709 13:02:30.014688 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.433036 (+418348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.009202 (+576166us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54002030d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:31.009482 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489757952}
I20260709 13:02:31.121594  1734 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
W20260709 13:02:31.117244  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.099418  5020 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.101768  4883 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.102468  4952 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:31.143815  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.162926  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.163661  1588 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:31.130586  4808 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.172505  1593 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:31.173869  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:31.174608  1593 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:31.189798  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 197248us
I20260709 13:02:31.145673  4325 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.197925  4953 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:31.202574  4781 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.144636  4876 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:31.203724  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.205302  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:31.206064  1592 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:31.161414  4883 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:31.197948  4325 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.158783  5020 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:31.217835  4967 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.218127  5030 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.218427  4967 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.220325  5030 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.220603  4967 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.221094  1668 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:02:31.221220  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.217011  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.222103  1660 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:31.222831  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:31.223798  5095 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:31.172189  4952 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.226346  5023 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.227954  4952 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:31.229655  4808 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.231750  4808 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.161648  4947 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.234187  1589 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 13:02:31.234483  4947 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:31.236123  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.224532  1584 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:31.239908  1592 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:31.241402  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:31.242609  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.246183  1667 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:31.142997  4515 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:31.253521  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.254340  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:31.254818  1592 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 13:02:31.252630  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:31.251734  5066 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:31.271903  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.252328  4947 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:31.275969  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:31.278568  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 398215us
I20260709 13:02:31.261341  4839 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:31.263794  5067 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:31.284145  1584 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:31.259009  4965 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 13:02:31.275328  5058 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:31.248220  4325 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.336470  1667 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:31.311733  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:31.338388  1584 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:31.305996  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:31.336341  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 287648us
I20260709 13:02:31.283818  4781 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:31.348243  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65037.4us
I20260709 13:02:31.287096  4839 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:31.323048  4666 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:31.281539  5066 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.326200  4515 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:31.357751  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51906.6us
W20260709 13:02:31.500016  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.662719  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.551345  1584 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:31.352262  4640 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.334028  4951 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:31.671100  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.691447  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:31.399823  4939 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.003s
W20260709 13:02:31.706462  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.710688  1593 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.002s
I20260709 13:02:31.718807  1593 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:31.624310  4956 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:31.389381  4737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.016s	sys 0.033s
I20260709 13:02:31.659878  4826 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.725596  5021 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.726394  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:31.660351  4839 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.423105  4876 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:31.388108  5027 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:31.495744  5040 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.329768  5067 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.337136  4883 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.341399  5058 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:31.668613  5042 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:31.756063  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 401225us
W20260709 13:02:31.757085  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 66018.6us
I20260709 13:02:31.782334  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:31.428005  4325 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:31.772328  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46044.2us
I20260709 13:02:31.416416  5020 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:31.752887  4737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.028s	sys 0.057s
I20260709 13:02:31.752699  4966 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.372577  4781 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.726949  4947 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:31.670596  4967 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.102s
I20260709 13:02:31.738692  5030 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.785208  4968 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.797617  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:31.799124  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.740244  4878 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:31.689358  4666 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.771972  5027 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:31.808357  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:31.833178  5027 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:31.761755  4826 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.840646  4826 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:31.842367  5020 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.843138  4781 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.843750  4878 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.844192  4781 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.806035  4966 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.847541  5117 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.848002  4966 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.848219  5117 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.728020  4951 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.831193   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:31.832294  4968 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.771888  5021 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.795765  4640 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:02:31.877977  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95436.2us
I20260709 13:02:31.762250  4939 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 13:02:31.843246  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.754483  4956 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:31.864916  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:31.885456  1706 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.536s (SENT) [suppressed 14 similar messages]
I20260709 13:02:31.776502  4876 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.867587  4878 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.897755  5110 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:31.898378  5110 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:31.863684  5020 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.899096  5020 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.899484  5110 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.853813  5118 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260709 13:02:31.903241  5106 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:31.903704  5106 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:31.903764  5118 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.904032  4906 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:31.904333  5118 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.860185  4826 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.906826  4826 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:31.908159  5104 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
W20260709 13:02:31.910354  5086 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:02:31.910633  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.911922  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.912513  1587 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:31.908512  4956 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.909142  5058 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.917878  4640 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:31.904181  5106 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:31.870800  5021 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:31.856371  5078 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:02:31.873270  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:31.921249  1659 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:31.922338  4972 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:31.922958  4972 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.923380  5067 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.924742  5066 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.925065  4972 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:31.925315  5114 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:31.714198  4812 consensus_queue.cc:1175] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ff44a4643e01482cb6b6a79b099fa766" responder_term: 6 status { last_received { term: 5 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:31.782547  5040 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.925941  5114 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.927878  5040 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:31.928814  5076 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 13:02:31.868450  4968 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:31.889922  4947 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:31.891641  5054 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:31.796316  5042 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.884163  4965 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:31.895589  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2288) took 1002 ms. Trace:
I20260709 13:02:31.851322  5030 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.844854  4781 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.949747  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:31.918846  4640 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:31.936110  4951 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:31.936344  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.935068  1584 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:31.928941  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:31.944988  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:31.937307  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:31.955061  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:31.896389  5099 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 13:02:31.888780  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91310.9us
I20260709 13:02:31.963728  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:31.951227  1736 rpcz_store.cc:276] 0709 13:02:30.893384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.942579 (+ 49195us) service_pool.cc:224] Handling call
0709 13:02:30.943063 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:30.983642 (+ 40579us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:31.287931 (+304289us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.786101 (+498170us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:31.786777 (+   676us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.786930 (+   153us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:31.786965 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.787039 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:31.895332 (+108293us) raft_consensus.cc:1685] finished
0709 13:02:31.895358 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:31.895518 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27612,"prepare.queue_time_us":243641,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":171477,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":12637,"thread_start_us":619455,"threads_started":3,"wal-append.queue_time_us":471528}
I20260709 13:02:31.909226  4826 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.004300  1587 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:31.995100  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:32.006891  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:32.172744  4878 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:02:32.061722  1666 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:02:32.223551  1586 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.011s	sys 0.005s
I20260709 13:02:32.002748  4965 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.226462  4910 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:32.227022  4910 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:32.013926  1667 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:32.229539  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2218) took 1365 ms. Trace:
I20260709 13:02:32.228379  5115 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.227993  4910 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:32.230861  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 342381us
I20260709 13:02:32.230814  5115 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.231585  4910 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:32.233110  5115 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.228969  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2286) took 1336 ms. Trace:
W20260709 13:02:32.228377  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.043257  1659 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:32.010766  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:32.234740  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.234756  1660 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:32.010393  2318 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:32.227932  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2277) took 1337 ms. Trace:
W20260709 13:02:32.224869  1586 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.012s	sys 0.005s
I20260709 13:02:32.237601  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.237941  1740 rpcz_store.cc:276] 0709 13:02:30.890248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.901648 (+ 11400us) service_pool.cc:224] Handling call
0709 13:02:30.901985 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.902235 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:31.262377 (+360142us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.525049 (+262672us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.901468 (+376419us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b18004a5900. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:02:31.901720 (+   252us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.227468 (+325748us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.227528 (+    60us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:32.227576 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.227650 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.227871 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":677059,"prepare.queue_time_us":726219,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":122,"raft.queue_time_us":433306,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":27273,"spinlock_wait_cycles":39102720,"thread_start_us":620492,"threads_started":2,"wal-append.queue_time_us":476445}
I20260709 13:02:32.237965  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:32.242163  1739 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:32.234133  1741 rpcz_store.cc:276] 0709 13:02:30.892859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.907190 (+ 14331us) service_pool.cc:224] Handling call
0709 13:02:30.907552 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.907818 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:31.256620 (+348802us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.266768 (+ 10148us) log.cc:844] Serialized 14 byte log entry
0709 13:02:31.918698 (+651930us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.228629 (+309931us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.228682 (+    53us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:32.228730 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.228798 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.228901 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":471488,"prepare.queue_time_us":491254,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":3346,"raft.queue_time_us":469611,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":21196,"thread_start_us":794694,"threads_started":2}
I20260709 13:02:32.017805  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:32.227231  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 369355us
I20260709 13:02:32.247072  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.248807  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:32.106439  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.826s	user 0.008s	sys 0.045s
W20260709 13:02:32.250025  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.009s	sys 0.051s
I20260709 13:02:32.230104  1668 rpcz_store.cc:276] 0709 13:02:30.863774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.867912 (+  4138us) service_pool.cc:224] Handling call
0709 13:02:30.868264 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.221466 (+353202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.229272 (+1007806us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:32.229467 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810908032}
W20260709 13:02:32.117146  1589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.020s	sys 0.035s
W20260709 13:02:32.253274  1589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.024s	sys 0.037s
W20260709 13:02:32.254029  1559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.006s	sys 0.045s
I20260709 13:02:32.240635  1738 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:02:32.255229  1559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.006s	sys 0.045s
I20260709 13:02:32.257688  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:32.220111  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.273384  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:32.275570  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:32.277005  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.279155  1741 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:32.280756  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 351896us
W20260709 13:02:32.282174  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.283404  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.269698  2316 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:31.951162  5054 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:32.285377  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.286491  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.278012  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:32.288919  1741 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:32.269155  1736 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:32.290369  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:32.284832  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.296479  1740 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 4. Current term is 5. Ops: []
I20260709 13:02:31.922343  4883 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.285364  4956 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.300452  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2290) took 1084 ms. Trace:
I20260709 13:02:31.938661  4876 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.297689  1739 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:32.226974  5116 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.219862  5117 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.220137  5118 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.064033  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:32.229382  5018 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:32.285172  5124 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:32.299464  5067 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:32.229244  4965 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:32.326514  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 313442us
I20260709 13:02:32.307230  1593 rpcz_store.cc:276] 0709 13:02:31.215925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.783538 (+567613us) service_pool.cc:224] Handling call
0709 13:02:31.783857 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.784059 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:31.804783 (+ 20724us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.300149 (+495366us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.300199 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:32.300242 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.300301 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.300398 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17882,"prepare.queue_time_us":30225,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":57,"raft.queue_time_us":97698,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":128,"thread_start_us":476512,"threads_started":1}
I20260709 13:02:32.295785  1738 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
I20260709 13:02:31.873266  5027 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.319408  4800 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:32.319911  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.402570  1734 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:32.298341  4487 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:32.298550  4972 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.294430  4968 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.287765  4839 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.294620  5066 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.351365  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.485149  1739 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 5: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:32.358812  1735 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:32.403712  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:32.404834  1738 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:02:32.360221  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:32.352350  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.351477  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:32.301146  5114 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.321183  4971 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:32.488678  1738 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:32.497956  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.489796  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.408187  4487 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.516139  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:32.314002  5112 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.324429  4967 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.320513  5101 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:32.482839  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 192607us
I20260709 13:02:32.409862  5030 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:32.435890  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202741us
I20260709 13:02:32.486299  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.486959  1661 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:32.490302  1740 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:32.485925  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.485546  4781 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.401829  5117 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.505143  4971 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.567001  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.568753  1738 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.482384  4972 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.525458  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:32.407872  5103 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 13:02:32.320016  5019 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.528805  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.529592  5125 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.332921  5122 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:32.497117  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.585693  1665 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:32.409655  4883 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.435055  5118 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:32.498799  5121 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.500963  5042 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.351411  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.595201  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:32.596325  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.506613  4876 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.521677  4487 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:32.539685  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:32.559577  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.562701  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.599927  1663 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:32.560770  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:32.328840  5116 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.320951  4956 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.559649  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2288) took 1344 ms. Trace:
I20260709 13:02:32.327268  5012 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.231s
I20260709 13:02:32.578524  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:32.609011  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.485981  4968 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.581018  1668 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:32.530220  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.611459  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.584197  1738 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.517663  5114 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 13:02:32.494995  5066 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:32.584663  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.494105  4969 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.559942  5021 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.566347  5105 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:32.529208  4965 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:32.563230  5113 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.646806  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.571734  4971 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:32.567103  4781 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.649322  4781 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.620354  4800 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:32.587882  1740 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:32.653789  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.655426  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.483210  5054 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.603390  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.604291  1589 rpcz_store.cc:276] 0709 13:02:31.215349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.235008 (+ 19659us) service_pool.cc:224] Handling call
0709 13:02:31.235289 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:31.235497 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:32.253689 (+1018192us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.559221 (+305532us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:32.559291 (+    70us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:32.559341 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.559410 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.559581 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63147,"prepare.queue_time_us":37051,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":70837,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":4382,"thread_start_us":1262720,"threads_started":2,"wal-append.queue_time_us":316374}
I20260709 13:02:32.660182  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.656468  4800 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.660386  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.661584  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.662231  1589 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:32.663272  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.663797  5054 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:32.663915  1589 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:32.666718  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.667755  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.669073  5058 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:32.669546  5058 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.669893  5058 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:32.647816  2317 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:32.672806  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.673748  4640 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:32.674017  2317 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:02:32.537726  4967 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.661954  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.676592  4967 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.613186  1667 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:02:32.652312  4781 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.677757  4781 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.678133  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.614472  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.678393  4781 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.571285  5117 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.594931  5118 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.620908  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.624564  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:32.583611  5019 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.628569  1660 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:32.629963  5142 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.633359  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.533197  5112 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.631479  4969 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.756067  5112 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.756630  5112 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:32.756947  5112 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:32.558506  5101 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.645112  4965 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.645195  5021 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.758544  4965 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:32.759050  5101 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.759629  5101 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:32.689141  5118 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:32.759941  5101 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.760372  5101 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:32.651368  5110 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.761790  5110 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.764017  5110 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.764727  5019 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.686566  5113 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.766263  1741 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:32.767714  5113 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.768074  1736 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:32.768266  5113 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.681932  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.685638  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.594472  4883 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:32.686576  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65766.7us
I20260709 13:02:32.770608  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.770829  5134 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.770933  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:32.771442  5134 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.683887  1587 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:32.687565  4640 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:32.773813  4640 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:32.775224  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.775369  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:32.776132  1587 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:32.626739  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.728935  4839 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.778309  5142 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.779008  5137 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.688418  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:32.779482  5137 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.780107  2313 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:32.782193  1738 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 3.
I20260709 13:02:32.705281  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:32.656872  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.586541  5125 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.601946  4876 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.623272  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:32.600939  5099 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:02:32.738400  5027 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:32.738749  5127 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:02:32.738976  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.799817  1735 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:32.800278  1733 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:32.740196  4968 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:32.794500  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 172443us
I20260709 13:02:32.744819  4972 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.745332  5130 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:32.745787  5133 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:32.746215  5135 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.602072  5121 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.747174  5136 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.748142  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2287) took 1533 ms. Trace:
W20260709 13:02:32.748661  4906 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.844s	user 0.007s	sys 0.005s
I20260709 13:02:32.748406  4487 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:32.749390  5132 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 13:02:32.750108  5131 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 13:02:32.751544  5116 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.751364  5138 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:32.752086  5066 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.752811  1664 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:32.825661  5134 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.685941  4325 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.753420  4666 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.886838  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2211) took 2511 ms. Trace:
I20260709 13:02:32.604108  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.777338  5030 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.784106  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.890179  1668 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:32.891172  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.785307  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.785970  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:32.786795  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.823674  4972 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 13:02:32.789290  5042 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:02:32.823415  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.870568  4906 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.830605  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.905901  1733 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 13:02:32.878110  4947 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.906817  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.908464  1733 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 13:02:32.908792  1733 pending_rounds.cc:85] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:02:32.906980  4947 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.768761  5113 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:32.811180  5106 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.806s	user 0.009s	sys 0.004s
I20260709 13:02:32.790748  4969 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.838640  1586 rpcz_store.cc:276] 0709 13:02:31.214961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.232692 (+ 17731us) service_pool.cc:224] Handling call
0709 13:02:31.233162 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:31.233598 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:31.233906 (+   308us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.270842 (+1036936us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.286605 (+ 15763us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:32.300145 (+ 13540us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.300286 (+   141us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:32.300319 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.300382 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:32.747916 (+447534us) raft_consensus.cc:1685] finished
0709 13:02:32.747937 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:32.748073 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58586,"prepare.queue_time_us":22013,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":94,"raft.queue_time_us":492750,"raft.run_cpu_time_us":356,"raft.run_wall_time_us":79653,"thread_start_us":1003599,"threads_started":2,"wal-append.queue_time_us":552015}
I20260709 13:02:32.848235  1587 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:32.802155  2309 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:32.849468  1737 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:02:32.920763  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.850540  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:32.805648  5012 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:32.890734  4325 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:32.874804  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92673.4us
W20260709 13:02:32.753144  4910 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.003s
W20260709 13:02:32.871169  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0255479380000452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260709 13:02:32.878116  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75132.3us
I20260709 13:02:32.883602  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.823201  4968 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.057s
W20260709 13:02:32.892796  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.803032  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.887256  1662 rpcz_store.cc:276] 0709 13:02:30.369794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.369950 (+   156us) service_pool.cc:224] Handling call
0709 13:02:30.370354 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.547409 (+177055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.881221 (+2333812us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b540030b454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:32.881424 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397682560}
I20260709 13:02:32.899602  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:32.899897  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:32.902976  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:32.957041  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57316.5us
I20260709 13:02:32.780519  5142 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.829046  5133 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.903127  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.959121  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.959407  5133 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.960089  1668 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:32.903141  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.960300  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.904529  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.905076  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.909039  1733 pending_rounds.cc:107] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 13:02:32.961647  1733 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:32.836186  1734 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:32.915077  4487 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.850963  5136 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.879590  5116 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:32.816450  5112 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.890245  4781 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.925171  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:32.987957  1587 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:32.874586  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.013s	sys 0.023s
I20260709 13:02:32.927649  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:32.988763  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.014s	sys 0.023s
I20260709 13:02:32.989116  1586 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:32.928201  5107 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:32.931007  5137 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.935340  5117 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.935678  5021 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:32.935822  4969 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:32.936736  4876 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.938449  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:32.938894  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88568.3us
I20260709 13:02:33.004609  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:32.961208  1663 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:32.965123  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.006251  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.007112  1659 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 9 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 13:02:32.960744  1740 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:32.960944  1662 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:33.008030  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69676.2us
I20260709 13:02:33.008677  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.009274  1662 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:32.964648  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.839052  5135 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.937369  4883 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.968104  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:32.901873  5030 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:32.971580  4910 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:32.972774  4325 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:32.973054  5118 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.967311  5142 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:32.978353  5125 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.919286  4906 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:32.979286  5113 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.980268  5136 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:32.949575  5099 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 13:02:32.938076  5012 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:32.991815  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:32.989192  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 887192us
I20260709 13:02:32.991672  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:33.037230  1587 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:32.925804  5106 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.006060  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.011752  1668 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:02:33.040876  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:32.960608  1660 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:33.016793  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.038134  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:33.021401  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:32.986156  5116 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.001139  5130 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:33.000000  5156 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.016546  5137 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:32.997862  5150 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:02:33.019575  4969 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:33.035913  1586 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:33.022603  4910 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.004781  5138 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 13:02:33.041539  1740 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:33.038075  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:33.061182  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:33.062779  1586 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:33.062311  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:33.000226  5107 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.070715  4970 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:33.072012  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:33.048545  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.039631  5012 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:33.051870  1589 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:33.050709  5130 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.053159  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:33.076795  1587 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:33.078598  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:33.086117  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:33.091992  1587 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:33.093158  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2291) took 1871 ms. Trace:
I20260709 13:02:33.093658  1587 rpcz_store.cc:276] 0709 13:02:31.221431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.091163 (+1869732us) service_pool.cc:224] Handling call
0709 13:02:33.091555 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.092826 (+  1271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.093086 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.054714  4969 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:33.098856  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62110.7us
I20260709 13:02:33.107730  1586 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:33.108248  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83473.7us
I20260709 13:02:33.109941  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:33.113350  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.077809  5106 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.139989  1739 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 13:02:33.055941  5042 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 13:02:33.111584  4969 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:33.152746  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57381.7us
I20260709 13:02:33.151152  5159 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:02:33.054839  5155 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260709 13:02:33.156173  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:33.143095  5137 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:33.156889  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 85017.6us
I20260709 13:02:33.151733  2309 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:33.098459  5058 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.166873  5121 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.058211  5156 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.150620  5099 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:33.192857  5164 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.024s
I20260709 13:02:33.164672  4947 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.163968  5159 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:33.168921  5137 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.151901  4640 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.176064  5099 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:33.219487  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 111690us
W20260709 13:02:33.223383  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 113557us
I20260709 13:02:33.229864  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:33.240693  5156 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:33.244784  5158 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 13:02:33.249401  1667 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:33.220027  5137 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:33.215107  5165 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:33.223160  5164 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:33.255033  5080 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:33.255671  5042 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.255896  4953 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:33.256249  5042 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.229029  5116 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.248672  4906 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.268311  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:33.249872  5118 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.255928  5080 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:33.178241  5151 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.239s
I20260709 13:02:33.260071  5030 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:33.297349  4971 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.570s	user 0.007s	sys 0.005s
I20260709 13:02:33.265378  5165 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:33.265816  5164 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:33.304155  5067 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.029s	sys 0.019s
I20260709 13:02:33.325652  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:33.317967  5067 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.030s	sys 0.019s
I20260709 13:02:33.310931  4910 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.302485  5135 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.312927  5165 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:33.307886  5159 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.321768  5166 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.098s
W20260709 13:02:33.358734  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48100.7us
I20260709 13:02:33.313815  4971 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.361615  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2294) took 2117 ms. Trace:
I20260709 13:02:33.362216  1587 rpcz_store.cc:276] 0709 13:02:31.243949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.095815 (+1851866us) service_pool.cc:224] Handling call
0709 13:02:33.096212 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.096531 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:33.096570 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.097169 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.288163 (+190994us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:33.288217 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.288312 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.361270 (+ 72958us) raft_consensus.cc:1685] finished
0709 13:02:33.361297 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.361526 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":190581,"threads_started":1,"wal-append.queue_time_us":227101}
I20260709 13:02:33.324483  5164 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.365491  5169 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:33.365823  4953 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:33.338169  5042 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.371071  1587 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:33.371960  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2299) took 1679 ms. Trace:
I20260709 13:02:33.372385  1587 rpcz_store.cc:276] 0709 13:02:31.693009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.370215 (+1677206us) service_pool.cc:224] Handling call
0709 13:02:33.370612 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.371628 (+  1016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.371870 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:33.384994  5161 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 13:02:33.385397  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.388288  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:33.390115  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:33.392774  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.395798  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:33.387635  5058 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.396795  1587 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:33.400303  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:33.412374  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:33.401213  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.417179  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:33.421689  4640 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:33.399722  5101 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.004s
W20260709 13:02:33.406956  4965 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.648s	user 0.011s	sys 0.004s
I20260709 13:02:33.400703  5042 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:33.433939  5030 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 13:02:33.416371  5169 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:33.450505  5179 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:33.415002  5168 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
W20260709 13:02:33.463253  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:33.445652  4640 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:33.398495  5173 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.061s
I20260709 13:02:33.466205  1587 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:33.469573  5174 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.121s
I20260709 13:02:33.472448  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45372 (request call id 3) took 1394 ms. Trace:
I20260709 13:02:33.473412  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:33.469179  5182 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:33.474529  1587 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:33.472958  2314 rpcz_store.cc:276] 0709 13:02:32.070903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.071385 (+   482us) service_pool.cc:224] Handling call
0709 13:02:33.450990 (+1379605us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe3735 00007fcdedfdb9fa 00007fcdf2fa34c9 00007fcdec51fc54 00007fcdec51fbe2 00007fcdec51fb11 00007fcdec51fa8c 00007fcdec51ed42 00007fcdec31848c 00007fcdec3178c6
0709 13:02:33.465066 (+ 14076us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296014208}
I20260709 13:02:33.444131  5101 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.448587  4965 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.464514  4640 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:33.505363  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 237141us
I20260709 13:02:33.507938  5165 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.496284  5182 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:33.497951  4640 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:33.504155  5093 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.660s	user 0.003s	sys 0.003s
I20260709 13:02:33.515077  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:33.527290  4965 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.529983  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2294) took 1178 ms. Trace:
W20260709 13:02:33.530370  5175 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.167s	user 0.001s	sys 0.001s
I20260709 13:02:33.538533  1741 rpcz_store.cc:276] 0709 13:02:32.351821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.352047 (+   226us) service_pool.cc:224] Handling call
0709 13:02:33.529781 (+1177734us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b5400404f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:33.529915 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":633669888}
I20260709 13:02:33.547706  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:33.548599  1659 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:33.522861  4939 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.513213  5176 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:33.554256  5101 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.556802  5020 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.537662  5151 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:33.558988  5142 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:33.559489  5142 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:33.571203  1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2295) took 2320 ms (client timeout 1984 ms). Trace:
W20260709 13:02:33.572249  1584 rpcz_store.cc:269] 0709 13:02:31.250331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.691174 (+440843us) service_pool.cc:224] Handling call
0709 13:02:33.570777 (+1879603us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54002184d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:33.571083 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":539463040}
I20260709 13:02:33.570282  5182 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.574255  5182 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:33.576223  4515 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:33.578991  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2367) took 2589 ms. Trace:
I20260709 13:02:33.579421  1591 rpcz_store.cc:276] 0709 13:02:30.989062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.989201 (+   139us) service_pool.cc:224] Handling call
0709 13:02:30.994550 (+  5349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:30.994931 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:30.994962 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:30.995549 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 13:02:31.345833 (+350284us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:31.345898 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:31.346016 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.337348 (+1991332us) spinlock_profiling.cc:243] Waited 54.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:33.578583 (+241235us) raft_consensus.cc:1685] finished
0709 13:02:33.578612 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.578848 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4280,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":119,"spinlock_wait_cycles":124496128,"thread_start_us":349592,"threads_started":1,"wal-append.queue_time_us":635750}
I20260709 13:02:33.545907  5168 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 13:02:33.584707  1660 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:33.585434  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.588001  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:33.593586  1591 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:33.594370  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2300) took 1809 ms. Trace:
I20260709 13:02:33.594668  1591 rpcz_store.cc:276] 0709 13:02:31.784820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.581746 (+1796926us) service_pool.cc:224] Handling call
0709 13:02:33.582130 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.594058 (+ 11928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.594309 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.596344  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2305) took 1669 ms. Trace:
I20260709 13:02:33.596668  1591 rpcz_store.cc:276] 0709 13:02:31.927455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.595488 (+1668033us) service_pool.cc:224] Handling call
0709 13:02:33.595829 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.596021 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.596049 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.596108 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:33.596140 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.596191 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.596290 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.597532  4969 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.599021  1660 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:33.572635  4939 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.602770  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 501) took 1329 ms. Trace:
I20260709 13:02:33.603243  1737 rpcz_store.cc:276] 0709 13:02:32.273620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.275446 (+  1826us) service_pool.cc:224] Handling call
0709 13:02:32.275752 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.920899 (+645147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.602464 (+681565us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:33.602689 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1318486272}
I20260709 13:02:33.611752  1660 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:33.618659  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:33.619307  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 202218us
I20260709 13:02:33.643024  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:33.643508  1587 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:33.645300  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2370) took 1413 ms. Trace:
I20260709 13:02:33.649117  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:33.649870  1741 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:33.651223  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.652935  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.645725  1587 rpcz_store.cc:276] 0709 13:02:32.232351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.644490 (+1412139us) service_pool.cc:224] Handling call
0709 13:02:33.644797 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.644998 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.645024 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.645068 (+    44us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:33.645103 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.645155 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.645247 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.658777  1587 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 13:02:33.659196  1587 pending_rounds.cc:85] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 4
I20260709 13:02:33.659432  1587 pending_rounds.cc:107] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 13:02:33.659701  1587 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:33.660053  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.660444  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2371) took 1427 ms. Trace:
I20260709 13:02:33.660825  1587 rpcz_store.cc:276] 0709 13:02:32.232606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.653940 (+1421334us) service_pool.cc:224] Handling call
0709 13:02:33.657465 (+  3525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.660189 (+  2724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.660391 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.660903  1737 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:33.665733  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:33.666138  1587 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:33.672487  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2373) took 1439 ms. Trace:
W20260709 13:02:33.667474  2310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 741) took 3738 ms (client timeout 1536 ms). Trace:
I20260709 13:02:33.673121  1587 rpcz_store.cc:276] 0709 13:02:32.233094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.665668 (+1432574us) service_pool.cc:224] Handling call
0709 13:02:33.665961 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.671809 (+  5848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.672347 (+   538us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.674901  1587 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 4. Current term is 6. Ops: [4.3-4.3]
I20260709 13:02:33.675594  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2374) took 1442 ms. Trace:
I20260709 13:02:33.675933  1587 rpcz_store.cc:276] 0709 13:02:32.233331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.674263 (+1440932us) service_pool.cc:224] Handling call
0709 13:02:33.674612 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.675315 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.675536 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:33.673239  2310 rpcz_store.cc:269] 0709 13:02:29.916924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:29.917134 (+   210us) service_pool.cc:224] Handling call
0709 13:02:33.654567 (+3737433us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:33.654761 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1642733952}
I20260709 13:02:33.644503  5032 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.681540  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.685178  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.681777  1587 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:33.683697  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:33.680097  1741 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:33.690625  5165 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.667284  5168 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.032s
W20260709 13:02:33.696553  5119 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.121s	user 0.001s	sys 0.002s
I20260709 13:02:33.684916  1733 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:33.701653  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:33.701857  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:33.702674  2309 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:33.669755  4952 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 13:02:33.702723  2318 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:33.720073  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:33.756599  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2309) took 1461 ms. Trace:
I20260709 13:02:33.710831  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1685) took 1119 ms. Trace:
I20260709 13:02:33.771261  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:33.829494  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 280343us
I20260709 13:02:33.851084  1736 rpcz_store.cc:276] 0709 13:02:32.591727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.596099 (+  4372us) service_pool.cc:224] Handling call
0709 13:02:33.710598 (+1114499us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54000e2ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:33.710748 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393326464}
I20260709 13:02:33.851779  1741 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:33.850677  1587 rpcz_store.cc:276] 0709 13:02:32.294895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.681293 (+1386398us) service_pool.cc:224] Handling call
0709 13:02:33.681592 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.686389 (+  4797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.756490 (+ 70101us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:33.857031  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141640us
I20260709 13:02:33.857331  1587 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 13:02:33.857461  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 137532us
I20260709 13:02:33.858228  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:33.813083  5027 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.859109  5190 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:33.863775  5190 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:33.864214  5190 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:33.864305  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:33.859578  5094 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.865684  4847 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:33.865526  5094 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.866089  5027 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.867722  5094 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.868327  5027 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.844676  5172 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:33.872028  1742 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
W20260709 13:02:33.833454  5119 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:02:33.872922  4967 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.873757  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2284) took 2712 ms. Trace:
I20260709 13:02:33.873432  4967 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.872128  5172 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:33.875759  1737 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:02:33.704792  5012 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.004s
W20260709 13:02:33.850108  4951 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.440s	user 0.013s	sys 0.021s
I20260709 13:02:33.879529  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:33.879441  5172 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.880534  5172 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:33.881196  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2268) took 3300 ms. Trace:
I20260709 13:02:33.875692  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2311) took 1465 ms. Trace:
W20260709 13:02:33.881630  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 982133us
I20260709 13:02:33.874125  1588 rpcz_store.cc:276] 0709 13:02:31.161228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.164295 (+  3067us) service_pool.cc:224] Handling call
0709 13:02:31.245041 (+ 80746us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:31.245587 (+   546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:31.245635 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:31.253099 (+  7464us) log.cc:844] Serialized 52 byte log entry
0709 13:02:32.250543 (+997444us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:32.298677 (+ 48134us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.492528 (+193851us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.539943 (+ 47415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.540052 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.579138 (+1039086us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b1800319240. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:33.695752 (+116614us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:33.873494 (+177742us) raft_consensus.cc:1685] finished
0709 13:02:33.873538 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.873682 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56005,"prepare.queue_time_us":57773,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":3284,"raft.queue_time_us":296641,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":17380,"spinlock_wait_cycles":22280192,"thread_start_us":1158108,"threads_started":2,"wal-append.queue_time_us":1164581}
I20260709 13:02:33.681277  4969 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.902009  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:33.902321  5196 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:02:33.902743  2311 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:33.903703  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:33.882287  1585 rpcz_store.cc:276] 0709 13:02:30.580719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.580907 (+   188us) service_pool.cc:224] Handling call
0709 13:02:30.585954 (+  5047us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:30.586265 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:30.646869 (+ 60604us) log.cc:844] Serialized 14 byte log entry
0709 13:02:32.890010 (+2243141us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b1800422f20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:02:32.937813 (+ 47803us) log.cc:844] Serialized 14 byte log entry
0709 13:02:33.880604 (+942791us) spinlock_profiling.cc:243] Waited 8.38 ms on lock 0x7b1800422f20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:33.880815 (+   211us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.880872 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:33.880921 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.881030 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.881142 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":937317,"prepare.queue_time_us":67279,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":962767,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":24567,"spinlock_wait_cycles":52415488,"thread_start_us":29966,"threads_started":1,"wal-append.queue_time_us":2348113}
I20260709 13:02:33.872062  5166 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:33.906142  4781 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:33.906401  5166 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:33.906844  4781 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:33.907389  5166 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.907922  4781 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.908677  5166 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:33.716710  5169 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 13:02:33.828539  4977 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:33.899154  5066 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:33.697312  5142 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.882885  1587 rpcz_store.cc:276] 0709 13:02:32.410410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.856517 (+1446107us) service_pool.cc:224] Handling call
0709 13:02:33.857018 (+   501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.861901 (+  4883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.875635 (+ 13734us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:33.892118  4951 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.555s	user 0.013s	sys 0.021s
I20260709 13:02:33.874348  4971 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:33.916204  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2301) took 2110 ms. Trace:
I20260709 13:02:33.941452  1584 rpcz_store.cc:276] 0709 13:02:31.805839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.583244 (+1777405us) service_pool.cc:224] Handling call
0709 13:02:33.583584 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.583826 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.583852 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.585353 (+  1501us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.838306 (+252953us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:33.838364 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.838471 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:33.915908 (+ 77437us) raft_consensus.cc:1685] finished
0709 13:02:33.915942 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.916128 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":218018,"threads_started":1,"wal-append.queue_time_us":301876}
I20260709 13:02:33.922804  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2312) took 1384 ms. Trace:
I20260709 13:02:33.942873  1588 rpcz_store.cc:276] 0709 13:02:32.538316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.921780 (+1383464us) service_pool.cc:224] Handling call
0709 13:02:33.922255 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.922497 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.922523 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.922562 (+    39us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:33.922591 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.922641 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.922748 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.701720  4956 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:33.874969  4952 consensus_queue.cc:1243] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:02:33.939954  4977 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.707443  5137 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.877372  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:33.957568  1737 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:33.938673  1587 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:02:33.924996  4640 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.945436  5190 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:33.959406  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2314) took 1177 ms. Trace:
I20260709 13:02:33.961558  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2297) took 2695 ms. Trace:
I20260709 13:02:33.975554  1584 rpcz_store.cc:276] 0709 13:02:32.782372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.958187 (+1175815us) service_pool.cc:224] Handling call
0709 13:02:33.958630 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.958934 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.958969 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.959038 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:33.959084 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.959170 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.959334 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:33.975628  1586 rpcz_store.cc:276] 0709 13:02:31.266581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.109463 (+1842882us) service_pool.cc:224] Handling call
0709 13:02:33.109899 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:33.110197 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:33.313619 (+203422us) log.cc:844] Serialized 14 byte log entry
0709 13:02:33.961089 (+647470us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:33.961167 (+    78us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:33.961232 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.961315 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:33.961465 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":441455,"prepare.queue_time_us":213842,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":441746,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":375052,"threads_started":2,"wal-append.queue_time_us":323973}
W20260709 13:02:33.963801  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52821us
I20260709 13:02:33.978487  1584 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:02:33.979538  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:33.911108  5012 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:33.948904  5142 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:33.966559  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2380) took 1374 ms. Trace:
I20260709 13:02:34.003934  1587 rpcz_store.cc:276] 0709 13:02:32.591727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.937963 (+1346236us) service_pool.cc:224] Handling call
0709 13:02:33.938437 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.966272 (+ 27835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.966504 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.005052  1588 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:02:33.949106  4977 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:33.991822  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2224) took 2133 ms (client timeout 2360 ms). Trace:
I20260709 13:02:33.992494  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 472) took 1171 ms. Trace:
I20260709 13:02:34.005708  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 470) took 1198 ms. Trace:
W20260709 13:02:34.007113  1661 rpcz_store.cc:269] 0709 13:02:31.841551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.842971 (+  1420us) service_pool.cc:224] Handling call
0709 13:02:33.974875 (+2131904us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b5400117e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:33.975050 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1384208128}
I20260709 13:02:33.944029  5066 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.009526  1588 rpcz_store.cc:276] 0709 13:02:32.807435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.957541 (+1150106us) service_pool.cc:224] Handling call
0709 13:02:34.001228 (+ 43687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.005395 (+  4167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.005652 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.012207  1740 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:34.082866  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.084214  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.085422  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:34.086949  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.087914  1664 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:34.005604  5124 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:34.089664  5124 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.081079  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.091333  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:33.993207  5038 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.434s	user 0.018s	sys 0.023s
I20260709 13:02:34.090206  5124 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:34.071439  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 475) took 1262 ms. Trace:
I20260709 13:02:34.110424  1587 rpcz_store.cc:276] 0709 13:02:32.808948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.004913 (+1195965us) service_pool.cc:224] Handling call
0709 13:02:34.005224 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.005411 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.005435 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:34.071138 (+ 65703us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:34.071190 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.071253 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.071371 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.098760  5137 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.100443  4963 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:34.083066  4685 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.010887  1584 rpcz_store.cc:276] 0709 13:02:32.808042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.977558 (+1169516us) service_pool.cc:224] Handling call
0709 13:02:33.978048 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.979421 (+  1373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.979696 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.106940  5198 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:34.105013  5038 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.566s	user 0.018s	sys 0.024s
I20260709 13:02:34.002146  5110 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 13:02:34.108450  4969 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:34.124584  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.107903  5115 raft_consensus.cc:2764] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:02:34.105492  5135 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.112272  4984 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 13:02:34.133133  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.127506  1584 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260709 13:02:34.142035  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 233608us
I20260709 13:02:34.143306  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.146692  1584 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:34.120271  5032 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 13:02:34.147794  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.149534  1584 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:34.145006  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.150125  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.150880  1592 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:34.151626  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 2.26622e+06us
I20260709 13:02:34.152081  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:34.152429  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.119725  4685 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.154469  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.156134  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.157483  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.158094  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.158763  2310 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:34.133548  4969 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:34.160617  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.162451  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.158696  1592 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:34.162868  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.120950  5198 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.164398  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.165356  1592 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:02:34.157367  1706 connection.cc:256] RPC call timeout handler was delayed by -1.2433296179999616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260709 13:02:34.165578  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.167433  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.171152  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:34.172698  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.173106  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.174572  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.174944  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.175953  1592 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:34.176059  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.176838  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.177631  5067 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:34.177850  2315 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:34.179565  1586 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: [6.7-7.8]
I20260709 13:02:34.180512  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.146554  4965 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.177241  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:34.174160  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.182946  1584 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:34.135588  5135 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.189146  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.163961  5121 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:34.197654  5142 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.198479  5105 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.202877  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.209980  2310 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:34.205394  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:34.203680  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.213011  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.214849  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.217806  1587 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:34.214430  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:34.183848  4640 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.221000  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:34.192584  4965 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.220301  4947 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:34.263971  4947 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.264514  4947 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:34.267124  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1680) took 1608 ms (client timeout 2030 ms). Trace:
I20260709 13:02:34.215848  4984 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:34.267871  1667 rpcz_store.cc:269] 0709 13:02:32.659214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.677797 (+ 18583us) service_pool.cc:224] Handling call
0709 13:02:34.266787 (+1588990us) spinlock_profiling.cc:243] Waited 570 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.267041 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1310896512}
I20260709 13:02:34.215734  5142 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.215904  5105 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.270901  5142 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:34.273375  5194 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.716s	user 0.001s	sys 0.000s
I20260709 13:02:34.233129  4939 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:34.245609  4969 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.246446  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.275940  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.277091  4971 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:34.277573  4971 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.278023  4971 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:34.278563  5144 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.343s	user 0.004s	sys 0.007s
I20260709 13:02:34.248158  5135 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.280061  5173 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.280687  5173 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:34.281337  5173 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.286545  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1673) took 1701 ms. Trace:
I20260709 13:02:34.287099  1665 rpcz_store.cc:276] 0709 13:02:32.585205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.586349 (+  1144us) service_pool.cc:224] Handling call
0709 13:02:32.587376 (+  1027us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:32.587771 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:32.587801 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:32.588348 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 13:02:32.686911 (+ 98563us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:32.686957 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:32.687025 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.277831 (+1590806us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:34.286043 (+  8212us) raft_consensus.cc:1685] finished
0709 13:02:34.286079 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.286454 (+   375us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6435,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":12731,"spinlock_wait_cycles":8212864,"thread_start_us":96492,"threads_started":1,"wal-append.queue_time_us":315302}
I20260709 13:02:34.200769  5110 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:34.248857  4640 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:34.289633  4640 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:34.207948  5067 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:34.250417  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:34.253216  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:34.291610  1736 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:02:34.253970  5066 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.256346  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2293) took 3012 ms. Trace:
I20260709 13:02:34.258383  5198 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.259199  5094 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.267266  1659 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 9. Ops: []
I20260709 13:02:34.268098  4984 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.216395  5118 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:34.207013  5126 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:34.228827  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:34.295910  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81185.5us
I20260709 13:02:34.302791  1590 rpcz_store.cc:276] 0709 13:02:31.243686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.097760 (+1854074us) service_pool.cc:224] Handling call
0709 13:02:33.099020 (+  1260us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.913260 (+814240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.913304 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.913886 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.121030 (+207144us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:34.121073 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.121358 (+   285us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.256097 (+134739us) raft_consensus.cc:1685] finished
0709 13:02:34.256124 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.256278 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":813781,"raft.queue_time_us":815125,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":206393,"threads_started":1,"wal-append.queue_time_us":252430}
I20260709 13:02:34.331575  2310 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:34.332069  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.209223  5023 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:34.305002  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.307597  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.335276  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.338150  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.302387  5105 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.310148  4956 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:34.311760  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:34.343786  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:34.344194  1733 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:34.344669  1590 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:34.318068  5142 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:34.322594  1667 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:34.224725  5137 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:34.339988  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.357049  5142 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.357544  5142 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.357887  5142 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.295624  5019 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.358522  5019 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.360705  5019 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:34.347551  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.305233  5121 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:34.348453  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.349109  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.363997  2311 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:34.367519  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.373776  1556 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:34.374166  1556 leader_election.cc:403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:34.385001  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:34.329721  5094 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.343250  5169 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 13:02:34.391906  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 502) took 2457 ms (client timeout 1522 ms). Trace:
I20260709 13:02:34.348937  4812 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:34.397998  1666 rpcz_store.cc:269] 0709 13:02:31.934791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.937048 (+  2257us) service_pool.cc:224] Handling call
0709 13:02:34.391682 (+2454634us) spinlock_profiling.cc:243] Waited 58.1 ms on lock 0x7b54003c9454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.391811 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133558016}
I20260709 13:02:34.424098  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:34.424988  2312 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:34.419718  1590 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:34.427943  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:34.459316  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:34.461977  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.462886  1590 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:34.464479  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:34.466090  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.340685  5023 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.388190  5198 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.390858  5066 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:34.480396  1736 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:02:34.480062  2318 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.003s	sys 0.006s
W20260709 13:02:34.483939  2318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.004s	sys 0.006s
I20260709 13:02:34.397882  5169 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:34.435130  5126 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:34.399626  5136 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 13:02:34.475354  5143 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:34.387966  5124 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.507085  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.508653  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.509779  4812 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:34.510228  4812 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:34.510622  4812 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:34.514782  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 474) took 1706 ms. Trace:
I20260709 13:02:34.515260  1586 rpcz_store.cc:276] 0709 13:02:32.808648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.978469 (+1169821us) service_pool.cc:224] Handling call
0709 13:02:34.083514 (+105045us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:34.179957 (+ 96443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.514447 (+334490us) spinlock_profiling.cc:243] Waited 95.8 ms on lock 0x7b54003b15d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:34.514708 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220301056}
I20260709 13:02:34.517040  1586 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 13:02:34.517835  5140 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:34.518277  5140 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:34.519346  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.520090  2311 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:34.520536  4965 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.521029  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:34.521224  5066 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:34.521674  5066 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:34.522220  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 504) took 2088 ms (client timeout 1618 ms). Trace:
W20260709 13:02:34.522665  1742 rpcz_store.cc:269] 0709 13:02:32.433977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.489467 (+ 55490us) service_pool.cc:224] Handling call
0709 13:02:34.522022 (+2032555us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400434c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.522134 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3157265152}
I20260709 13:02:34.522514  5066 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.523978  5066 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:34.526043  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2303) took 2093 ms (client timeout 1580 ms). Trace:
I20260709 13:02:34.526209  5142 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:34.526512  1739 rpcz_store.cc:269] 0709 13:02:32.432358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.486035 (+ 53677us) service_pool.cc:224] Handling call
0709 13:02:34.525805 (+2039770us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.525964 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1383138688}
I20260709 13:02:34.528656  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2301) took 2112 ms. Trace:
I20260709 13:02:34.528483  5142 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.529067  1735 rpcz_store.cc:276] 0709 13:02:32.416046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.535041 (+118995us) service_pool.cc:224] Handling call
0709 13:02:32.535405 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:32.800301 (+264896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.528406 (+1728105us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54000e1d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:34.528591 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607261952}
I20260709 13:02:34.533476  1735 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:02:34.535629  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.537189  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2321) took 1228 ms. Trace:
I20260709 13:02:34.391402  4640 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.538357  2310 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:34.539103  2311 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 13:02:34.538008  1586 rpcz_store.cc:276] 0709 13:02:33.308514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.516283 (+1207769us) service_pool.cc:224] Handling call
0709 13:02:34.516647 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.517431 (+   784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.537127 (+ 19696us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:34.540740  5142 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:34.486814  5166 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:02:34.566409  2312 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:34.564376  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.495333  5169 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.583254  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:34.583496  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.499280  4947 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.584432  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.585042  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.566026  5142 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.588868  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.591571  5142 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:34.593900  4953 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:34.591779  5172 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:34.602306  5172 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:34.602754  5172 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:34.493103  5198 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.605386  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:34.581067  5023 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.607882  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.639308  2311 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:34.634086  2310 raft_consensus.cc:2368] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:34.604828  4967 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.581696  4942 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:02:34.647539  5142 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.592357  5137 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.575935  4971 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.650619  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.639448  4969 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 13:02:34.651757  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 313726us
I20260709 13:02:34.683495  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.669317  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.685248  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.659230  5137 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.670596  5135 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.694020  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.683368  4969 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.700398  4325 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.696310  5217 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:34.700907  5137 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.705145  4965 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.725076  4971 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.723726  4965 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:34.713167  4325 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:34.734141  4965 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.734319  4325 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.736070  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2319) took 1682 ms. Trace:
I20260709 13:02:34.729985  5124 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.736488  1590 rpcz_store.cc:276] 0709 13:02:33.053202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.476518 (+1423316us) service_pool.cc:224] Handling call
0709 13:02:34.476863 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.477135 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:34.479493 (+  2358us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.735674 (+256181us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:34.735734 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:34.735782 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.735858 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.736004 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2924,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":53,"raft.queue_time_us":104,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":255525,"threads_started":1}
W20260709 13:02:34.720788  5211 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.070s	user 0.001s	sys 0.001s
I20260709 13:02:34.707334  5135 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:34.731213  5099 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.738837  5135 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.739751  5099 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:34.740299  5215 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 13:02:34.713749  4969 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:34.748495  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.758301  1666 raft_consensus.cc:2368] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10.
I20260709 13:02:34.760586  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.762840  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:34.762643  4325 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.764482  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.766436  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.711081  5214 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:34.789770  5214 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.790359  1666 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:34.790243  5173 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.786314  5126 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:34.720139  5217 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:34.716403  5136 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:34.790277  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.795725  1557 leader_election.cc:400] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:34.803973  1557 leader_election.cc:403] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:02:34.804003  1664 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:34.796394  4325 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:34.713795  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.799597  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:34.738837  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.802913  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:34.803102  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2298) took 2432 ms (client timeout 1984 ms). Trace:
I20260709 13:02:34.805732  1733 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:34.793722  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:34.806770  5210 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:02:34.807374  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.833160  1668 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:34.808147  5221 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:34.808645  5135 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.809875  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.837001  2316 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:34.790333  5214 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:34.811687  5137 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.813916  5156 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:34.799086  5126 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:34.816025  4966 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:02:34.814564  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 100853us
I20260709 13:02:34.804874  5217 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:34.821060  5190 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:34.825066  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:34.825889  1734 rpcz_store.cc:269] 0709 13:02:32.365933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.403385 (+ 37452us) service_pool.cc:224] Handling call
0709 13:02:34.797685 (+2394300us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400404554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.797831 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444779392}
I20260709 13:02:34.827162  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:34.863713  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:34.864274  1590 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:34.828248  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:34.829993  5136 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:34.829154  1589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2372) took 2596 ms (client timeout 1884 ms). Trace:
W20260709 13:02:34.829571  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.829631  1588 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 13:02:34.807410  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.798668  5173 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.852345  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 661) took 1079 ms. Trace:
I20260709 13:02:34.852814  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 657) took 1164 ms. Trace:
I20260709 13:02:34.816015  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:34.874531  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58618.7us
I20260709 13:02:34.875921  2316 rpcz_store.cc:276] 0709 13:02:33.688072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.688986 (+   914us) service_pool.cc:224] Handling call
0709 13:02:34.809735 (+1120749us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:34.852749 (+ 43014us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":386874112}
I20260709 13:02:34.877801  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.856554  5124 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:34.856288  5142 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:34.854251  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 262215us
I20260709 13:02:34.855417  2280 leader_election.cc:400] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10.
I20260709 13:02:34.894733  2280 leader_election.cc:403] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:34.817910  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 477) took 1892 ms. Trace:
I20260709 13:02:34.894948  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:34.853770  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 344331us
I20260709 13:02:34.830192  5066 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:34.864377  1733 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:34.899856  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:34.872923  5166 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:34.869153  1590 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 13:02:34.867213  1589 rpcz_store.cc:269] 0709 13:02:32.232835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.664718 (+431883us) service_pool.cc:224] Handling call
0709 13:02:34.828941 (+2164223us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:34.829086 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":615962880}
I20260709 13:02:34.872371  2318 rpcz_store.cc:276] 0709 13:02:33.773379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.860113 (+ 86734us) service_pool.cc:224] Handling call
0709 13:02:33.860820 (+   707us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:33.878661 (+ 17841us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:33.882019 (+  3358us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:33.927999 (+ 45980us) log.cc:844] Serialized 52 byte log entry
0709 13:02:34.485318 (+557319us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.485362 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.485426 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.852090 (+366664us) raft_consensus.cc:1685] finished
0709 13:02:34.852114 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.852275 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3548,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":34186,"thread_start_us":515636,"threads_started":1,"wal-append.queue_time_us":711989}
I20260709 13:02:34.886981  5190 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:02:34.898833  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43571us
I20260709 13:02:34.887414  5198 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.867214  4947 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.859071  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.900471  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.868302  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 656) took 1197 ms. Trace:
I20260709 13:02:34.927677  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.928334  2314 rpcz_store.cc:276] 0709 13:02:33.680773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.686370 (+  5597us) service_pool.cc:224] Handling call
0709 13:02:34.807254 (+1120884us) spinlock_profiling.cc:243] Waited 954 us on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:34.868230 (+ 60976us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2193152}
I20260709 13:02:34.862800  5126 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:34.895470  1592 rpcz_store.cc:276] 0709 13:02:32.925138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.177699 (+1252561us) service_pool.cc:224] Handling call
0709 13:02:34.178005 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.303157 (+125152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.303207 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.327821 (+ 24614us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.575785 (+247964us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:34.575842 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.734141 (+158299us) spinlock_profiling.cc:243] Waited 77.5 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:34.734202 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.817483 (+ 83281us) raft_consensus.cc:1685] finished
0709 13:02:34.817533 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.817834 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54693,"raft.queue_time_us":48930,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"spinlock_wait_cycles":178197888,"thread_start_us":212942,"threads_started":1,"wal-append.queue_time_us":378823}
I20260709 13:02:34.920325  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:34.936601  2318 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:34.936990  2318 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:34.922058  1590 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:34.923564  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.942939  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:34.926035  1589 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:34.945434  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86492.8us
I20260709 13:02:34.886929  4956 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 13:02:34.946410  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:34.948742  1590 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:02:34.949782  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2316) took 1235 ms. Trace:
W20260709 13:02:34.959569  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59270.1us
W20260709 13:02:34.927508  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51997.5us
I20260709 13:02:34.963276  1589 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:34.906376  4942 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10.
W20260709 13:02:34.953433  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:34.964675  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:34.926712  4640 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:34.944260  2318 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:34.960646  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:34.955452  1737 rpcz_store.cc:276] 0709 13:02:33.714003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.960337 (+246334us) service_pool.cc:224] Handling call
0709 13:02:34.071497 (+111160us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:34.072439 (+   942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.072483 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:34.073344 (+   861us) log.cc:844] Serialized 52 byte log entry
0709 13:02:34.483464 (+410120us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:34.483506 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.483576 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:34.949493 (+465917us) raft_consensus.cc:1685] finished
0709 13:02:34.949528 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:34.949719 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"spinlock_wait_cycles":1408,"thread_start_us":409261,"threads_started":1,"wal-append.queue_time_us":534568}
W20260709 13:02:34.957993  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55868us
I20260709 13:02:34.938606  5190 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:34.973284  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.002722  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.988534  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:34.988747  4325 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:34.975068  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:34.975711  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:34.987425  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.933086  4966 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 13:02:35.008520  5237 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:35.008831  4966 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:35.009119  5237 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.009985  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:34.988780  4947 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.011281  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.011400  4947 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.012508  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.002491  5141 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:02:34.989082  5209 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
I20260709 13:02:35.015547  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.016733  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.024927  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2322) took 1148 ms. Trace:
I20260709 13:02:35.025466  1742 rpcz_store.cc:276] 0709 13:02:33.876678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.524665 (+647987us) service_pool.cc:224] Handling call
0709 13:02:34.525026 (+   361us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:34.525456 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.525484 (+    28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:34.569928 (+ 44444us) log.cc:844] Serialized 78 byte log entry
0709 13:02:34.880373 (+310445us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.880433 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.880534 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.024642 (+144108us) raft_consensus.cc:1685] finished
0709 13:02:35.024667 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.024849 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16300,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":20464,"thread_start_us":286243,"threads_started":1,"wal-append.queue_time_us":437299}
I20260709 13:02:35.002358  4942 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.031s
W20260709 13:02:34.988620  5223 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:02:35.013564  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2326) took 1005 ms. Trace:
I20260709 13:02:34.955199  5166 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:35.042836  5166 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.043325  5166 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:35.045940  5141 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:34.970408  5156 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:35.046752  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2320) took 1787 ms. Trace:
I20260709 13:02:35.047439  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 507) took 1496 ms. Trace:
I20260709 13:02:35.047084  5141 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:34.970585  5231 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:35.048465  5231 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:35.047209  1593 rpcz_store.cc:276] 0709 13:02:33.259310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.498646 (+1239336us) service_pool.cc:224] Handling call
0709 13:02:34.499076 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.499558 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:34.499599 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.513797 (+ 14198us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.856338 (+342541us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:34.856392 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.856474 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.046473 (+189999us) raft_consensus.cc:1685] finished
0709 13:02:35.046497 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.046679 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14949,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":339114,"threads_started":1,"wal-append.queue_time_us":352105}
I20260709 13:02:35.052539  5105 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.993852  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:35.079713  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 85934.3us
I20260709 13:02:35.081878  4325 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:35.083870  1668 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 13:02:35.038300  5220 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 13:02:35.086728  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2318) took 1235 ms. Trace:
I20260709 13:02:35.085121  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.034494  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:34.995079  1733 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:35.036389  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:35.037484  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1690) took 2438 ms (client timeout 2347 ms). Trace:
W20260709 13:02:35.100286  1738 rpcz_store.cc:269] 0709 13:02:32.599102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:32.681595 (+ 82493us) service_pool.cc:224] Handling call
0709 13:02:35.037229 (+2355634us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:35.037407 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29227264}
I20260709 13:02:35.038363  1735 rpcz_store.cc:276] 0709 13:02:34.008167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.534548 (+526381us) service_pool.cc:224] Handling call
0709 13:02:34.534922 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.760774 (+225852us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:34.799370 (+ 38596us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.881337 (+ 81967us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.884755 (+  3418us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.884996 (+   241us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.885704 (+   708us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.885887 (+   183us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:34.886129 (+   242us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.886345 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.886456 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.013330 (+126874us) raft_consensus.cc:1685] finished
0709 13:02:35.013352 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.013480 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":136929,"raft.queue_time_us":49295,"raft.run_cpu_time_us":568,"raft.run_wall_time_us":15642,"spinlock_wait_cycles":17251712,"thread_start_us":306633,"threads_started":2,"wal-append.queue_time_us":210919}
I20260709 13:02:35.041177  5136 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.041054  5240 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:35.053224  5237 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.048938  5231 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:35.055902  5159 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:02:35.058175  5137 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:35.058902  5216 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.267s	user 0.002s	sys 0.000s
I20260709 13:02:35.058344  5172 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.059166  5135 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:34.930495  4812 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.032908  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.047940  1659 rpcz_store.cc:276] 0709 13:02:33.550752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.551092 (+   340us) service_pool.cc:224] Handling call
0709 13:02:34.266996 (+715904us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.304880 (+ 37884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.047244 (+742364us) spinlock_profiling.cc:243] Waited 715 ms on lock 0x7b5400118850. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:35.047380 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1645511936}
I20260709 13:02:35.151012  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.151700  1659 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:35.152632  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.115010  4947 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:34.966436  5142 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:35.157531  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.158289  1668 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:35.099499  1664 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:35.103544  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.081202  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:35.115360  1735 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:35.119037  1738 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:35.087126  1741 rpcz_store.cc:276] 0709 13:02:33.851362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.478403 (+627041us) service_pool.cc:224] Handling call
0709 13:02:34.478761 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.479173 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:34.479202 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.479645 (+   443us) log.cc:844] Serialized 26 byte log entry
0709 13:02:34.786107 (+306462us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:34.812780 (+ 26673us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.813352 (+   572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:34.813436 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.086489 (+273053us) raft_consensus.cc:1685] finished
0709 13:02:35.086511 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.086659 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13355,"prepare.queue_time_us":125213,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":232,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":9034,"thread_start_us":296864,"threads_started":1,"wal-append.queue_time_us":390641}
I20260709 13:02:35.135605  5166 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:35.139793  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 171385us
I20260709 13:02:35.140709  2310 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:35.143615  1733 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 9. Ops: []
I20260709 13:02:35.146469  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2319) took 1288 ms. Trace:
I20260709 13:02:35.156229  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.163493  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:35.156387  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202923us
I20260709 13:02:35.208870  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2330) took 1077 ms. Trace:
W20260709 13:02:35.163579  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82538.2us
I20260709 13:02:35.211427  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.209851  1659 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:35.212209  1592 raft_consensus.cc:2368] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:02:35.216270  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.209349  1738 rpcz_store.cc:276] 0709 13:02:34.131872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.117812 (+985940us) service_pool.cc:224] Handling call
0709 13:02:35.118731 (+   919us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.208587 (+ 89856us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.208804 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:35.216904  1592 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:35.218879  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.220422  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:35.222170  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.222882  1592 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:35.224051  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.212021  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:35.229637  1589 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 13:02:35.244194  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.229534  1593 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 13:02:35.245215  1664 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:35.227947  1735 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:35.246260  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.209323  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.248301  1733 raft_consensus.cc:2368] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:35.229384  1585 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:35.228595  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.254566  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.255323  1741 raft_consensus.cc:2368] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:35.256064  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.256599  1741 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:35.257488  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.227614  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2327) took 1152 ms. Trace:
I20260709 13:02:35.213377  1736 rpcz_store.cc:276] 0709 13:02:33.857901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.481781 (+623880us) service_pool.cc:224] Handling call
0709 13:02:34.482125 (+   344us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:34.489851 (+  7726us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:34.853075 (+363224us) log.cc:844] Serialized 14 byte log entry
0709 13:02:34.930412 (+ 77337us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:34.947659 (+ 17247us) log.cc:844] Serialized 78 byte log entry
0709 13:02:34.947883 (+   224us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:35.136628 (+188745us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.138188 (+  1560us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.145704 (+  7516us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.146100 (+   396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.146185 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.146227 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:35.146240 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.146407 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42768,"prepare.queue_time_us":119569,"prepare.run_cpu_time_us":195,"prepare.run_wall_time_us":30576,"raft.queue_time_us":46240,"raft.run_cpu_time_us":690,"raft.run_wall_time_us":57466,"spinlock_wait_cycles":22912,"thread_start_us":533024,"threads_started":3,"wal-append.queue_time_us":109163}
I20260709 13:02:35.259603  1737 rpcz_store.cc:276] 0709 13:02:34.074905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.109534 (+1034629us) service_pool.cc:224] Handling call
0709 13:02:35.109993 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.124466 (+ 14473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.124502 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.149307 (+ 24805us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.225776 (+ 76469us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:35.225839 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.225932 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.227378 (+  1446us) raft_consensus.cc:1685] finished
0709 13:02:35.227402 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.227555 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":175,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":72665,"threads_started":1,"wal-append.queue_time_us":73081}
I20260709 13:02:35.262099  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2323) took 1350 ms. Trace:
I20260709 13:02:35.263063  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.265765  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.267498  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.267632  4640 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.268003  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.262126  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.268831  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.273993  1741 raft_consensus.cc:2368] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
W20260709 13:02:35.275302  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 302132us
I20260709 13:02:35.276175  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.260198  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.278285  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.279635  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.280647  1736 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:35.282848  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.284098  1592 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:35.285168  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.286896  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.287552  1585 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.279836  1741 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:35.291101  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:35.291606  1593 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:35.268778  1733 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:35.289697  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:35.268697  4640 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.292860  1664 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:35.300299  2314 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:02:35.300134  1737 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:35.306689  1741 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:35.307416  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.311833  1668 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:02:35.313091  1659 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:35.253997  4906 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:35.317155  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.318077  1733 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:35.317914  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.318966  1741 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 8
W20260709 13:02:35.295680  5232 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 13:02:35.327364  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:35.304875  5238 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:02:35.331094  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:35.285573  5142 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.331127  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.332847  1736 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 13:02:35.336723  1735 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:35.334218  1733 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 13:02:35.338306  1733 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:35.269584  5099 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:02:35.265065  5141 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 13:02:35.344642  1741 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:35.325160  5156 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:35.343261  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 130014us
I20260709 13:02:35.284685  4939 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:02:35.162657  5190 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.354708  1585 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:35.363569  1741 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:35.380870  1741 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:35.382592  1741 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:35.321775  4812 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:35.385608  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.386293  1585 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.373410  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.387104  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:35.391901  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:35.378201  4965 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.643s	user 0.013s	sys 0.000s
W20260709 13:02:35.397168  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69911.5us
I20260709 13:02:35.353690  5244 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:35.399011  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:35.402554  1585 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:35.405365  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.405684  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:35.406122  1585 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:35.388542  5141 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:35.344128  5142 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.416563  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.419023  1585 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:35.422914  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:35.425757  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.426568  1585 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:35.427397  5166 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:35.427810  5169 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.843s	user 0.008s	sys 0.005s
I20260709 13:02:35.406445  5244 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.428299  5169 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.429874  1662 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 13:02:35.430174  5244 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.434204  5248 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:35.434517  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.308s	user 0.028s	sys 0.040s
I20260709 13:02:35.434798  5248 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.435307  5248 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:35.437160  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.438714  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.439222  1592 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.442266  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 507) took 2216 ms. Trace:
I20260709 13:02:35.442798  1740 rpcz_store.cc:276] 0709 13:02:33.226134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.243976 (+ 17842us) service_pool.cc:224] Handling call
0709 13:02:33.244319 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.083133 (+838814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.441937 (+1358804us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:35.442176 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1765553024}
W20260709 13:02:35.446161  4969 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.704s	user 0.011s	sys 0.001s
I20260709 13:02:35.447031  4977 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:35.447469  4977 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:35.448489  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:35.448338  4977 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.449211  2317 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.416106  5228 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:35.455063  4977 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.455524  5012 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.446655  4969 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.458920  5038 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:35.459203  4969 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:35.459424  5038 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.460420  5038 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.405668  5224 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.461808  5224 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.464327  5224 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.465950  1738 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:35.471925  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.474064  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.474794  1585 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:35.477564  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.478354  1585 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:35.482203  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.482961  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:35.487613  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62940.7us
W20260709 13:02:35.434918  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.309s	user 0.028s	sys 0.040s
I20260709 13:02:35.483968  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:35.492045  2317 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:35.492934  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 660) took 1744 ms. Trace:
I20260709 13:02:35.493594  1592 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:35.495384  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.496106  1592 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:35.488883  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:35.465242  4984 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.498466  1585 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:35.420812  5137 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:35.403002  4965 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:35.507534  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108590us
I20260709 13:02:35.511762  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:35.509493  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2369) took 3277 ms. Trace:
I20260709 13:02:35.507917  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:35.493633  2317 rpcz_store.cc:276] 0709 13:02:33.748190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.748364 (+   174us) service_pool.cc:224] Handling call
0709 13:02:35.448313 (+1699949us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:35.492846 (+ 44533us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2928000}
I20260709 13:02:35.466954  4737 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:35.508955  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:35.519439  1585 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:35.513025  1591 rpcz_store.cc:276] 0709 13:02:32.232078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.597598 (+1365520us) service_pool.cc:224] Handling call
0709 13:02:33.597963 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.961612 (+363649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.961646 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:34.102418 (+140772us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.489389 (+1386971us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:35.508917 (+ 19528us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.509164 (+   247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.509252 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.509294 (+    42us) raft_consensus.cc:1685] finished
0709 13:02:35.509306 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.509436 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":363667,"prepare.queue_time_us":20588,"raft.queue_time_us":363678,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":76,"thread_start_us":1363590,"threads_started":1,"wal-append.queue_time_us":596718}
I20260709 13:02:35.520533  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:35.455128  5228 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.522770  1708 leader_election.cc:400] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:35.523317  1708 leader_election.cc:403] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 13:02:35.491663  4939 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 13:02:35.525871  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 101466us
I20260709 13:02:35.533416  1736 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:02:35.497586  5038 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.512884  5254 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:35.508551  5012 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:35.511226  5110 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:35.543011  1590 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:35.547257  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:35.548043  2315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 750) took 3577 ms (client timeout 1737 ms). Trace:
I20260709 13:02:35.545267  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:35.538612  4984 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:35.578316  5169 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.578490  4984 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:35.555579  5254 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:35.579658  4984 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.580132  5254 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.581732  4969 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:35.582201  4969 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:35.515836  5137 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.539767  4808 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.538802  4826 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:35.555871  5228 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.488683  5126 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.497517  5077 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 13:02:35.538335  4939 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:35.542996  4971 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:35.608078  5149 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.696s	user 0.000s	sys 0.011s
I20260709 13:02:35.576423  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:35.609987  5224 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:35.610889  5224 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:35.539218  5247 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.582556  4969 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:35.613895  5247 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.615180  5107 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.262921  1739 rpcz_store.cc:276] 0709 13:02:33.911758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.527485 (+615727us) service_pool.cc:224] Handling call
0709 13:02:34.527785 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:34.528015 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:34.828294 (+300279us) log.cc:844] Serialized 14 byte log entry
0709 13:02:35.261740 (+433446us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:35.261802 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:35.261857 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.261931 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.262043 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36561,"prepare.queue_time_us":139411,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":10915,"thread_start_us":696273,"threads_started":2,"wal-append.queue_time_us":423398}
I20260709 13:02:35.566815  5141 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:35.583113  5214 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.719s	user 0.012s	sys 0.000s
I20260709 13:02:35.596078  4826 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:35.604305  5126 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.606590  4685 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.606786  4939 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:35.607615  4971 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:35.576026  2315 rpcz_store.cc:269] 0709 13:02:31.970746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:31.970889 (+   143us) service_pool.cc:224] Handling call
0709 13:02:35.547849 (+3576960us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b54007233d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:35.547970 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4468752768}
I20260709 13:02:35.599845  4808 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.622268  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:35.616251  5247 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.583022  5264 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:35.630995  2313 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:35.633723  2313 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 654) took 1967 ms (client timeout 2392 ms). Trace:
W20260709 13:02:35.634166  2313 rpcz_store.cc:269] 0709 13:02:33.666572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.666755 (+   183us) service_pool.cc:224] Handling call
0709 13:02:35.621963 (+1955208us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:02:35.633637 (+ 11674us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":399000960}
I20260709 13:02:35.627143  4826 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:35.642311  5200 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.660s	user 0.004s	sys 0.000s
I20260709 13:02:35.627271  5172 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:35.596812  1738 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:35.624764  5214 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:35.655189  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 109954us
I20260709 13:02:35.658723  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 513) took 1065 ms. Trace:
I20260709 13:02:35.659267  1737 rpcz_store.cc:276] 0709 13:02:34.592931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.301863 (+708932us) service_pool.cc:224] Handling call
0709 13:02:35.302350 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.313808 (+ 11458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.313840 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.315023 (+  1183us) log.cc:844] Serialized 26 byte log entry
0709 13:02:35.391541 (+ 76518us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:35.391583 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.391673 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.658417 (+266744us) raft_consensus.cc:1685] finished
0709 13:02:35.658450 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.658638 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24081,"raft.queue_time_us":62283,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":12684,"thread_start_us":37328,"threads_started":1,"wal-append.queue_time_us":156363}
I20260709 13:02:35.649546  4974 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:35.677991  1740 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:02:35.665370  5142 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:35.685652  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.676393  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:35.690050  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 513) took 2074 ms. Trace:
W20260709 13:02:35.691486  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.690477  1660 rpcz_store.cc:276] 0709 13:02:33.616086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:33.616236 (+   150us) service_pool.cc:224] Handling call
0709 13:02:33.616524 (+   288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:33.617625 (+  1101us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:33.617656 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:33.624170 (+  6514us) log.cc:844] Serialized 26 byte log entry
0709 13:02:33.849975 (+225805us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:33.850030 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:33.850124 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:35.641430 (+1791306us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:35.683259 (+ 41829us) raft_consensus.cc:1685] finished
0709 13:02:35.683292 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:35.689955 (+  6663us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":653,"raft.queue_time_us":1635,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":934,"spinlock_wait_cycles":27505920,"thread_start_us":170497,"threads_started":1,"wal-append.queue_time_us":288213}
W20260709 13:02:35.687963  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 162151us
I20260709 13:02:35.689646  1590 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:35.696889  1586 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 9. Ops: []
I20260709 13:02:35.680181  5169 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 13:02:35.687038  4965 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.678794  5237 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 13:02:35.708855  5137 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.710043  4941 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:35.711251  4941 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:35.703656  4737 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.711664  4941 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:35.750600  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:35.722308  5217 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.849s	user 0.002s	sys 0.011s
I20260709 13:02:35.715767  5041 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 13:02:35.757032  5217 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.759589  1734 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:35.753695  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45801.1us
I20260709 13:02:35.712625  4941 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.745568  5169 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 13:02:35.750633  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:35.798537  1634 leader_election.cc:400] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:02:35.737154  4969 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.728523  5140 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.928853  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:35.928161  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:35.932312  1741 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.899472  2318 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.699903  5267 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:35.931420  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:35.935876  1634 leader_election.cc:403] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 13:02:35.812112  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:35.936631  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.942737  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:35.944205  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:35.943473  2310 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:35.895700  5275 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 13:02:35.942200  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 143908us
I20260709 13:02:35.956096  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:35.948565  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:35.948534  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:35.884625  5231 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.737s	user 0.013s	sys 0.001s
W20260709 13:02:35.886554  5159 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.744s	user 0.009s	sys 0.004s
W20260709 13:02:35.852324  5154 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.916s	user 0.019s	sys 0.000s
I20260709 13:02:35.954262  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:35.857218  5155 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:35.960054  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:35.805701  4977 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:35.959267  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:36.011422  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52223us
I20260709 13:02:35.964526  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.013167  2318 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:36.014112  1738 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:35.886745  5094 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:35.803949  4965 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:35.934283  5112 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:35.942046  5226 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:35.974992  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:35.978047  5217 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:35.978798  4941 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:35.976929  5169 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:35.980012  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 476) took 3139 ms. Trace:
I20260709 13:02:35.980398  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:35.980535  5247 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:35.980991  5239 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:02:35.977967  5137 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:35.712774  4737 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.967933  5140 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:35.990228  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:35.985368  4969 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:35.935534  5143 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 13:02:35.993901  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:36.045819  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.046643  2311 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:35.804086  4939 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:35.995476  1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2389) took 1881 ms (client timeout 1556 ms). Trace:
I20260709 13:02:35.959146  5267 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:35.857652  5248 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.905406  4826 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:35.947045  5237 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:02:35.947536  5146 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.087378  5146 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:35.999727  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:36.091145  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91491.5us
W20260709 13:02:36.093696  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.003257  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:36.094223  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2316) took 3159 ms. Trace:
I20260709 13:02:36.029791  5122 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 13:02:36.094724  1584 rpcz_store.cc:276] 0709 13:02:32.934641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.183605 (+1248964us) service_pool.cc:224] Handling call
0709 13:02:35.995454 (+1811849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.995715 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.995756 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:35.995815 (+    59us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:35.995862 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.995933 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.094023 (+ 98090us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400431050. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:36.094153 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4166413568}
I20260709 13:02:36.006538  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 755) took 1220 ms. Trace:
I20260709 13:02:36.001977  1741 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:36.096509  5280 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:02:36.096247  2314 rpcz_store.cc:276] 0709 13:02:34.786681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.945437 (+158756us) service_pool.cc:224] Handling call
0709 13:02:35.979020 (+1033583us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:36.006468 (+ 27448us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678519552}
I20260709 13:02:36.034194  4685 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.097401  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.097961  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.097925  4685 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.098608  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:36.098783  2314 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:36.100293  2314 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:36.100459  4685 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:35.967305  5221 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.102133  5221 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.102777  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:36.103660  1663 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:36.006114  5154 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.072s	user 0.021s	sys 0.000s
I20260709 13:02:36.104099  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1704) took 1008 ms. Trace:
I20260709 13:02:36.104481  1663 rpcz_store.cc:276] 0709 13:02:35.095293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.102479 (+1007186us) service_pool.cc:224] Handling call
0709 13:02:36.104034 (+  1555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:36.023155  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:36.105309  1736 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:02:36.026093  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.106145  2315 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.001045  5231 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.031466  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51175.2us
I20260709 13:02:36.007894  5155 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:36.073642  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79874.4us
W20260709 13:02:36.073958  1587 rpcz_store.cc:269] 0709 13:02:34.114179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.139891 (+ 25712us) service_pool.cc:224] Handling call
0709 13:02:35.995261 (+1855370us) spinlock_profiling.cc:243] Waited 54 ms on lock 0x7b5400431054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:35.995396 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125174912}
I20260709 13:02:36.034525  4953 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:36.028084  5172 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:36.080163  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.018654  4808 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:36.083590  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.137131  1738 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:36.138355  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.139071  2314 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:36.141963  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:36.142489  4906 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.143052  4906 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.144104  5214 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.146699  5077 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:36.091233  4956 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:36.147257  5077 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.147747  5077 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.147778  4956 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.030547  4965 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.150399  4685 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:36.150866  4685 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:36.151309  4965 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:36.151991  4685 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.000932  4325 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.153787  4325 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.154505  5281 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.059s	user 0.003s	sys 0.000s
I20260709 13:02:36.034915  2309 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:36.156106  5172 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:36.156740  4325 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.110482  5270 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:36.003362  1592 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: [6.7-7.8]
I20260709 13:02:36.030331  1588 rpcz_store.cc:276] 0709 13:02:32.841054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.097020 (+1255966us) service_pool.cc:224] Handling call
0709 13:02:34.829072 (+732052us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:34.867971 (+ 38899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:35.979767 (+1111796us) spinlock_profiling.cc:243] Waited 732 ms on lock 0x7b540025e250. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:35.979940 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1682752768}
I20260709 13:02:36.108028  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.162166  1584 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:36.034098  5185 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.076157  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.163533  2316 raft_consensus.cc:2376] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9. Re-sending same reply.
I20260709 13:02:36.014070  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 757) took 1208 ms. Trace:
I20260709 13:02:36.111099  4971 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.165686  2318 rpcz_store.cc:276] 0709 13:02:34.805534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.920009 (+114475us) service_pool.cc:224] Handling call
0709 13:02:35.899193 (+979184us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b58005eeed0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfe127c 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c
0709 13:02:36.013997 (+114804us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":425384320}
I20260709 13:02:36.166762  4826 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.112216  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:36.167303  5274 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.196s	user 0.002s	sys 0.001s
W20260709 13:02:36.168072  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55939.9us
I20260709 13:02:36.168408  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:36.118057  2317 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:36.170399  5155 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.163097  5040 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:36.171505  5040 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.171789  5155 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.172492  5040 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.194541  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:36.134029  1736 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:36.135037  1660 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:36.135597  1739 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:36.018343  4977 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.146117  5267 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.157094  5172 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.074553  5226 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:36.175575  5231 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.179250  1588 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
W20260709 13:02:36.180307  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.180723  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.235945  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.236176  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.236845  1586 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:36.184531  1742 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:36.116577  1741 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:36.186116  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.187089  1589 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:36.188593  1585 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:36.210013  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:36.210848  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.212374  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:36.081300  5266 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:36.241618  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.242281  1588 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.230707  2315 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:36.094225  5041 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.094s
I20260709 13:02:36.029055  5159 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 13:02:36.169378  5239 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:36.248369  4325 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:36.264719  1588 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:36.287263  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.289044  1588 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:36.213837  5221 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:36.244549  5253 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 13:02:36.189116  4800 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.149008  4956 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.234117  5226 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.300024  1588 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:36.163136  5185 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.291077  5160 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:36.248441  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:36.294196  5281 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:02:36.322775  5226 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.328155  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79803.4us
I20260709 13:02:36.325240  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.331369  1588 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.244688  5214 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:36.352186  4325 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:36.353816  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 185546us
I20260709 13:02:36.325558  5160 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.245752  5122 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.100s
I20260709 13:02:36.374549  1588 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:36.376135  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.377758  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.259390  5231 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 13:02:36.347472  5143 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:36.313575  4800 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.398277  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 516) took 1738 ms. Trace:
I20260709 13:02:36.399053  1733 rpcz_store.cc:276] 0709 13:02:34.660079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.339916 (+679837us) service_pool.cc:224] Handling call
0709 13:02:35.340219 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.895418 (+555199us) spinlock_profiling.cc:243] Waited 3.26 ms on lock 0x7b18003b6680. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfd5bfb 00007fcdedfe064b
0709 13:02:35.895600 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:35.895633 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:35.896268 (+   635us) log.cc:844] Serialized 26 byte log entry
0709 13:02:36.077897 (+181629us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:36.078298 (+   401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.078408 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:36.397788 (+319380us) raft_consensus.cc:1685] finished
0709 13:02:36.397822 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:36.398176 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19176,"raft.queue_time_us":44862,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6227,"spinlock_wait_cycles":7491328,"thread_start_us":212065,"threads_started":2,"wal-append.queue_time_us":404167}
I20260709 13:02:36.405536  4939 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.409376  4977 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:36.411517  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 260452us
I20260709 13:02:36.413149  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:36.415197  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.419128  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:36.372222  5160 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.422742  1588 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:36.428128  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:36.431885  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.434710  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.383821  5146 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.442276  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.425825  4984 raft_consensus.cc:2764] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 13:02:36.452380  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.450713  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:36.475934  2315 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:02:36.462797  1588 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:36.426873  5143 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:36.440476  4969 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:36.478430  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.440709  5217 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:36.481552  1588 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:36.483223  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:36.484467  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 512) took 1892 ms. Trace:
I20260709 13:02:36.484743  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.485963  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:36.485034  1735 rpcz_store.cc:276] 0709 13:02:34.592571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.285910 (+693339us) service_pool.cc:224] Handling call
0709 13:02:35.336547 (+ 50637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:35.337098 (+   551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:36.481986 (+1144888us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b54002030d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:36.484388 (+  2402us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115789696}
I20260709 13:02:36.489688  1706 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:02:36.490260  1706 leader_election.cc:403] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:36.486358  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:36.453858  5248 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:36.504904  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.513473  5159 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:36.514151  5159 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.514601  5159 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.502269  5254 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:02:36.495307  4826 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 13:02:36.503021  4812 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.519101  4812 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.521086  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:36.521999  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:36.524428  1588 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 13:02:36.524575  5106 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.545938  5106 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.546665  1588 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 10
I20260709 13:02:36.529064  1593 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
W20260709 13:02:36.531432  5286 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 13:02:36.533870  5288 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:02:36.568640  4984 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.534863  5143 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.540251  1592 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20260709 13:02:36.536690  4953 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:36.572562  4953 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.574823  4953 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.575275  4734 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:36.576035  4953 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:36.549410  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.444005  5094 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:02:36.504166  5231 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:36.568326  5106 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:36.559301  1593 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:36.580876  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.595311  1593 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 13:02:36.583199  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.607959  1665 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:36.584203  1588 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:36.569327  4984 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.584589  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.585383  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:36.545622  4800 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.581140  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:36.554629  5248 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.617187  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.548830  4969 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.622987  2316 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:36.624795  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:36.627549  2317 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:02:36.585410  5146 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:36.608961  5239 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.667447  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:36.669121  1665 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:36.671437  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:36.607458  4974 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 13:02:36.641570  5231 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:36.587786  4953 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.643278  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:36.702883  2312 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:36.706775  2312 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:36.707870  2312 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.665889  5297 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:36.651666  4969 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.641336  5248 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.701841  5295 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:36.695358  5185 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.719192  1665 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:36.700591  4956 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:36.719229  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:36.650975  5291 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.071s
I20260709 13:02:36.718952  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:36.732645  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.730089  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:36.733405  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 155728us
I20260709 13:02:36.735787  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:36.736334  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3385369149999633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1501s for a response
W20260709 13:02:36.737138  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3196369149999896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260709 13:02:36.726094  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:36.737470  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3070369149999692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260709 13:02:36.738436  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1072369149999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
I20260709 13:02:36.739523  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:36.742303  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0415369149999947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1831s for a response
W20260709 13:02:36.742865  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0199369149999598s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20260709 13:02:36.743247  1557 connection.cc:256] RPC call timeout handler was delayed by -1.012736915000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
I20260709 13:02:36.743300  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:36.669652  5146 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.746181  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:36.748291  1659 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 13:02:36.748894  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:36.772837  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 327228us
I20260709 13:02:36.745801  2309 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:36.724514  5295 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.782541  2318 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.746184  5106 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.746347  4947 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.746021  4956 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.835942  2318 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:36.746006  5221 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:36.809901  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:36.771318  5185 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:36.796623  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:36.747187  5239 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.846047  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.846889  1736 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:36.776818  4941 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.739s	user 0.004s	sys 0.009s
I20260709 13:02:36.839237  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.835490  4974 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.850767  1740 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:36.839493  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:36.852057  1735 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.839883  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.778684  5267 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.727298  5297 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.846207  5077 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.698s	user 0.017s	sys 0.000s
I20260709 13:02:36.815835  5294 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:36.772743  4965 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 13:02:36.830180  5303 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 13:02:36.839823  5143 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:36.779588  5231 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.845711  5185 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:36.833913  5106 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.855119  4906 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.001s
I20260709 13:02:36.867801  4906 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:36.869796  5300 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 13:02:36.870059  4906 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.868052  4969 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.847276  5307 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:36.846897  5221 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.856009  4974 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.845191  4947 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.858928  5267 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.883642  5185 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.884763  5185 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:36.875721  5307 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:36.886240  5307 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:36.886564  4974 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.886643  5190 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:36.887236  5190 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:36.864831  5143 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.888284  5190 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.889421  5143 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.891211  5297 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.860432  5077 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:36.891845  5282 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.802s	user 0.003s	sys 0.000s
I20260709 13:02:36.893095  5221 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.893673  5248 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:36.893954  5077 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:36.894559  5140 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:36.894958  5140 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:36.895293  5146 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.895720  5106 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:36.895793  5140 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.875562  5160 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.898933  5140 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:36.858158  5295 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:36.861371  5294 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:36.910887  5295 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:36.913848  5294 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:36.952065   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:02:36.846767  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50232.1us
I20260709 13:02:36.979624  1661 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:36.782451  1664 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:36.983752  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:36.854733  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:36.859928  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:36.844861  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:37.005401  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:36.902985  5190 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:36.900907  5155 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.729s	user 0.007s	sys 0.007s
I20260709 13:02:37.029374  2315 raft_consensus.cc:2368] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:36.877668  4953 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:37.030237  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:36.880755  5159 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.049896  5077 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:37.050702  5267 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.052079  5248 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:02:37.053776  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.046243  5307 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.055325  5261 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:36.900358  5160 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:37.066144  5307 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:37.065650  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:37.074450  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.077108  5231 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:37.056784  5261 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.094909  5249 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:37.098377  5297 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:37.096760  5249 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 13:02:37.118692  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.065788  4956 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.105703  5297 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:37.126899  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97935.6us
I20260709 13:02:37.127882  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.127803  5115 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 13:02:37.129068  5146 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.130105  1667 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:37.132247  1666 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:37.139571  1667 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:37.129475  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.130577  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 523) took 1203 ms. Trace:
I20260709 13:02:37.131848  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:37.154115  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 273191us
I20260709 13:02:37.133296  5267 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:37.169092  4947 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:37.172742  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.173017  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.169167  5267 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:37.176684  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47415.8us
I20260709 13:02:37.178017  5143 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:37.178228  5267 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.179404  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2342) took 1871 ms. Trace:
I20260709 13:02:37.179944  1742 rpcz_store.cc:276] 0709 13:02:35.307419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.436782 (+129363us) service_pool.cc:224] Handling call
0709 13:02:35.437150 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.237345 (+800195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.178603 (+941258us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:37.179043 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1718457728}
I20260709 13:02:37.181366  1557 leader_election.cc:400] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:37.181897  1557 leader_election.cc:403] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:37.187340  5261 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.160616  1660 rpcz_store.cc:276] 0709 13:02:35.926810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.926985 (+   175us) service_pool.cc:224] Handling call
0709 13:02:37.130135 (+1203150us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:37.130387 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":386105600}
I20260709 13:02:37.201719  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2345) took 1016 ms. Trace:
I20260709 13:02:37.203096  1557 leader_election.cc:400] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:37.203536  2310 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:37.205029  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:37.129911  5146 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:37.209409  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.584s (SENT) [suppressed 18 similar messages]
I20260709 13:02:37.209646  1589 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.211393  5160 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.210736  1667 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.216087  5261 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:37.169169  5106 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:37.227365  2311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 753) took 2467 ms (client timeout 1947 ms). Trace:
I20260709 13:02:37.222874  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.130788  5231 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.218657  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:37.226485  5160 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.227759  5106 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.237978  5143 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.239161  5106 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.239610  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:37.240444  5261 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:37.238922  5143 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.230990  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.202476  1591 rpcz_store.cc:276] 0709 13:02:36.185585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.185732 (+   147us) service_pool.cc:224] Handling call
0709 13:02:36.234292 (+ 48560us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:36.234761 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.234792 (+    31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:36.258510 (+ 23718us) log.cc:844] Serialized 78 byte log entry
0709 13:02:36.586396 (+327886us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:36.629808 (+ 43412us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.076899 (+447091us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b180018a900. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:02:37.114031 (+ 37132us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.192313 (+ 78282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.199357 (+  7044us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.201221 (+  1864us) raft_consensus.cc:1685] finished
0709 13:02:37.201237 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.201469 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":407232,"prepare.queue_time_us":539106,"prepare.run_cpu_time_us":163,"prepare.run_wall_time_us":4471,"raft.queue_time_us":30284,"raft.run_cpu_time_us":551,"raft.run_wall_time_us":33592,"spinlock_wait_cycles":4074752,"thread_start_us":265107,"threads_started":1,"wal-append.queue_time_us":427679}
I20260709 13:02:37.241199  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.239300  4812 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.240559  2314 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
W20260709 13:02:37.227911  2311 rpcz_store.cc:269] 0709 13:02:34.759318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.760225 (+   907us) service_pool.cc:224] Handling call
0709 13:02:37.226864 (+2466639us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b5400724054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:37.227047 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2718297856}
I20260709 13:02:37.150502  5297 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.241115  5159 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.267004  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.268916  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:37.270709  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65911us
I20260709 13:02:37.272058  1557 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:37.272435  1557 leader_election.cc:403] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:37.274014  5326 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:37.274657  5106 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:37.275099  5106 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:37.275652  5106 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.278420  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2354) took 1026 ms. Trace:
I20260709 13:02:37.278708  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:37.279548  1588 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.278995  1592 rpcz_store.cc:276] 0709 13:02:36.251465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.574154 (+322689us) service_pool.cc:224] Handling call
0709 13:02:36.574513 (+   359us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:36.574966 (+   453us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.574998 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:36.575784 (+   786us) log.cc:844] Serialized 78 byte log entry
0709 13:02:36.787912 (+212128us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:37.218834 (+430922us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.261536 (+ 42702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.277789 (+ 16253us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:37.277894 (+   105us) raft_consensus.cc:1685] finished
0709 13:02:37.277906 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.278222 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":289080,"prepare.queue_time_us":289824,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":3121,"raft.queue_time_us":5034,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":287,"spinlock_wait_cycles":2944,"thread_start_us":214615,"threads_started":2,"wal-append.queue_time_us":268322}
I20260709 13:02:37.283577  5143 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:37.283977  5160 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.284910  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:37.285097  1587 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:37.286077  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.286500  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.286990  1591 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:37.287977  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:37.288762  1587 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:37.288834  1591 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:37.289175  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:37.289978  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:37.290789  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.290894  1667 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.292747  5231 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:37.294371  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.304186  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.271103  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.252620  5261 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.298017  5106 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.256727  5249 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:37.309718  5249 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.275180  5326 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:37.294950  5146 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.314575  5324 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:37.315697  4967 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:37.315717  5146 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:37.316529  5146 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.317045  5146 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:37.316690  4967 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.321130  5249 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.321169  5261 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.322798  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.328341  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.329267  1660 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:37.329483  5214 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.330094  5214 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.330757  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.331489  1666 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:37.333439  4974 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:37.333984  4974 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:37.335054  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.335472  4974 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:37.342298  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48264.2us
I20260709 13:02:37.343727  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.324757  5297 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:37.345634  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:37.345628  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136466us
I20260709 13:02:37.307312  4812 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.356998  1588 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:37.358744  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:37.352141  4974 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:37.375670  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.378863  1659 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:37.388605  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:37.403748  4906 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:37.404413  4906 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.404760  4906 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:37.391124  5110 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:37.432482  1592 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:37.435457  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76803.3us
W20260709 13:02:37.442745  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99366.4us
I20260709 13:02:37.446048  1591 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:02:37.458844  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.413173  4812 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.439903  4967 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:37.461546  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.457129  1735 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.511s	user 0.001s	sys 0.018s
W20260709 13:02:37.465914  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.466033  1588 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:37.467818  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.477399  4812 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:37.481446  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.487808  2314 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:37.482963  5140 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.008s
I20260709 13:02:37.493028  5270 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.499117  5270 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:37.501696  5319 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:37.502506  5319 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:37.503918  5319 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.507864  1735 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:37.509702  5106 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:37.511075  5295 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.013s
I20260709 13:02:37.483297  2312 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:37.516983  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:37.519362  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 119100us
I20260709 13:02:37.523638  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.524994  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:37.525117  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.526047  4910 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:37.526324  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.526939  4910 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:37.527426  4910 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.527340  5182 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:37.527925  4910 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:37.528489  5182 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:37.528880  5182 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:37.530164  5182 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.531503  5182 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.532109  2314 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:37.537319  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1702) took 2442 ms. Trace:
I20260709 13:02:37.537890  1662 rpcz_store.cc:276] 0709 13:02:35.094634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.260558 (+165924us) service_pool.cc:224] Handling call
0709 13:02:35.260881 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:35.430606 (+169725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.536688 (+2106082us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:37.537088 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388063488}
I20260709 13:02:37.540432  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.488467  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:37.547256  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.551003  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80083.5us
W20260709 13:02:37.559170  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.558342  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.569569  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 765) took 1625 ms. Trace:
I20260709 13:02:37.571257  2311 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:37.574354  1589 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:37.570140  2310 rpcz_store.cc:276] 0709 13:02:35.944424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.948263 (+  3839us) service_pool.cc:224] Handling call
0709 13:02:37.569267 (+1621004us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54003ee4d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:37.569467 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059462400}
I20260709 13:02:37.583515  5226 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:37.594722  2312 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:02:37.591490  1708 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.556s (SENT)
I20260709 13:02:37.598572  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:37.586182  5214 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.601469  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:37.605669  4906 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.607339  4984 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.608501  4906 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:37.609091  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 525) took 1680 ms (client timeout 1576 ms). Trace:
W20260709 13:02:37.609560  1668 rpcz_store.cc:269] 0709 13:02:35.928383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.928544 (+   161us) service_pool.cc:224] Handling call
0709 13:02:37.608557 (+1680013us) spinlock_profiling.cc:243] Waited 813 ms on lock 0x7b54002f1a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:37.608785 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1869012736}
I20260709 13:02:37.620995  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.601025  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.631589  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:37.632011  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52218.8us
W20260709 13:02:37.634145  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 163166us
I20260709 13:02:37.637753  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:37.638660  2311 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
W20260709 13:02:37.645128  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48413.7us
I20260709 13:02:37.638684  4906 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:37.617612  5146 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.661918  5077 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.664621  5077 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.666126  5146 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.667019  5292 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:37.667311  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 351) took 1086 ms. Trace:
I20260709 13:02:37.667719  5292 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.668133  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 664) took 3565 ms. Trace:
I20260709 13:02:37.667821  2315 rpcz_store.cc:276] 0709 13:02:36.580751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.580893 (+   142us) service_pool.cc:224] Handling call
0709 13:02:37.667068 (+1086175us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540001c554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:37.667251 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":890568064}
I20260709 13:02:37.668689  2312 rpcz_store.cc:276] 0709 13:02:34.103125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.591198 (+488073us) service_pool.cc:224] Handling call
0709 13:02:34.596087 (+  4889us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:37.482326 (+2886239us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b58008ac2d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:02:37.595300 (+112974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.667803 (+ 72503us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54003ef154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:37.668044 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6272688128}
I20260709 13:02:37.663908  4812 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:37.662745  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:37.662189  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:37.673696  1557 leader_election.cc:400] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7.
I20260709 13:02:37.673839  2310 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:37.684680  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.662220  4906 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:37.686178  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:37.686975  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:37.691138  1668 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:37.691224  1589 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:37.692067  5038 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 13:02:37.692575  2312 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:37.693358  5077 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:37.685672  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.697849  4906 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:37.710137  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46894.9us
I20260709 13:02:37.711457  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.712491  5173 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:02:37.714691  5106 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.715426  5106 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.718261  5106 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.719913  2310 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:37.721154  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2354) took 1778 ms. Trace:
I20260709 13:02:37.721588  1739 rpcz_store.cc:276] 0709 13:02:35.943024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:35.943168 (+   144us) service_pool.cc:224] Handling call
0709 13:02:35.943433 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.197538 (+254105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.720669 (+1523131us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400338e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:37.720933 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":441481600}
I20260709 13:02:37.729637  1587 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:37.735453  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.737952  5106 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:37.745636  4812 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.746543  4812 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.747334  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.748838  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:37.749090  4812 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:37.750730  5319 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:37.751351  5319 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:37.752354  5319 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.753742  5182 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:37.754653  5326 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:37.727111  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:37.737869  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.808014  1668 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:37.808877  2314 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:37.766785  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1707) took 1041 ms. Trace:
I20260709 13:02:37.770601  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:37.768596  1662 raft_consensus.cc:1217] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Deduplicated request from leader. Original: 6.7->[7.8-7.8]   Dedup: 7.8->[]
I20260709 13:02:37.814749  1662 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 13:02:37.818226  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69505.8us
W20260709 13:02:37.821000  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.762950  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2418) took 1585 ms. Trace:
I20260709 13:02:37.702028  5297 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:37.825731  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 237000us
I20260709 13:02:37.828109  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:37.828279  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 117050us
W20260709 13:02:37.759146  5339 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
W20260709 13:02:37.833011  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 151632us
I20260709 13:02:37.837271  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.835928  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.758337  5292 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:37.841323  1667 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:37.851257  4685 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.822961  4906 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.841956  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.842619  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.720863  5270 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:37.843006  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.847095  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:37.845773  1587 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:37.840718  2310 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:37.861812  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:37.826946  5297 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:37.846951  4325 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.846926  5214 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:37.870252  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 143269us
I20260709 13:02:37.845847  5294 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:37.810945  1665 rpcz_store.cc:276] 0709 13:02:36.725407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.768604 (+ 43197us) service_pool.cc:224] Handling call
0709 13:02:36.769166 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.986428 (+217262us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:37.035066 (+ 48638us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.261467 (+226401us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:37.261562 (+    95us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:37.261603 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.766355 (+504752us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:37.766432 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.766587 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6796,"prepare.queue_time_us":120830,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":3269,"raft.queue_time_us":15435,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":9098,"spinlock_wait_cycles":497706368,"thread_start_us":243595,"threads_started":2,"wal-append.queue_time_us":273883}
I20260709 13:02:37.868362  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:37.882025  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.877982  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:37.896231  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.823230  1590 rpcz_store.cc:276] 0709 13:02:36.177761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.182530 (+  4769us) service_pool.cc:224] Handling call
0709 13:02:36.182913 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.183213 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:36.217118 (+ 33905us) log.cc:844] Serialized 14 byte log entry
0709 13:02:36.532946 (+315828us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b18002db140. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:02:37.762012 (+1229066us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b18002db140. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:37.762287 (+   275us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:37.762378 (+    91us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:37.762452 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.762562 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.762725 (+   163us) inbound_call.cc:177] Queueing success response
0709 13:02:37.819731 (+ 57006us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b4800080358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
Metrics: {"prepare.queue_time_us":33789,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":27818,"raft.queue_time_us":330409,"raft.run_cpu_time_us":3670,"raft.run_wall_time_us":700160,"spinlock_wait_cycles":61764224,"thread_start_us":391993,"threads_started":2,"wal-append.queue_time_us":645528}
I20260709 13:02:37.884645  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:37.907291  1663 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:37.920842  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:37.867336  4967 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:37.839999  4974 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:37.872649  5297 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.907904  5106 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:37.929204  5267 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:37.931655  4906 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:37.931813  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:37.947125  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 93696.7us
I20260709 13:02:37.904196  5326 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:37.952268  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61400.3us
W20260709 13:02:37.956944  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88729.6us
W20260709 13:02:37.959270  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116440us
I20260709 13:02:37.948180  4325 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:37.927635  5294 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:37.984622  1584 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:37.985576  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103689us
I20260709 13:02:37.987229  4812 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:37.988134  4812 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:37.992427  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.975656  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:38.006816  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.009887  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.018260  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:37.950933  5038 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.961760  5292 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:37.939802  4910 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.030290  5021 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.030591  5241 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:38.024070  5106 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:38.033648  1480 maintenance_manager.cc:419] P e6b16382849d482ba544fc5ee58e0b0a: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033359
I20260709 13:02:38.034226  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:37.992786  5214 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:38.027459  5292 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:38.021428  5338 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:02:38.042821  4965 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:02:38.027962  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45500.7us
I20260709 13:02:37.975973  5110 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:37.965654  4974 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.063596  5038 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.098757  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.083277  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2350) took 1628 ms. Trace:
I20260709 13:02:38.083683  4325 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.045797  5146 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:38.100752  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.107172  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.102591  4325 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 13:02:38.116060  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.059098  4984 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:38.019549  5326 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.075505  5214 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:38.096712  5292 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:38.103329  1586 rpcz_store.cc:276] 0709 13:02:36.250316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.541539 (+291223us) service_pool.cc:224] Handling call
0709 13:02:36.541936 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:36.542182 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:36.542597 (+   415us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.741917 (+1199320us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b18003883c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:37.742643 (+   726us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.768833 (+ 26190us) log.cc:844] Serialized 14 byte log entry
0709 13:02:37.823419 (+ 54586us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:37.829818 (+  6399us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:37.836826 (+  7008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:37.842103 (+  5277us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18003883c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:37.855937 (+ 13834us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:37.878098 (+ 22161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":194566,"prepare.queue_time_us":16020,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":3112,"raft.queue_time_us":335355,"raft.run_cpu_time_us":595,"raft.run_wall_time_us":49551,"spinlock_wait_cycles":30513152,"thread_start_us":390969,"threads_started":2,"wal-append.queue_time_us":312018}
I20260709 13:02:38.141710  1741 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:38.135500  1589 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:38.146035  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:38.117388  5319 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:38.170405  1591 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:38.071516  5021 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.138062  5214 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:38.113075  5106 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:38.192701  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.197248  5122 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:38.211238  5038 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.211922  5294 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.214119  4974 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.214788  5118 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:38.215478  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:38.215780  4812 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.216176  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.219355  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.219462  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.219736  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:38.219436  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.222967  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.224134  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:38.224514  5343 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.313s	user 0.000s	sys 0.001s
I20260709 13:02:38.224892  5266 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:38.226559  5270 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:38.228076  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 775) took 1793 ms. Trace:
I20260709 13:02:38.188580  1585 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:02:38.121884  5146 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.168752  4984 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:38.229041  5326 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.264245  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:38.286062  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:38.175854  5110 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:38.291415  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 304758us
I20260709 13:02:38.294227  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.257727  2313 rpcz_store.cc:276] 0709 13:02:36.434626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.445343 (+ 10717us) service_pool.cc:224] Handling call
0709 13:02:36.478835 (+ 33492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:36.503519 (+ 24684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:36.503577 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:36.508629 (+  5052us) log.cc:844] Serialized 26 byte log entry
0709 13:02:37.748187 (+1239558us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b1800366a60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:38.136194 (+388007us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:38.136267 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.175811 (+ 39544us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:38.227519 (+ 51708us) raft_consensus.cc:1685] finished
0709 13:02:38.227543 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:38.227857 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24132,"raft.queue_time_us":25563,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3085,"spinlock_wait_cycles":5183744,"thread_start_us":370712,"threads_started":1,"wal-append.queue_time_us":1716636}
I20260709 13:02:38.287572  1736 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:38.299186  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.299664  1733 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:38.300133  1734 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:38.272222  5266 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:38.316422  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.294322  5326 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:38.294364  4910 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.327981  1584 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:38.297904  5351 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.290874  4984 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:38.344060  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.323629  4967 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:38.328177  5326 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:38.332067  5297 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.353806  5326 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:38.452343  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:38.359303  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47167us
W20260709 13:02:38.438666  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 352) took 1857 ms (client timeout 1771 ms). Trace:
W20260709 13:02:38.409567  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 262591us
W20260709 13:02:38.354712  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 333486us
W20260709 13:02:38.408519  2314 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.595s	user 0.007s	sys 0.003s
I20260709 13:02:38.458616  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:38.482133  2314 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:38.395004  1735 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:02:38.364729  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:38.405627  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 429672us
W20260709 13:02:38.353974  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116084us
W20260709 13:02:38.459209  5346 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 13:02:38.410115  1742 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:02:38.460045  5348 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
W20260709 13:02:38.481139  2318 rpcz_store.cc:269] 0709 13:02:36.581135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.582851 (+  1716us) service_pool.cc:224] Handling call
0709 13:02:38.438166 (+1855315us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54007f0654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:38.438392 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":426314496}
I20260709 13:02:38.465269  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.397929  2310 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:38.405839  1737 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:38.462594  5349 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 13:02:38.495064  1666 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:38.495122  4984 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.448021  5146 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:38.437659  5355 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:02:38.497944  5352 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.522305  1734 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:38.498342  5267 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.552s	user 0.013s	sys 0.000s
I20260709 13:02:38.350282  4685 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.460235  5356 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:38.458145  2311 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:38.386466  4974 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:38.569084  1589 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:38.390697  4977 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.571110  4977 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.572431  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:38.364729  5266 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.574379  5352 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.576768  5352 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:38.409477  1490 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:38.463362  5021 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.435791  5040 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:38.439309  5106 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.398039  5350 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.561365  2312 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:38.595019  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.565191  1592 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:38.554183  5356 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:38.566771  5297 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.502297  5110 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:38.589628  1584 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:38.353181  5351 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.590579  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.591492  4965 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:38.621210  4965 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.622044  5356 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.623200  5356 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:38.623296  5352 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:38.623486  4965 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:38.602113  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.604044  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:38.619447  5363 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:38.620278  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:38.627557  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138848us
I20260709 13:02:38.410063  4967 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.651175  1739 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:38.652462  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 199292us
I20260709 13:02:38.654405  1589 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:38.660815  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.662102  5352 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:38.662735  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.663792  5352 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:38.666172  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:38.577553  4685 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.673115  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.676592  2312 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:38.681404  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:38.682670  1592 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:38.684461  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.688292  1587 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:02:38.619041  5292 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.684155  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.699765  1584 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:38.629943  1490 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 2 to 5, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:38.678754  5363 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:38.607199  5350 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.692740  4965 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:38.715103  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.716244  2315 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:38.720639  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:38.721490  1738 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:38.721868  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.723485  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.724648  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.725997  2309 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:38.725225  1587 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:38.730919  4685 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.731050  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.732044  1587 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:38.737046  1587 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:38.692557  5349 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 13:02:38.738785  1587 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:38.739156  1587 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:38.693532  5110 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:38.716737  1490 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 5, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 5 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 13:02:38.748198  5214 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.506s	user 0.009s	sys 0.003s
I20260709 13:02:38.748769  5214 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.752682  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.760586  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.750560  4685 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.765136  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:38.762800  1736 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:38.766589  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:38.767125  1584 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:38.768124  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:38.768247  1584 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:38.771149  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.771925  1738 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:38.775368  5363 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.771750  1587 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:38.783144  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 188232us
I20260709 13:02:38.772225  5266 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:38.786232  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.770474  4965 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:38.787422  5140 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:38.826807  5351 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:38.790233  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:38.791679  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75960.6us
I20260709 13:02:38.779407  5349 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 13:02:38.792737  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.772172  5358 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:02:38.781718  5374 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:38.794695  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.795917  4974 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 13:02:38.799002  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.868860  1705 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:38.802520  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:38.803884  5241 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:38.802677  5214 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.772385  4685 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:38.808120  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 294015us
I20260709 13:02:38.808379  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.870862  1490 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 6 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:38.884814  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.808220  5326 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.808733  5267 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:38.789304  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.890413  5241 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.752645  5110 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:38.821244  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.823302  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.829164  1737 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:38.788707  5350 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:38.832504  1733 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:38.862717  1742 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:38.865023  1587 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:38.866259  1588 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:38.782164  5292 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.869169  1590 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 13:02:38.869365  1705 leader_election.cc:403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
W20260709 13:02:38.869076  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.873481  1663 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 13:02:38.877527  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.878854  1661 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 13:02:38.880036  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.868661  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.830425  5266 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:38.883440  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81121.5us
I20260709 13:02:38.885459  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
W20260709 13:02:38.909602  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 88515.3us
W20260709 13:02:38.883450  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72381.3us
I20260709 13:02:38.910490  1706 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.883478  5351 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.886525  5122 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:38.902311  1662 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:38.905313  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:38.916906  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49560.8us
I20260709 13:02:38.917160  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:38.918767  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:38.919341  1660 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:38.919936  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.920719  1735 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:38.906124  1738 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:38.881372  5140 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.886797  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:38.916083  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:38.931596  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:38.884548  1659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "421876549c8f408587a97543338bca1d"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:02:38.933046  1659 raft_consensus.cc:606] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:38.912443  5266 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:38.933763  4966 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:38.911775  5226 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.934546  4966 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:38.934865  5226 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.935564  4966 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:38.937176  5226 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.937646  5214 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:38.937742  4906 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:38.938381  4984 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:38.933416  1659 raft_consensus.cc:614] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:38.972429  4685 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:38.973191  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.973326  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:38.937814  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:38.932201  1740 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 13:02:38.942834  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.943711  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.974058  1490 catalog_manager.cc:5697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 3, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 3 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:38.945243  1587 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 13:02:38.946481  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:38.946681  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79251.6us
I20260709 13:02:38.948594  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.940652  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:38.916031  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.978148  1663 raft_consensus.cc:2410] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:02:38.957119  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.959239  5379 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:38.950965  4325 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 13:02:38.994060  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.974567  1740 pending_rounds.cc:85] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 2
I20260709 13:02:38.969619  1664 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:38.970746  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:38.985718  5384 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:38.976150  1587 pending_rounds.cc:85] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 3
I20260709 13:02:38.988772  1490 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 5, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 5 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:38.990543  4734 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:38.991282  5038 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:02:38.993778  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 204553us
I20260709 13:02:38.999402  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.915266  4974 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:38.916033  5122 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:39.002432  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2371) took 1114 ms. Trace:
I20260709 13:02:39.002875  1591 rpcz_store.cc:276] 0709 13:02:37.879930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.880117 (+   187us) service_pool.cc:224] Handling call
0709 13:02:38.993706 (+1113589us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b5400442d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:38.993861 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538674560}
I20260709 13:02:39.003852  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.947706  5350 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:39.006518  5267 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:38.994868  1740 pending_rounds.cc:107] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 13:02:39.011059  1740 raft_consensus.cc:2889] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:39.011178  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:39.011895  1735 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.011999  1666 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 13:02:39.008852  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:39.010068  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.997155  1587 pending_rounds.cc:107] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 13:02:39.014387  1587 raft_consensus.cc:2889] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:39.017859  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:38.997131  4685 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:39.022799  1668 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:38.939910  5351 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.024884  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.006824  5023 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:39.040365  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 99818.8us
I20260709 13:02:39.003444  5140 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.041916  5270 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:39.005133  5297 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:02:39.018800  5261 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.008184  5350 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:38.998898  5379 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.065037  5379 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:39.067675  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.028622  4984 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:39.051571  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.019517  5214 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 13:02:39.015537  5358 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.041354  5372 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:39.064483  5350 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.001914  4974 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:39.079406  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 192765us
I20260709 13:02:39.086350  1736 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:39.018723  5267 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.090205  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65446.8us
I20260709 13:02:39.057440  5270 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.109604  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.040489  5247 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 13:02:39.054178  5023 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.114019  5353 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 13:02:39.077229  5352 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:39.115105  1490 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:02:39.127058  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111535us
I20260709 13:02:39.130893  1742 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:39.101910  4974 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:39.134939  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.135361  1742 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:02:39.090984  5214 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:39.139410  5214 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.136209  5372 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:39.143838  5372 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:39.144361  5372 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.145817  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.146588  4965 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:39.147389  4965 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.147944  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:39.148097  5290 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:39.148890  5290 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:39.149794  5297 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:39.075909  5266 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.116123  5270 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:39.159112  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.121212  5352 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:39.134334  5261 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.170789  1490 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:02:39.176039  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:39.176757  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.136451  4967 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.177075  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.176281  4906 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:39.182278  5358 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.182902  5358 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.183300  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.183961  1593 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.183369  1490 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 6, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 6 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:39.176457  5122 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.187727  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.188378  1587 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:39.184970  1490 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 4, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 4 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:02:39.191717  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.192430  1585 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:39.193516  4906 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:39.194219  1592 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:39.195183  1660 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:39.182925  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.197885  1591 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:39.199826  1735 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:39.203382  2313 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:39.203549  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.204263  1591 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:39.204583  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.205260  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.206010  1591 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:39.160467  5118 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.161620  5349 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.111373  5241 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:02:39.170149  5326 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:39.224917  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1718) took 2384 ms (client timeout 1960 ms). Trace:
W20260709 13:02:39.225342  1741 rpcz_store.cc:269] 0709 13:02:36.844487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:36.844622 (+   135us) service_pool.cc:224] Handling call
0709 13:02:39.224679 (+2380057us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400284954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:39.224813 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1741641472}
W20260709 13:02:39.232543  5385 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.162s	user 0.003s	sys 0.000s
I20260709 13:02:39.235085  5023 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.236805  5373 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:39.237540  5373 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:39.238862  5266 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.240365  1739 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:39.240547  5110 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:39.173903  5352 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:39.247823  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.248557  1591 raft_consensus.cc:2376] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10. Re-sending same reply.
I20260709 13:02:39.249495  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.250394  5140 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.201894  5358 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.253820  4685 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:39.254036  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77043.2us
W20260709 13:02:39.254786  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.258523  1587 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:39.250635  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.169417  5297 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:39.281188  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:39.209523  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:39.217375  4967 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.218488  5377 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260709 13:02:39.151053  4734 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 13:02:39.177460  5247 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:02:39.223575  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.294114  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:39.262349  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.264561  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:39.298882  1735 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:39.300076  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:39.297461  1585 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:39.290354  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.291886  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:39.283777  1593 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:39.224040  5267 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.304761  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.305467  1587 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:39.297216  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.302776  1733 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:39.308303  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.310469  1740 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:39.311038  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.311717  1735 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:39.311641  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 109657us
I20260709 13:02:39.307371  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:39.281883  5297 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.318609  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.319362  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:39.313362  1591 raft_consensus.cc:2376] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10. Re-sending same reply.
W20260709 13:02:39.320878  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.314275  1736 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:39.315574  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.316416  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:39.327102  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.340986  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.342151  1587 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.371585  4685 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:39.372249  4685 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.257079  5146 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 13:02:39.306593  5226 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:39.382529  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.383739  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.383179  4325 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.387784  4325 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.389284  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.364262  4966 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:39.382781  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:39.395076  1585 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:39.397334  1591 raft_consensus.cc:2410] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
W20260709 13:02:39.399341  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118317us
I20260709 13:02:39.328310  5297 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:39.330997  5266 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:39.385605  5377 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:02:39.383250  4685 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:39.436254  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:39.437201  1660 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 13:02:39.438305  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:39.430992  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.410012  4966 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.441479  4325 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:39.321586  5267 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 13:02:39.395773  5226 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.446812  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.474656  1592 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:39.452315  5386 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 13:02:39.453840  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.399770  5392 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.456553  4325 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:39.333328  5110 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260709 13:02:39.463284  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.463634  5384 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:39.464056  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.487551  1591 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.469491  5023 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:39.406137  5363 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260709 13:02:39.470175  5388 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 13:02:39.500723  1584 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:39.502123  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.502988  1584 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:39.503731  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.504534  1589 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 13:02:39.504585  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.428365  5118 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:39.505718  5118 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:39.242012  5290 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.506115  5118 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.506938  5290 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:39.507019  5351 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:39.508194  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.508822  1593 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:39.479696  1587 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
W20260709 13:02:39.483354  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.441293  5106 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.442284  5266 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:39.454840  5241 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:39.500183  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80190.6us
I20260709 13:02:39.528959  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2356) took 1212 ms. Trace:
I20260709 13:02:39.458467  4966 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.529274  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.193601  5364 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.299s
W20260709 13:02:39.547430  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.557456  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:39.530403  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:39.511566  2281 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.481572  5226 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:39.555099  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:39.492606  5392 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.553259  1734 rpcz_store.cc:276] 0709 13:02:38.317094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.323610 (+  6516us) service_pool.cc:224] Handling call
0709 13:02:38.522072 (+198462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:38.548088 (+ 26016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.528685 (+980597us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54001081d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:39.528858 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394572544}
W20260709 13:02:39.500730  5387 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
I20260709 13:02:39.558823  1589 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:02:39.441601  5319 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:02:39.499068  5349 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.492432  5023 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:39.465409  4974 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.560462  5386 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 13:02:39.593680  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 278148us
I20260709 13:02:39.238806  5373 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.614107  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.570185  5160 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:39.509096  5379 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.616528  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:39.614719  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 224501us
I20260709 13:02:39.623147  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:39.611786  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115729us
I20260709 13:02:39.576025  5270 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.576869  4910 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:39.627887  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.601362  4965 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:39.641809  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:39.647348  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.560811  5241 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.651396  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:39.622189  5160 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.659827  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2383) took 1094 ms. Trace:
I20260709 13:02:39.650281  5021 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.636554  4734 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:39.553215  5384 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.663492  1586 rpcz_store.cc:276] 0709 13:02:38.565029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.566882 (+  1853us) service_pool.cc:224] Handling call
0709 13:02:38.567281 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:38.567746 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:38.650609 (+ 82863us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.803668 (+153059us) spinlock_profiling.cc:243] Waited 23.5 ms on lock 0x7b18001b8500. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:38.809403 (+  5735us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b18001b8500. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:02:38.809740 (+   337us) log.cc:844] Serialized 14 byte log entry
0709 13:02:38.809954 (+   214us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:38.865039 (+ 55085us) log.cc:844] Serialized 52 byte log entry
0709 13:02:38.865206 (+   167us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:38.865241 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:38.865313 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:39.659574 (+794261us) raft_consensus.cc:1685] finished
0709 13:02:39.659600 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:39.659757 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76827,"prepare.queue_time_us":259959,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":1438,"raft.queue_time_us":4919,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":313,"spinlock_wait_cycles":92750848,"thread_start_us":112399,"threads_started":2,"wal-append.queue_time_us":193348}
W20260709 13:02:39.679579  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 296870us
I20260709 13:02:39.649523  5363 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:39.644547  4965 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:39.656148  5077 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:39.616005  5214 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.709664  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 198229us
I20260709 13:02:39.717545  5357 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.717833  5397 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.718462  5397 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.672374  4734 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.679554  5040 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:39.699245  5373 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:39.725823  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:39.670779  5160 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:39.677129  5122 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.680768  4967 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.663194  5352 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.738276  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56672.9us
I20260709 13:02:39.744127  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.707927  4965 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:39.755141  5270 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:39.713433  5267 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.746817  5397 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.757987  5241 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:39.754164  5394 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
W20260709 13:02:39.759234  5402 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 13:02:39.760298  5379 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:39.728857  4734 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.728098  5040 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.782315  5040 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.757863  5267 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.763388  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:39.767941  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.770685  1737 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:39.724107  5363 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:39.679430  5021 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.776067  1586 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:39.778870  5384 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:39.796265  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56022us
W20260709 13:02:39.786588  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54871.1us
I20260709 13:02:39.669047  5266 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.810379  1742 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:39.738967  5077 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:39.742142  5319 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:02:39.825446  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 222259us
I20260709 13:02:39.826150  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:39.676409  5023 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:39.829412  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 66104.6us
I20260709 13:02:39.835604  1586 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:02:39.836058  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:39.841212  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.810948  5379 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:39.830641  5146 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.806175  5400 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:39.825683  5190 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:39.860852  5190 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:39.861287  5190 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:39.864372  5077 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.853021  5352 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.871930  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.855521  5146 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.858669  5021 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:39.859092  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:39.859462  4974 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.889324  1708 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:39.854841  5214 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.810508  5182 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:02:39.893893  5293 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:02:39.896579  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:39.898159  5293 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:39.900147  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:39.902087  5412 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:39.876127  5364 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:39.874394  4967 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:39.904619  4967 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:39.905774  4967 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.908506  5392 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:39.910493  5363 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.913707  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.914580  5267 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:39.874380  5415 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 13:02:39.919961  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:39.921926  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:39.860432  5350 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.754s	user 0.009s	sys 0.001s
I20260709 13:02:39.893373  1737 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:39.901908  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.902853  5412 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.892129  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.924350  4325 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:39.926785  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.928395  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.929251  5392 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:39.931550  4325 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.931413  5364 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:39.938077  5415 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:39.938714  5415 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:39.944856  1737 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:39.951943  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50341.2us
W20260709 13:02:39.952001  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52049.9us
I20260709 13:02:39.938863  5226 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:39.958726  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:39.959256  5267 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:39.960083  5411 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:39.960794  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:39.933377  5374 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:39.977130  1586 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 13:02:39.939719  5241 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:39.992591  1591 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:39.999347  4325 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.007385  5414 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:40.008998  5405 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:40.009686  5351 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:40.010339  5351 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:39.954818  5415 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.016747  5415 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.023470  5146 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.027876  5349 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:02:40.030490  5413 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 13:02:40.031929  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:40.032194  5290 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.525s	user 0.006s	sys 0.006s
I20260709 13:02:39.934289  5350 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:40.041060  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.049283  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:40.078275  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:40.034238  5118 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.528s	user 0.014s	sys 0.000s
I20260709 13:02:40.098676  1740 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:40.100044  4325 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:02:40.100239  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51067.7us
I20260709 13:02:39.990329  5241 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:40.109879  4325 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:40.103746  1735 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:40.009727  5405 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:39.996614  4734 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.122366  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:39.996981  4966 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.100939  5411 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:40.114257  5241 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:40.133867  4685 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.680s	user 0.015s	sys 0.001s
W20260709 13:02:40.134804  5372 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.990s	user 0.010s	sys 0.013s
I20260709 13:02:40.149310  5077 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:40.148064  5266 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:40.155105  4734 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.159855  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.178103  1586 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:40.162010  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2369) took 1411 ms. Trace:
I20260709 13:02:40.180202  1739 rpcz_store.cc:276] 0709 13:02:38.750161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:38.750292 (+   131us) service_pool.cc:224] Handling call
0709 13:02:39.240148 (+489856us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.240849 (+   701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.161803 (+920954us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400284950. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:40.161946 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1107702656}
I20260709 13:02:40.182825  5326 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:40.183274  5326 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:40.184214  5326 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.187462  4977 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.192780  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2382) took 1400 ms. Trace:
I20260709 13:02:40.193209  1736 rpcz_store.cc:276] 0709 13:02:38.795293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.118581 (+323288us) service_pool.cc:224] Handling call
0709 13:02:39.136820 (+ 18239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.321875 (+185055us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.192504 (+870629us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400404f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:40.192715 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407265152}
I20260709 13:02:40.195082  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.195750  5351 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.196503  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.221930  1590 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:40.165192  5405 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.167728  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.239698  1591 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:40.170404  5190 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.198316  5266 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.200311  5226 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.203604  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.210521  5412 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.214323  5261 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.214803  5146 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:40.142354  5297 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.708s	user 0.010s	sys 0.003s
I20260709 13:02:40.111569  5118 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.163098  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.217471  1738 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:40.223191  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.250022  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.140687  5411 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.134347  5040 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.314495  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.319365  5146 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.319964  1739 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:40.313331  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.313827  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.314163  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.323312  1591 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:40.310099  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.304117  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:40.322957  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.324417  2310 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:40.318876  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:40.323549  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.315040  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.325933  2314 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:40.318397  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.304894  2313 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.333297  1587 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:40.257449  2312 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.326877  1585 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:40.339593  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.340047  5411 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:40.252152  1557 connection.cc:256] RPC call timeout handler was delayed by -2.1626495529999374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
W20260709 13:02:40.345134  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:40.363487  1557 connection.cc:256] RPC call timeout handler was delayed by -1.9529495529999394s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1771s for a response
I20260709 13:02:40.364308  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:40.365059  1557 connection.cc:256] RPC call timeout handler was delayed by -1.7828495529999486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.196s for a response
W20260709 13:02:40.365440  1557 connection.cc:256] RPC call timeout handler was delayed by -1.6829495529999576s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
W20260709 13:02:40.365794  1557 connection.cc:256] RPC call timeout handler was delayed by -1.6487495529999592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260709 13:02:40.365891  2312 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
W20260709 13:02:40.366133  1557 connection.cc:256] RPC call timeout handler was delayed by -1.5722495529999492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
W20260709 13:02:40.363386  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 216171us
I20260709 13:02:40.364637  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:40.369437  5247 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.003s	sys 0.001s
W20260709 13:02:40.369925  5247 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.003s	sys 0.001s
W20260709 13:02:40.313272  4965 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.558s	user 0.008s	sys 0.010s
W20260709 13:02:40.372852  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44781.8us
I20260709 13:02:40.375325  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.375991  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.376434  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:40.318564  5160 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 13:02:40.380455  2318 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
W20260709 13:02:40.382884  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 383789us
I20260709 13:02:40.384585  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.220345  4685 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.389642  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.392604  1591 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: [5.6-6.7]
I20260709 13:02:40.256032  5353 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:40.383849  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:40.308598  5266 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.314388  5190 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.387928  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.383808  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:40.310577  4966 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.297463  4977 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.388788  4325 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:40.238049  5372 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.414203  4325 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.414976  4325 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:40.415021  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:40.396358  1664 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:40.398327  5353 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.399021  1557 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:40.422554  1557 leader_election.cc:403] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:02:40.318643  5350 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.399928  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.423153  1592 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:40.423897  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.382198  5351 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:40.404187  1662 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 13:02:40.429610  1662 pending_rounds.cc:85] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 8
I20260709 13:02:40.429322  1668 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:40.429889  1662 pending_rounds.cc:107] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 13:02:40.430267  1662 raft_consensus.cc:2889] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:40.403999  2313 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:40.420480  1661 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:02:40.371677  5261 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.402237  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.426332  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.396016  5146 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:40.448421  2311 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:40.446599  5364 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
W20260709 13:02:40.450229  5404 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.700s	user 0.001s	sys 0.004s
I20260709 13:02:40.450829  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:40.451545  2316 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:40.452561  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2386) took 1249 ms. Trace:
I20260709 13:02:40.453081  1742 rpcz_store.cc:276] 0709 13:02:39.203507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.203733 (+   226us) service_pool.cc:224] Handling call
0709 13:02:39.204028 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:39.811580 (+607552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.452307 (+640727us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:40.452494 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1394051968}
W20260709 13:02:40.453116  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 54251.6us
I20260709 13:02:40.342434  5297 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:40.444523  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60784.4us
I20260709 13:02:40.423566  4967 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:40.460569  1593 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:40.452495  1663 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.462424  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:40.397405  5247 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:02:40.387471  5349 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
W20260709 13:02:40.465313  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83964.9us
I20260709 13:02:40.388283  4965 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.467710  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:40.447249  5411 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:40.388175  5160 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.473587  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:40.394861  5374 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.058s
I20260709 13:02:40.468859  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:40.443270  5429 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:40.443665  5403 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.484547  1634 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:40.495164  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:40.466925  4967 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.496503  2312 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:40.429373  4977 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.507301  4967 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.508503  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:40.509188  4966 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:40.509403  4977 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.517607  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:40.511677  2315 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:02:40.512610  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.534273  5267 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:40.531152  5021 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:40.545651  5424 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 13:02:40.543673  5436 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.498826  5429 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:40.580538  2314 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:40.494205  5358 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:40.502754  5364 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:40.509661  4966 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:40.566021  1742 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:40.588768  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.566543  2316 raft_consensus.cc:606] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:40.589787  2316 raft_consensus.cc:614] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:40.452198  5261 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.575598  1663 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:40.569260  5267 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:40.597256  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.574314  5118 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.568452  5021 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.568938  5040 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.546353  5038 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.038s
I20260709 13:02:40.614109  4325 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:40.612016  5397 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.016s	sys 0.023s
I20260709 13:02:40.513031  5403 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.584084  5429 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.607064  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:40.582679  5190 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:02:40.593948  4966 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:40.627586  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:40.617678  5397 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.016s	sys 0.023s
I20260709 13:02:40.611455  5267 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.617514  5372 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:40.615473  5021 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.639447  1585 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.641784  4325 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.642781  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.643712  1742 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.611032  4734 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 13:02:40.588622  5247 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 13:02:40.594221  5358 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.634989  4685 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.640414  5417 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.661646  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:40.665012  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76143.9us
I20260709 13:02:40.649264  5297 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.659503  4965 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.663286  5438 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.020s
W20260709 13:02:40.680022  5435 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 13:02:40.680657  5392 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:40.664515  5160 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.671598  5437 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:40.702005  1742 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:40.681100  5392 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:40.689378  5426 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
W20260709 13:02:40.742863  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46815.2us
I20260709 13:02:40.674413  4685 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260709 13:02:40.699999  5432 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:02:40.695817  5437 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:40.751475  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 277996us
I20260709 13:02:40.757015  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:40.757545  2312 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.770778  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.771710  1733 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:40.772825  1741 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:40.774329  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.774761  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:40.775429  2317 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:40.775882  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.780586  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:40.781199  1735 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 13:02:40.782830  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.783941  2313 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:02:40.784893  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.786610  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.789235  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:40.789330  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 124070us
I20260709 13:02:40.790756  1736 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:40.790545  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.785403  1734 raft_consensus.cc:2376] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9. Re-sending same reply.
I20260709 13:02:40.791268  1741 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.791328  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:40.792430  1733 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.793628  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.756197  5437 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.792529  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.794281  1738 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:40.795713  4325 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election won for term 6
I20260709 13:02:40.796444  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.750598  5247 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:02:40.800079  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.799791  5454 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:40.802635  5454 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.803004  5454 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:40.806607  5374 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.807190  5374 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.808741  5021 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.810331  5182 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.810922  5182 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.812213  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:40.800595  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:40.813091  2309 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:40.813714  5443 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 13:02:40.797600  5392 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.815165  5392 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:40.816325  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.788486  5364 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:40.817000  1592 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.817102  5364 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.818259  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.818462  5356 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:40.818925  1591 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.772761  5353 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:40.822790  4325 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:40.829351  1741 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:02:40.822021  5350 consensus_queue.cc:1175] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e6adef9da7da44f6af0af5695cf9b704" responder_term: 8 status { last_received { term: 7 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:02:40.831426  1739 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:40.826884  5182 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:40.832388  5182 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:40.833204  5241 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.687s	user 0.005s	sys 0.006s
I20260709 13:02:40.833669  5241 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:40.834260  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.832857  5182 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.834689  1660 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:40.834856  1739 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:40.835690  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:40.836766  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 341747us
I20260709 13:02:40.836922  1733 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:40.774794  5040 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 13:02:40.847587  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:40.848631  4325 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:40.799433  4734 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:40.852381  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:40.854704  5407 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.108s	user 0.000s	sys 0.004s
I20260709 13:02:40.788821  5431 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 13:02:40.856375  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.856696  5021 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.840581  5266 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:40.858109  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.876508  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.865417  1591 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:40.878952  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.868083  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.797832  5451 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:40.827330  5411 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:40.873025  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:40.873246  1735 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:02:40.840471  5261 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:40.874168  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.884089  1740 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:40.886631  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 43884.5us
I20260709 13:02:40.890364  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.917469  2317 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:40.934403  1668 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 6: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:40.894026  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:40.934842  2318 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:02:40.935081  1593 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:40.868181  5353 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:40.894816  5431 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:40.894973  1661 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:02:40.895156  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.896101  1665 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:40.834735  5293 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:02:40.894622  1667 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:02:40.936419  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.898588  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2461) took 1605 ms. Trace:
I20260709 13:02:40.894026  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:40.956132  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.956907  1662 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:40.958786  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:40.952737  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.931721  5266 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:40.957031  1734 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.963996  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:40.982652  1593 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:40.938048  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.964993  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.967099  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.886389  5451 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:40.955125  1589 rpcz_store.cc:276] 0709 13:02:39.293175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.577454 (+284279us) service_pool.cc:224] Handling call
0709 13:02:39.616285 (+ 38831us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:39.713763 (+ 97478us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:39.838921 (+125158us) log.cc:844] Serialized 14 byte log entry
0709 13:02:40.117459 (+278538us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:40.135320 (+ 17861us) log.cc:844] Serialized 26 byte log entry
0709 13:02:40.135480 (+   160us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:40.135515 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.135596 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:40.897829 (+762233us) raft_consensus.cc:1685] finished
0709 13:02:40.898381 (+   552us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:40.898527 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34556,"prepare.queue_time_us":208337,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":77950,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":7660,"thread_start_us":276922,"threads_started":3,"wal-append.queue_time_us":511948}
I20260709 13:02:40.969081  2311 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:40.932554  5358 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:40.948057  5431 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:40.943432  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:41.004786  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61465.8us
W20260709 13:02:41.005362  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 132434us
I20260709 13:02:40.972241  4734 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:40.987550  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:40.944849  5353 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:40.875974  5372 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 13:02:40.989950  1736 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:40.972752  5441 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
I20260709 13:02:40.993615  1557 leader_election.cc:400] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.020758  1668 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:40.999908  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.963161  5261 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 13:02:41.003096  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:40.989316  5293 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:41.007032  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:40.989691  5401 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.257s	user 0.002s	sys 0.002s
I20260709 13:02:40.974236  5403 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.013407  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:40.957653  5040 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:02:41.015977  5411 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.020874  1557 leader_election.cc:403] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:02:40.978091  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.071346  1742 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:41.007032  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.025604  1593 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:41.023685  5261 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.077991  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2395) took 1881 ms. Trace:
W20260709 13:02:41.031550  5449 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
W20260709 13:02:41.032377  5452 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:02:41.018260  5353 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:41.034610  5446 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 13:02:41.035056  5021 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:41.016798  5241 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.016541  5247 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:41.011533  4734 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.044507  5417 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:41.113971  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.114081  4734 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.115526  4734 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:41.115643  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:40.989197  5451 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:41.045102  5293 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.050659  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:41.051761  5456 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:02:41.054390  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.056949  5436 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:41.059430  1735 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:41.093921  5270 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 13:02:41.017572  1585 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:41.061621  1739 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:41.075109  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:41.030366  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.069229  1740 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:41.094635  1660 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:41.078425  1588 rpcz_store.cc:276] 0709 13:02:39.196421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.209910 (+ 13489us) service_pool.cc:224] Handling call
0709 13:02:39.282401 (+ 72491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:39.294372 (+ 11971us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:39.294408 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:39.309640 (+ 15232us) log.cc:844] Serialized 26 byte log entry
0709 13:02:39.581958 (+272318us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:39.582025 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.582129 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:40.775344 (+1193215us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:41.057753 (+282409us) raft_consensus.cc:1685] finished
0709 13:02:41.057781 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.077908 (+ 20127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12291,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":7952,"spinlock_wait_cycles":66241792,"thread_start_us":185523,"threads_started":1,"wal-append.queue_time_us":324272}
W20260709 13:02:41.095458  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 113820us
I20260709 13:02:41.098153  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.100574  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.092470  5351 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.095932  5106 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.103617  5021 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.107836  5326 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:41.091311  5261 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:41.101627  5405 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.123637  1736 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:41.128607  1586 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:41.132777  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.136672  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2472) took 1027 ms. Trace:
I20260709 13:02:41.136821  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 521) took 1906 ms. Trace:
I20260709 13:02:41.142266  1742 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.133374  5270 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:41.082228  5392 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.162020  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:41.163136  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:41.191021  1741 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:41.181702  1590 rpcz_store.cc:276] 0709 13:02:40.109615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.371958 (+262343us) service_pool.cc:224] Handling call
0709 13:02:40.396007 (+ 24049us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:40.396391 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:40.396425 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:40.418824 (+ 22399us) log.cc:844] Serialized 52 byte log entry
0709 13:02:40.642967 (+224143us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:40.643013 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:40.643093 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.136448 (+493355us) raft_consensus.cc:1685] finished
0709 13:02:41.136472 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.136610 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":191065,"threads_started":1,"wal-append.queue_time_us":231230}
I20260709 13:02:41.181838  1584 rpcz_store.cc:276] 0709 13:02:39.230507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:39.507434 (+276927us) service_pool.cc:224] Handling call
0709 13:02:39.507743 (+   309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:39.508100 (+   357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:39.508127 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:39.508738 (+   611us) log.cc:844] Serialized 52 byte log entry
0709 13:02:39.634875 (+126137us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:39.634941 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:39.635052 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:40.775789 (+1140737us) spinlock_profiling.cc:243] Waited 27.2 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:41.124361 (+348572us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b24001868d8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2efb0 00007fcdedba5488 00007fcdedba5be8 00007fcdedb7d398 00007fcdedb7876a 00007fcdedb77bb6
0709 13:02:41.136627 (+ 12266us) raft_consensus.cc:1685] finished
0709 13:02:41.136651 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.136770 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":45535,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":5013,"spinlock_wait_cycles":181360128,"thread_start_us":125615,"threads_started":1,"wal-append.queue_time_us":211336}
I20260709 13:02:41.156651  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:41.081180  5411 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.193387  1584 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [6.4-9.6]
I20260709 13:02:41.181456  1588 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:41.185835  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.166989  5349 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:41.175256  5021 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:41.174006  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.192309  1660 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:41.182785  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.196485  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.123152  5293 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:41.189527  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 114474us
I20260709 13:02:41.178290  5436 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:41.051810  5267 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.117470  5364 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.199649  1586 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:41.186785  5414 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.202718  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:41.215605  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:41.211971  1661 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:41.187311  5405 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.218183  4906 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:41.198422  5392 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.217972  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:41.096580  5266 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.168965  5351 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.223166  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.228767  1666 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:41.189821  5270 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.248574  1734 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:41.249686  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:41.270490  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47436.3us
I20260709 13:02:41.250087  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:41.262140  5403 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 13:02:41.262317  5364 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 13:02:41.226719  5040 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.266135  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:41.254546  5444 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.508s	user 0.003s	sys 0.000s
I20260709 13:02:41.250581  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.232496  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.297073  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:41.226580  5267 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.245786  5118 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:41.224195  5462 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.301483  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.305002  1557 leader_election.cc:403] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:02:41.306713  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.350936  1660 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:41.310911  1659 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:41.242355  4906 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:41.241770  5405 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.109395  5431 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.366196  5190 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:41.366694  5190 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.367744  5190 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.369335  5190 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:41.347750  1592 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.003s
I20260709 13:02:41.371954  1592 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:41.248682  5392 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:41.248704  5293 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:41.290302  4965 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:41.203956  5349 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.274313  5373 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:02:41.248910  5430 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 13:02:41.362808  5040 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:02:41.394773  5467 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:02:41.303875  5412 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 13:02:41.237895  5414 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.374095  5405 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
W20260709 13:02:41.388703  4967 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.881s	user 0.015s	sys 0.000s
I20260709 13:02:41.386168  5106 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.385015  4965 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.358270  5451 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:41.317191  5454 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.514s	user 0.013s	sys 0.000s
W20260709 13:02:41.432503  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 214604us
W20260709 13:02:41.451838  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 155010us
I20260709 13:02:41.421067  5040 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.435351  4965 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:41.444389  4966 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.800s	user 0.007s	sys 0.006s
W20260709 13:02:41.471638  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 208538us
I20260709 13:02:41.362534  5462 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.442524  5412 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:41.439206  5373 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:02:41.449443  5429 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.813s	user 0.008s	sys 0.007s
I20260709 13:02:41.483891  1739 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 13:02:41.489542  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.402081  5270 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.424587  4967 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.469192  5118 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:41.459115  5436 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.444794  5454 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.500931  1740 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:41.444000  5106 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.486218  5412 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:41.489286  5373 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:41.508767  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.510144  1737 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:41.511847  5392 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:41.518216  5412 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.518795  5349 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:41.519831  5412 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:41.525942  5351 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:41.526227  5411 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.527453  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:41.530443  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.532513  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.533991  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:41.542411  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.543197  1587 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:41.542448  1586 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:41.490363  4966 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.498200  5293 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 13:02:41.512269  5270 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:41.539639  5372 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:41.502391  4734 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260709 13:02:41.492370  5429 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.511658  5463 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:41.528553  5106 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.573050  1740 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:41.542569  5261 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:41.576618  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77140.8us
I20260709 13:02:41.551291  5349 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:02:41.589551  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:41.603740  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76382.8us
I20260709 13:02:41.592180  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.609293  1593 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:41.599715  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:41.594933  4325 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:41.612958  1667 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:41.622347  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:41.624577  1663 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
W20260709 13:02:41.624701  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71222.5us
W20260709 13:02:41.617223  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 132821us
I20260709 13:02:41.625875  1705 leader_election.cc:400] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:41.626191  1705 leader_election.cc:403] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 13:02:41.622330  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.596776  4685 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.628335  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.632268  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.628700  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.635799  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.640579  1556 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.642715  1556 leader_election.cc:403] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:02:41.653792  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.656598  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:41.558017  5040 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.629889  5372 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7.
I20260709 13:02:41.663792  1593 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:41.630122  5411 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.672240  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:41.684701  1659 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:41.636976  5486 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.637794  5414 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.647131  5451 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.682437  5478 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:41.617424  5373 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.701148  5480 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:41.637877  5463 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 13:02:41.702790  5477 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:02:41.703984  5470 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 13:02:41.704692  5488 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:41.630013  5436 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.705567  1734 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:02:41.705857  4967 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:41.706485  5437 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.878s	user 0.011s	sys 0.004s
I20260709 13:02:41.706935  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.738212  1738 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:02:41.709863  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:41.739035  1739 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.701412  4966 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.636978  4685 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.644698  5473 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260709 13:02:41.725018  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:41.725906  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.715396  5480 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:41.742213  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:41.741823  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 142645us
I20260709 13:02:41.744933  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.749105  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:41.747004  5480 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.745440  5373 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:41.718792  5411 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:41.781440  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 150164us
I20260709 13:02:41.803471  5474 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:41.804678  5481 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:41.805995  5472 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 13:02:41.807744  5487 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:02:41.809904  4966 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:41.810352  4966 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.811538  5411 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.812193  5411 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.741200  5488 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.814083  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2431) took 1088 ms. Trace:
I20260709 13:02:41.746124  5454 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.814680  1590 rpcz_store.cc:276] 0709 13:02:40.725948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.213079 (+487131us) service_pool.cc:224] Handling call
0709 13:02:41.224649 (+ 11570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.245940 (+ 21291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.245995 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.246610 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.447167 (+200557us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:41.447216 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.447295 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.813631 (+366336us) raft_consensus.cc:1685] finished
0709 13:02:41.813666 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.813973 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20712,"raft.queue_time_us":22254,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":200093,"threads_started":1,"wal-append.queue_time_us":265206}
I20260709 13:02:41.828356  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2430) took 1219 ms. Trace:
I20260709 13:02:41.817847  1668 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 13:02:41.829914  1588 rpcz_store.cc:276] 0709 13:02:40.608644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.195361 (+586717us) service_pool.cc:224] Handling call
0709 13:02:41.201410 (+  6049us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:41.201800 (+   390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:41.201836 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.212969 (+ 11133us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.356084 (+143115us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:41.356132 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.356210 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.828127 (+471917us) raft_consensus.cc:1685] finished
0709 13:02:41.828150 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.828290 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22412,"raft.queue_time_us":475,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":19758,"thread_start_us":131853,"threads_started":1,"wal-append.queue_time_us":259739}
I20260709 13:02:41.717833  5486 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.771766  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:41.847764  5503 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 13:02:41.866320  5479 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.138s	user 0.000s	sys 0.003s
I20260709 13:02:41.867157  5403 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:41.867630  5403 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:41.868528  5403 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.869747  5429 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.870811  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:41.879992  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:41.883546  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:41.743526  5437 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:41.877976  5021 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.630s	user 0.009s	sys 0.008s
I20260709 13:02:41.885782  2318 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:41.886369  5503 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:41.886401  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:41.888077  5501 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:41.805284  5481 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.889984  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:41.813531  5372 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.891825  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.937168  1593 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:41.894570  5021 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.896584  4967 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:41.898506  5488 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.899204  5023 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:41.898535  5437 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.812692  5411 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.899302  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:41.902316  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2426) took 1324 ms. Trace:
I20260709 13:02:41.903072  5478 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:41.903965  5480 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.904860  5403 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:41.961778  5483 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.324s	user 0.003s	sys 0.000s
I20260709 13:02:41.963136  5486 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.963807  5486 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.964318  5480 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.964808  5503 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.957449  5437 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260709 13:02:41.967981  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:41.968619  1590 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.969724  2309 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:41.969831  5190 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.007s
W20260709 13:02:41.954603  5353 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.844s	user 0.006s	sys 0.008s
I20260709 13:02:41.972796  5021 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.973577  5353 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.942939  5372 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.975545  1586 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:41.905360  5495 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:41.979574  5462 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:41.980129  5412 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:41.910566  5160 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:41.910948  5454 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:41.915403  5247 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:41.916224  5266 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:41.922295  5261 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:41.922398  5106 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:41.989533  5247 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:42.009845  5247 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.011556  5247 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:41.926330  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 789) took 1721 ms. Trace:
I20260709 13:02:42.013736  2315 rpcz_store.cc:276] 0709 13:02:40.204856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.205008 (+   152us) service_pool.cc:224] Handling call
0709 13:02:40.205372 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.565702 (+360330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.925932 (+1360230us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b54006a0254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:41.926242 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":703634176}
I20260709 13:02:41.934490  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2313) took 1076 ms. Trace:
I20260709 13:02:41.942029  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:42.015816  1664 rpcz_store.cc:276] 0709 13:02:40.857960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.871465 (+ 13505us) service_pool.cc:224] Handling call
0709 13:02:41.934269 (+1062804us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54001a81d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:41.934411 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":382546688}
W20260709 13:02:41.950661  5465 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.502s	user 0.001s	sys 0.012s
I20260709 13:02:41.979965  5495 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.976383  1662 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:42.007208  5106 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.018690  5481 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:41.964339  5486 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:41.979224  5372 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:41.811551  4966 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:41.954056  1589 rpcz_store.cc:276] 0709 13:02:40.578305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.168281 (+589976us) service_pool.cc:224] Handling call
0709 13:02:41.179815 (+ 11534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.262332 (+ 82517us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:41.416108 (+153776us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.588732 (+172624us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.616717 (+ 27985us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.623488 (+  6771us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.623644 (+   156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:41.623690 (+    46us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:41.623747 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.901992 (+278245us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b540025add4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:41.902124 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:41.902255 (+   131us) inbound_call.cc:177] Queueing success response
0709 13:02:41.910008 (+  7753us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b18003861a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
Metrics: {"mutex_wait_us":62185,"prepare.queue_time_us":289618,"prepare.run_cpu_time_us":199,"prepare.run_wall_time_us":7525,"raft.queue_time_us":12677,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":9752,"spinlock_wait_cycles":221546112,"thread_start_us":304257,"threads_started":2,"wal-append.queue_time_us":273513}
I20260709 13:02:41.908037  4685 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:41.982064  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.047434  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.045841  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 128494us
W20260709 13:02:42.048142  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.002764  1590 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:42.005149  5392 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:41.932134  5500 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:42.026384  1660 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:42.026877  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:41.987277  5160 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:42.028585  5267 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:41.980845  4967 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:42.063221  1593 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:02:42.067970  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.048174  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:42.062636  5392 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:41.990136  5412 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.064854  5481 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:42.076167  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:42.079845  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:42.074677  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 329873us
W20260709 13:02:42.082919  4965 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.006s
I20260709 13:02:42.069321  5267 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.088598  5481 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.089174  4965 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.083196  5160 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.080024  5372 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 13:02:42.086318  2315 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:42.101397  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:42.087809  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2414) took 1242 ms. Trace:
W20260709 13:02:42.088277  5270 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.508s	user 0.014s	sys 0.001s
I20260709 13:02:42.091646  5481 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.074081  4966 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.084959  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.071463  4685 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:42.084304  5353 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.102201  1593 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:42.067224  5411 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.102351  1733 rpcz_store.cc:276] 0709 13:02:40.845871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.846100 (+   229us) service_pool.cc:224] Handling call
0709 13:02:40.846435 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:40.846843 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:40.872204 (+ 25361us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.189706 (+317502us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:41.203183 (+ 13477us) log.cc:844] Serialized 26 byte log entry
0709 13:02:41.203509 (+   326us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:41.204059 (+   550us) log.cc:844] Serialized 14 byte log entry
0709 13:02:41.204418 (+   359us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:41.204522 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:41.922690 (+718168us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:42.087579 (+164889us) raft_consensus.cc:1685] finished
0709 13:02:42.087604 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.087742 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19283,"prepare.queue_time_us":49511,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":107,"raft.queue_time_us":29237,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":4930,"spinlock_wait_cycles":13457536,"thread_start_us":198431,"threads_started":1,"wal-append.queue_time_us":439764}
I20260709 13:02:42.113236  4685 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:42.109843  5485 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 13:02:42.103678  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:42.103348  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:42.112005  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:42.104748  5326 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.920s	user 0.012s	sys 0.010s
I20260709 13:02:42.103318  5261 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:42.101786  5373 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:42.123490  5160 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.182318  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59800 (request call id 6) took 1705 ms. Trace:
I20260709 13:02:42.182853  2316 rpcz_store.cc:276] 0709 13:02:40.467177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.468666 (+  1489us) service_pool.cc:224] Handling call
0709 13:02:42.145955 (+1677289us) spinlock_profiling.cc:243] Waited 71.8 ms on lock 0x7b54006a0254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfd7853 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:42.172878 (+ 26923us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":165213568}
I20260709 13:02:42.184748  5392 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.187088  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.188570  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.171437  5486 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.149243  4685 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.189810  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.190501  1666 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:42.191409  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.192459  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.116936  4967 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.193614  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.195132  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.175451  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.196241  5511 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:42.177783  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.178567  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.180032  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:42.200220  5351 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:42.200632  5350 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:42.200680  5351 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.202023  5351 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.204181  5412 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.173314  5501 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.205191  5501 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.206565  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.208439  5485 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:42.208572  5493 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:42.208877  5485 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.209105  5493 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:42.209908  5364 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:42.210310  5364 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:42.210165  5493 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.211244  5364 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.212270  5364 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.213678  5504 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:42.214123  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.214222  5504 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.214849  5493 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.215442  5480 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:42.215524  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.216457  5480 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.217691  5478 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:42.197213  5372 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:42.177353  1668 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:42.227316  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:42.181890  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.196617  5481 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:02:42.237128  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.209363  5485 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:42.245122  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.238197  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:42.219901  5392 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.246807  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.247466  1668 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:42.248929  1663 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:42.221017  5466 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.021s	user 0.000s	sys 0.003s
I20260709 13:02:42.250214  4325 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.254355  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:42.254745  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:42.256263  4325 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.261336  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.257887  5500 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.267056  1735 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:42.268366  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.268996  1735 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.270277  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.221302  5495 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.173418  5506 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:42.274372  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.277115  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.226289  5372 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:42.277894  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:42.283906  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 199076us
I20260709 13:02:42.284740  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.284744  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.279752  1741 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:02:42.287473  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:42.288092  1740 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:42.289186  1740 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:42.291258  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.244941  5261 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:42.292313  1736 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:42.293262  4325 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.281925  1668 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:42.300629  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.254263  5481 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:42.302858  1735 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:42.303274  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:42.299522  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.273044  5463 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.267683  5451 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:42.308830  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.281679  5106 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.320284  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:42.320603  4325 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:42.297377  5412 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.324599  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:42.325443  1736 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:42.326783  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2420) took 1750 ms. Trace:
I20260709 13:02:42.321027  1588 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:42.327180  1585 rpcz_store.cc:276] 0709 13:02:40.576705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.135339 (+558634us) service_pool.cc:224] Handling call
0709 13:02:41.135600 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.135798 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:41.268772 (+132974us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.325894 (+1057122us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b180009afc0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:42.326248 (+   354us) log.cc:844] Serialized 14 byte log entry
0709 13:02:42.326445 (+   197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:42.326499 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:42.326548 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.326614 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:42.326726 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6441,"prepare.queue_time_us":253577,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":18283,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":3499,"spinlock_wait_cycles":2719616,"thread_start_us":342542,"threads_started":3,"wal-append.queue_time_us":215232}
I20260709 13:02:42.332540  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 559) took 1152 ms. Trace:
I20260709 13:02:42.332787  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:42.311295  4967 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:42.310985  5405 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.336984  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.332979  1667 rpcz_store.cc:276] 0709 13:02:41.171967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.172950 (+   983us) service_pool.cc:224] Handling call
0709 13:02:41.173493 (+   543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:41.623676 (+450183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.324128 (+700452us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54002f2954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:42.324323 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":998700544}
I20260709 13:02:42.318120  5454 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:42.330895  5106 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:42.355778  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.348845  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:42.305328  5261 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.289737  5500 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.339164  4965 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.339990  4967 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.383481  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.354846  5160 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.391275  1738 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.318706  5463 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.378863  5454 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:42.355407  5372 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.378209  5513 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260709 13:02:42.381181  5266 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:02:42.416962  2316 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:42.453330  1667 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 13:02:42.356851  5040 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20260709 13:02:42.439046  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.449437  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.460728  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.463573  1662 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:42.462538  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.473263  2318 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:42.441607  5515 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 13:02:42.451622  5514 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 13:02:42.460618  5403 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.502s	user 0.010s	sys 0.003s
I20260709 13:02:42.480306  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.492239  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.492693  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.391495  4967 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.483563  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.438580  5182 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.500926  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.450542  5513 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:42.391851  5451 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:42.494707  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:42.500232  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.497246  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.540936  1661 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:42.505940  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.439456  5261 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.451764  5511 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:42.452049  5392 consensus_queue.cc:1175] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "316f3bf18ed94e32b238130a6bfedf1a" responder_term: 10 status { last_received { term: 10 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 10 index: 10 } }
I20260709 13:02:42.532915  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.414760  5481 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.440539  5372 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:42.415246  5498 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:42.546802  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 91911.2us
I20260709 13:02:42.506815  5505 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:42.559782  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.507261  5430 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.435101  5351 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.446573  5454 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.533842  5513 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:42.571257  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110638us
I20260709 13:02:42.573582  1666 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:42.545009  5403 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:42.576028  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.536700  5478 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 13:02:42.545563  5480 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.583626  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:42.586303  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.594772  1740 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 8. Ops: [6.7-6.7]
W20260709 13:02:42.588400  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 127001us
I20260709 13:02:42.552603  5511 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.600936  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.577595  5160 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:42.555539  5270 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:02:42.604107  1667 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:42.607666  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:42.609941  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.614902  5160 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:42.615119  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.617715  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:42.605165  5266 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:42.581993  5430 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.599984  5478 consensus_queue.cc:1243] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:02:42.590651  4685 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:02:42.587037  5403 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.616322  5160 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:42.600405  5498 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:42.652889  5519 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 13:02:42.659875  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.656828  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:42.653625  5463 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.664891  5430 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.666123  1666 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:42.666630  5511 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:42.673687  5518 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:02:42.674257  5498 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.632251  5500 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.657873  5160 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.682016  5160 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:42.683336  5351 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.684350  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.685034  1668 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.620217  5266 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:42.695705  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.696944  1668 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:42.698122  1738 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:42.707144  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:42.707899  1737 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:42.710175  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.702507  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:42.715402  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.607908  5451 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.728307  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.730149  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.731084  2318 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:42.662482  5523 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:42.698145  5511 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:42.746817  5403 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.747861  4967 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.715137  5493 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:42.698601  5040 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:42.713382  5266 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.713868  4685 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.799991  4685 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.802628  2310 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:42.804137  4685 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.714843  5106 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:42.756477  4967 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.759290  5431 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:42.763478  5405 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:42.764372  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:42.768224  5520 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.231s	user 0.001s	sys 0.001s
I20260709 13:02:42.772562  5485 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.773192  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.753912  5412 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:42.788661  5507 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.790303  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1736) took 1184 ms. Trace:
I20260709 13:02:42.867048  1660 rpcz_store.cc:276] 0709 13:02:41.606329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.606446 (+   117us) service_pool.cc:224] Handling call
0709 13:02:42.026020 (+419574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.064265 (+ 38245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:42.789971 (+725706us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b5400014ad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:42.790195 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964160384}
I20260709 13:02:42.788749  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.754441  5493 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:42.823959  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:42.870736  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82069.1us
W20260709 13:02:42.843396  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:42.834098  5040 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.835894  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.875993  1733 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:02:42.874991  1737 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:42.889775  5485 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.890344  1734 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:42.878029  4967 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:42.874751  5493 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:42.894176  5374 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:42.895793  1740 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:42.897439  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.901145  1740 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:42.901463  5160 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.903956  5040 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.904958  2316 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:02:42.905409  5488 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:42.905581  5349 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:42.908181  4685 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:42.916625  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:42.916819  1736 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260709 13:02:42.921339  1668 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:42.919813  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.922540  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:42.925544  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:42.870757  5507 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.927987  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.930838  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:42.903987  5160 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:42.931834  1667 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:42.923131  4967 raft_consensus.cc:2764] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:02:42.919059  5522 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 13:02:42.940598  5523 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:42.943802  5501 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.933156  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:42.954689  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 605966us
I20260709 13:02:42.955444  5437 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:42.955893  5437 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:42.956228  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:42.957442  4965 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:42.958425  4965 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:42.961297  5507 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:42.962323  5411 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.962380  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:42.963097  5411 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.964058  1588 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:42.965967  1591 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:42.965874  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:42.966782  5353 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.967571  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:42.968218  5326 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:42.968498  1587 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:42.969159  5482 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.967382  5353 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.969789  5482 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.972012  5353 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.972412  5482 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.969807  1586 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:42.968940  5326 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.973654  1589 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:42.974786  5326 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:42.975039  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 202045us
I20260709 13:02:42.975364  5326 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:42.976028  5326 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.976435  5326 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:42.979633  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.968310  5500 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:42.985422  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:42.986982  5412 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:42.995333  5502 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:42.995400  5500 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.995960  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:42.996593  5405 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:42.997071  5405 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:42.998638  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:42.998405  5405 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:42.999318  2310 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:42.999895  5405 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:42.987731  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.003605  5160 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:43.007071  5267 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.008847  5182 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:42.988467  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
W20260709 13:02:43.012593  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55093.3us
I20260709 13:02:43.013528  1557 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:43.013582  2313 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:42.990581  5524 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:42.981544  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.009855  5182 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:42.993911  5374 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:42.995975  5502 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:42.989691  5525 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:02:43.013963  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.039058  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:42.989988  4906 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.014602  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.051100  2310 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:43.015844  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.019405  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:43.055852  1660 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:43.021220  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:43.010483  2316 raft_consensus.cc:606] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:43.042484  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:43.056607  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.058207  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:43.060573  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 44820.7us
I20260709 13:02:43.061357  5451 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:02:43.066901  5349 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:43.070415  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:43.073936  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59432.8us
I20260709 13:02:42.981573  5411 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:42.916373  5266 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:42.979445  5513 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.153153  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:43.035425  5267 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.147363  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.079680  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:43.022167  5392 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.040144  5182 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:43.144615  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69242.1us
I20260709 13:02:43.175072  1557 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10.
I20260709 13:02:43.176894  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.178351  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.164705  1662 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
W20260709 13:02:43.103344  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.170344  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:43.181751  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 111502us
I20260709 13:02:43.074427  5482 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:43.041293  5118 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260709 13:02:43.038362  5397 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.139s
I20260709 13:02:43.071436  5374 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.079035  5353 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:43.076346  4966 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260709 13:02:43.162760  5411 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.078919  5106 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.075824  4965 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:43.078753  5040 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.074505  5501 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:43.223259  5499 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:43.224314  5182 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.227964  5266 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.179114  5392 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.232316  5023 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.236757  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:43.170925  5267 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.238250  5513 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:43.186374  4906 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:43.181231  5480 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:43.207886  5500 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.271919  1736 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:43.217155  5353 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:43.210839  5482 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:43.294212  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115729us
I20260709 13:02:43.273083  5412 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:43.296188  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:43.298210  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 127991us
I20260709 13:02:43.300010  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.287987  5353 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:43.252650  5411 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.292040  5182 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:43.249413  5106 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.288451  5482 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:43.306617  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.191164  5502 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.296066  4965 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:43.267310  5040 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.305470  1736 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:43.267014  5501 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.289744  5500 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:43.283346  5023 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.349823  5500 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:43.350701  5267 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:43.352198  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.352932  2313 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:43.353222  4965 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:43.352943  5106 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.353809  4965 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:43.354787  4965 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.362841  5505 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:43.363420  5505 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:43.364320  5505 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:43.366389  5534 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:02:43.372738  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2428) took 1111 ms. Trace:
I20260709 13:02:43.374032  5504 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:43.374586  5477 consensus_queue.cc:1175] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e6adef9da7da44f6af0af5695cf9b704" responder_term: 9 status { last_received { term: 9 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 9 index: 9 } }
I20260709 13:02:43.376587  1733 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:43.377889  1735 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:43.355118  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:43.383265  1733 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 13:02:43.386222  1733 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:02:43.387580  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.373379  1738 rpcz_store.cc:276] 0709 13:02:42.262211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.270029 (+  7818us) service_pool.cc:224] Handling call
0709 13:02:43.372397 (+1102368us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:43.372605 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":276874624}
I20260709 13:02:43.392505  1665 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 13:02:43.392582  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.78711e+06us
I20260709 13:02:43.393676  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:43.397827  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:43.397943  5504 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:43.401484  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:43.403070  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.409540  1738 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:43.411139  1738 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:43.411574  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:43.365422  5505 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:43.398066  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:43.403786  1736 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:43.413789  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:43.414072  1738 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:43.378207  5502 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:43.377882  5392 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:43.419577  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.421993  2312 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:43.398669  5190 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.392023  5021 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:43.402150  5160 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:43.432768  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.449142  2312 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:43.434377  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.410533  5488 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:43.401394  1733 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 5. Current term is 6. Ops: []
I20260709 13:02:43.416344  5106 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:43.412101  5241 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:43.429450  5480 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.465333  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.466167  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.466495  2312 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:43.467242  2314 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:43.402033  5267 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 13:02:43.399497  5326 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:43.446003  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 409909us
I20260709 13:02:43.439935  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:43.458251  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.443339  5021 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:43.410248  4966 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 13:02:43.419701  5543 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260709 13:02:43.469080  5241 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:43.489672  2280 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.225s (SENT) [suppressed 9 similar messages]
I20260709 13:02:43.487486  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:43.480108  5021 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:43.471885  5373 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:43.476053  2309 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:43.517192  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93685.2us
I20260709 13:02:43.513803  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:43.518307  1662 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:43.511615  1733 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:43.465636  5488 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:43.447993  5190 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.440538  5504 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.447732  5481 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 13:02:43.490590  5374 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:43.508038  5499 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:43.477680  5480 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.503477  5106 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:43.514084  5241 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:43.524394  5493 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.005s
W20260709 13:02:43.544622  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111413us
W20260709 13:02:43.553828  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64129.4us
I20260709 13:02:43.563189  5190 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:43.563930  5535 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 13:02:43.566680  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.567463  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.568094  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.570333  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51775.8us
I20260709 13:02:43.573583  1587 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:43.574494  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2425) took 1313 ms. Trace:
I20260709 13:02:43.575790  1584 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:43.574955  1739 rpcz_store.cc:276] 0709 13:02:42.260676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.260797 (+   121us) service_pool.cc:224] Handling call
0709 13:02:42.261170 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.273755 (+ 12585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.273802 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:42.274445 (+   643us) log.cc:844] Serialized 26 byte log entry
0709 13:02:43.009909 (+735464us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b18003243a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:43.463535 (+453626us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:43.463580 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.463673 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.568268 (+104595us) raft_consensus.cc:1685] finished
0709 13:02:43.568291 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.574419 (+  6128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11754,"raft.queue_time_us":13025,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4951,"spinlock_wait_cycles":2464128,"thread_start_us":453398,"threads_started":1,"wal-append.queue_time_us":1289259}
I20260709 13:02:43.579068  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.579831  1585 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:43.581365  5537 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 13:02:43.581887  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.583956  5480 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:43.585575  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:43.586508  5533 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.258s	user 0.000s	sys 0.002s
W20260709 13:02:43.587415  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.588251  5411 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:43.588995  5040 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.540150  5541 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 13:02:43.590204  5326 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:43.594422  5437 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.595482  5500 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:43.595706  5437 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:43.556044  5493 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:43.598508  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.600422  5493 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:43.600618  5499 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:43.542102  5552 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 13:02:43.602499  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2343) took 1154 ms. Trace:
W20260709 13:02:43.602808  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53433.3us
I20260709 13:02:43.603559  5543 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:43.605268  5482 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.598348  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.631336  1590 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:43.631610  1668 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:43.598996  5488 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.611662  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.611979  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.613047  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.614934  1591 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:43.615530  1666 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:43.630380  1741 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:43.644227  1585 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:43.626405  1659 rpcz_store.cc:276] 0709 13:02:42.448425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.454184 (+  5759us) service_pool.cc:224] Handling call
0709 13:02:42.454496 (+   312us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.554927 (+100431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.554985 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:42.971379 (+416394us) spinlock_profiling.cc:243] Waited 3.08 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:42.971865 (+   486us) log.cc:844] Serialized 26 byte log entry
0709 13:02:43.267316 (+295451us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:43.267389 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.267494 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:43.602143 (+334649us) raft_consensus.cc:1685] finished
0709 13:02:43.602169 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:43.602434 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91741,"raft.queue_time_us":92530,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":10655,"spinlock_wait_cycles":7077248,"thread_start_us":294543,"threads_started":1,"wal-append.queue_time_us":444312}
I20260709 13:02:43.656018  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.657068  1590 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:43.656061  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:43.652248  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53021.4us
I20260709 13:02:43.669044  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.671437  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.672075  1590 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:43.672885  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.673827  1590 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:43.674636  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.675467  1590 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 13:02:43.679590  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.681830  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:43.766345  5499 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.752256  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.782383  1590 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:43.783331  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.783921  1590 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 5.
I20260709 13:02:43.784721  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:43.785250  1590 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:43.786847  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.787400  1590 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:43.685269  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:43.732975  1739 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:02:43.647665  5480 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.794796  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:43.806890  1739 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:43.814549  1587 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:43.748278  5374 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:43.820151  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 290438us
I20260709 13:02:43.682544  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:43.824106  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.825687  2312 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:43.760596  5560 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:43.833743  1737 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 13:02:43.669874  5507 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.148s
I20260709 13:02:43.750073  5503 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.088s
I20260709 13:02:43.677657  5550 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.142s
I20260709 13:02:43.857227  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.857870  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:43.859448  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.849594  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:43.862308  1587 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:43.853770  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:43.856658  5555 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:43.864471  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44760.7us
I20260709 13:02:43.865727  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:43.804999  5267 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 13:02:43.858582  2315 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:43.869112  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.874642  5550 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:43.875459  5550 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:43.880163  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:43.869225  5500 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:43.869347  5326 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:43.882635  5430 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:43.882727  5326 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:43.883198  5430 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:43.883311  5326 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:43.883653  2312 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:43.884245  5430 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.864825  5555 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.887703  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:43.896317  1739 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:43.897554  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.809855  5353 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.810340  5480 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.876508  5550 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.907667  1590 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:43.926177  1590 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:43.818353  5558 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 13:02:43.913643  5430 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:43.934131  1590 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260709 13:02:43.927193  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:43.940404  1590 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 3
I20260709 13:02:43.940812  1590 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 13:02:43.941241  1590 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:43.841460  5374 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.899220  5397 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:43.963999  1587 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:02:43.965663  1587 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:43.965752  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.968611  1587 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:43.968721  5462 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.969308  5462 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.969998  1587 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:43.970402  1587 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:43.971204  1587 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:43.971870  5462 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:43.974483  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:43.847656  5560 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:43.962028  5266 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:43.947510  1590 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:43.960415  5118 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:43.937860  5493 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:43.996081  1737 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:43.944430  5500 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:43.982075  5551 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:43.962971  5552 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:43.988040  5550 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:44.012267  5565 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 13:02:44.002301  5146 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.055289  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:44.012614  5560 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:44.018440  5502 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.584s	user 0.007s	sys 0.006s
I20260709 13:02:44.025906  5567 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:43.975383  5462 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:44.031507  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57149.3us
W20260709 13:02:44.032600  1662 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.004s
I20260709 13:02:44.063352  1662 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:43.980957  1659 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:44.043638  5392 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:43.962221  5267 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:44.038964  1733 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.520s	user 0.012s	sys 0.002s
I20260709 13:02:44.067503  1733 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:44.042861  5493 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:44.044406  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 156825us
W20260709 13:02:44.058522  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 96238.7us
I20260709 13:02:44.072160  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:43.945066  5481 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 13:02:44.038506  5118 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.065126  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:44.033087  5482 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:44.043969  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78341.9us
W20260709 13:02:44.099357  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.067288  5392 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.067222  5477 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:44.099601  5118 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.105159  5374 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.073858  5493 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:44.101658  1634 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:44.047037  5266 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.064805  5146 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.066634  5462 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:44.128014  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65273.6us
W20260709 13:02:44.116452  5505 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.693s	user 0.005s	sys 0.009s
I20260709 13:02:44.131886  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:44.109444  5392 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:44.084352  5500 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:44.107928  5437 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.109290  5477 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.129016  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:44.127826  5436 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:02:44.134538  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.108513  5267 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:44.150559  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.117357  5566 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:44.155300  5021 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.000s
I20260709 13:02:44.155793  5021 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.149030  5374 raft_consensus.cc:2764] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:02:44.148988  5437 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.158432  1662 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [8.4-10.5]
I20260709 13:02:44.159873  5437 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.160104  1662 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 13:02:44.160581  1662 pending_rounds.cc:85] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:02:44.160838  1662 pending_rounds.cc:107] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 13:02:44.161100  1662 raft_consensus.cc:2889] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:44.149823  5436 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:44.162976  5241 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.000s
W20260709 13:02:44.147645  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.163391  5480 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.163470  5241 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.140249  5499 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:44.068408  5552 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.151115  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.167861  1660 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:02:44.132944  5562 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:44.154320  5266 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:44.154484  5579 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:02:44.144090  5477 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:44.162544  5436 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.189990  5436 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:44.190637  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.191378  1662 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:44.192438  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.194139  5566 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:44.195178  5146 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.137566  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:44.153443  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.172323  1667 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:44.120457  5560 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:44.167030  5552 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.169670  5505 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.178974  5374 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.185436  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.186443  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 566) took 1273 ms. Trace:
W20260709 13:02:44.197415  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60005.4us
I20260709 13:02:44.164167  5023 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:44.161181  5326 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.205364  1660 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:44.205912  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:44.210762  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:44.205365  1663 rpcz_store.cc:276] 0709 13:02:42.913001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.913698 (+   697us) service_pool.cc:224] Handling call
0709 13:02:42.914097 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.916293 (+  2196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.916322 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:42.918955 (+  2633us) log.cc:844] Serialized 26 byte log entry
0709 13:02:43.886448 (+967493us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b1800251e00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:44.118657 (+232209us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:44.118703 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.118778 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:44.186093 (+ 67315us) raft_consensus.cc:1685] finished
0709 13:02:44.186118 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:44.186371 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1769,"raft.queue_time_us":4483,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3468,"spinlock_wait_cycles":2654464,"thread_start_us":231988,"threads_started":1,"wal-append.queue_time_us":1265033}
I20260709 13:02:44.216679  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:44.207873  5374 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.222038  5374 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.222611  5374 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.223361  5182 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.889s	user 0.012s	sys 0.000s
I20260709 13:02:44.224200  5267 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.223071  5374 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.228284  2310 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:02:44.228770  2310 pending_rounds.cc:85] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 8
I20260709 13:02:44.229333  2310 pending_rounds.cc:107] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260709 13:02:44.229642  2310 raft_consensus.cc:2889] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:44.195233  5566 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.204008  5578 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:44.268162  5578 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 13:02:44.269433  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.269680  5578 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:44.228499  5241 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.226177  4734 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.240012  5374 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.242563  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:44.283167  2312 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.282852  4734 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.284999  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.242316  5118 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.285853  5583 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:44.230948  5562 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:44.286588  5562 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:44.286437  5583 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.287680  5562 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.288112  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.228125  5498 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.289553  5498 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.228832  5552 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:44.291829  5552 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:44.288841  5583 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.230775  5584 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.295476  5584 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:44.256067  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1742) took 2543 ms (client timeout 1809 ms). Trace:
I20260709 13:02:44.256470  5373 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:44.309531  5373 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:44.309360  1734 rpcz_store.cc:269] 0709 13:02:41.712669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.742719 (+ 30050us) service_pool.cc:224] Handling call
0709 13:02:44.255755 (+2513036us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540039e4d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:44.255951 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2385338112}
I20260709 13:02:44.312790  2312 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:44.263010  5477 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.236493  5397 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:02:44.235486  5326 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:44.265002  5353 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.266604  5504 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.285269  4734 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.292398  5552 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:44.208541  5505 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.219493  5023 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:44.145452  5500 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.266497  5499 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:02:44.318039  5106 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.002s
I20260709 13:02:44.360562  5477 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.366856  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.393059  1733 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:44.387108  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.406425  1742 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 13:02:44.408347  5106 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:44.411810  5500 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.273149  5493 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.433013  5118 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:44.436720  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:44.432363  5392 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.440630  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:44.446146  2315 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.436812  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.453851  1733 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:44.451628  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.451929  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:44.452124  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.465152  1663 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:44.474370  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.475418  1733 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:44.476907  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.715925  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.857039  5578 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.697232  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:44.857861  5578 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:44.858815  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 161744us
I20260709 13:02:44.859812  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:44.860422  5578 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.478242  5118 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:44.513413  5583 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:44.581437  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.861388  5118 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.861855  5583 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.716240  1661 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:44.763886  5595 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.862234  5593 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 13:02:44.746650  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.713163  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.490149  5326 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.866387  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.865757  5597 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:44.866683  5326 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.620179  5552 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.857060  1733 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:44.869402  5591 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 13:02:44.533007  5504 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.874193  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2461) took 1119 ms. Trace:
I20260709 13:02:44.864544  5595 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.557798  5505 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.874841  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:44.710980  1479 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.242s	user 0.002s	sys 0.012s
I20260709 13:02:44.763325  5498 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.479401  5353 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:44.876425  5597 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:44.639708  5182 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.861440  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.434685  5584 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.677803  2315 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:44.530428  5566 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.867789  1736 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:44.914265  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.866952  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.513830  1734 raft_consensus.cc:2368] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
W20260709 13:02:44.856431  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:44.864346  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.915830  1661 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.878903  1590 rpcz_store.cc:276] 0709 13:02:43.754630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.034588 (+279958us) service_pool.cc:224] Handling call
0709 13:02:44.035097 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:44.132749 (+ 97652us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:44.207436 (+ 74687us) log.cc:844] Serialized 14 byte log entry
0709 13:02:44.478237 (+270801us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:44.533209 (+ 54972us) log.cc:844] Serialized 26 byte log entry
0709 13:02:44.533425 (+   216us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:44.533475 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:44.533597 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:44.873974 (+340377us) raft_consensus.cc:1685] finished
0709 13:02:44.874001 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:44.874134 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":85001,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":26898,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":14748,"thread_start_us":317313,"threads_started":3,"wal-append.queue_time_us":653034}
I20260709 13:02:44.917058  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:44.917690  1590 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:44.576695  5373 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.871157  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:44.689918  5146 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.864228  5583 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.717528  5500 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.891683  1662 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:44.556777  5392 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:44.936442  5500 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.938787  5326 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.913725  5353 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:44.940016  5353 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.940536  1661 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:44.941675  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.942880  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.943615  1661 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:44.944432  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.888041  5504 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:44.945811  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.946908  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.533166  5482 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:44.895519  1663 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:44.947571  5482 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:44.948277  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.949420  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.949682  5482 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.864316  5118 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.904955  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.951186  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.951470  5555 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.952330  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:44.952919  1667 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:44.905674  5412 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:44.907217  4965 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.912051  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59800 (request call id 7) took 1964 ms. Trace:
I20260709 13:02:44.888033  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:44.954360  1733 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.919564  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:44.954120  2316 rpcz_store.cc:276] 0709 13:02:42.950265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.988258 (+ 37993us) service_pool.cc:224] Handling call
0709 13:02:44.911572 (+1923314us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b5400570e20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf5a520 00007fcdedfd7652 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:44.911982 (+   410us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":962316672}
I20260709 13:02:44.955566  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:44.929577  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.956494  1660 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:44.956585  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:44.923647  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52607.9us
I20260709 13:02:44.921906  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:44.930786  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.959144  1662 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:44.928637  5584 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:44.934222  5595 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:44.935582  5552 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:44.950556  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:44.930658  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:44.992476  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61922.5us
I20260709 13:02:44.984891  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:44.984616  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62774.6us
I20260709 13:02:44.966285  1733 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:44.988229  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:44.989468  5583 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:44.989584  5591 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:44.957967  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.001844  1667 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
W20260709 13:02:45.002561  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44668.6us
I20260709 13:02:44.922489  5267 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:44.991175  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 131499us
I20260709 13:02:44.986478  5412 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:45.004920  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.987147  5584 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.011250  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.003346  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.014503  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:44.991657  5505 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.008245  5477 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:45.011101  1733 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:44.924113  5182 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.013175  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:44.952734  5146 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.015465  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:44.947203  5504 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:44.992751  5392 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.019483  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:44.997052  5578 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:45.020268  5477 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.021168  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.021657  5595 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:44.924924  5566 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.037819  5566 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.037987  5182 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.028570  5599 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:44.990623  4965 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.038947  5477 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.018455  5584 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.032222  5392 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.038465  5578 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.030278  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:45.058355  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 560) took 3878 ms (client timeout 2225 ms). Trace:
I20260709 13:02:45.021364  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.042912  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:45.059057  1665 rpcz_store.cc:269] 0709 13:02:41.172441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:41.173627 (+  1186us) service_pool.cc:224] Handling call
0709 13:02:45.050224 (+3876597us) spinlock_profiling.cc:243] Waited 2.19 s on lock 0x7b5400075dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.050395 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5043277696}
I20260709 13:02:45.059932  1584 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:45.017812  5600 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 13:02:45.063643  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.052981  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2467) took 1249 ms. Trace:
I20260709 13:02:45.026423  5522 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:45.010406  5591 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.061022  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.066915  1584 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.050945  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2464) took 1366 ms. Trace:
I20260709 13:02:45.062327  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.040479  4685 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:45.060834  5595 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.028937  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:45.060354  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 44978.7us
I20260709 13:02:45.064265  1740 rpcz_store.cc:276] 0709 13:02:43.803203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.065621 (+262418us) service_pool.cc:224] Handling call
0709 13:02:44.067773 (+  2152us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.141005 (+ 73232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:44.141038 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:44.141086 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:44.141125 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.052683 (+911558us) spinlock_profiling.cc:243] Waited 73 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:45.052741 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.052912 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":167882496}
I20260709 13:02:45.067852  1737 rpcz_store.cc:276] 0709 13:02:43.684162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.853420 (+169258us) service_pool.cc:224] Handling call
0709 13:02:45.050715 (+1197295us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.050840 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":303690624}
I20260709 13:02:45.072703  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.013324  5267 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.049871  4965 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.025995  5505 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.077565  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:45.077961  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49113.8us
I20260709 13:02:45.075721  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.078830  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.078926  1733 raft_consensus.cc:2376] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11. Re-sending same reply.
I20260709 13:02:45.079871  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.061550  5578 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.079476  1740 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:45.083654  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.080478  1733 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:45.084208  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.084955  1740 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:45.084853  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.088896  1736 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:45.089244  1733 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:45.091127  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.091749  1737 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:45.092514  5505 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.093104  5505 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:45.074826  5599 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.094421  5599 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.088240  5353 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:45.096000  4965 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:45.096446  4965 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.096905  4965 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.097671  5595 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.101707  5267 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.103042  5146 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.103204  5600 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.065264  5118 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:45.104038  5118 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.103765  5600 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.104976  5118 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.106516  5600 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.107129  5353 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.069242  4685 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:45.108178  5353 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.108301  4685 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.111346  5392 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.113160  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.071372  5503 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:45.114575  5503 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.114965  5503 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:45.120306  5600 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.120854  5600 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.123142  5600 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.066030  5522 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.138628  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.139495  1591 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:02:45.127410  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54830.9us
I20260709 13:02:45.140846  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.128269  5600 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:45.148157  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.152882  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.167047  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.167702  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.142760  4685 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.174252  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.171578  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.144654  5522 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:45.117208  4965 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.199431  5539 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:45.200127  5539 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:45.207228  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 576) took 1473 ms. Trace:
I20260709 13:02:45.207916  1659 rpcz_store.cc:276] 0709 13:02:43.733644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.733816 (+   172us) service_pool.cc:224] Handling call
0709 13:02:43.734255 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.066137 (+331882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.206821 (+1140684us) spinlock_profiling.cc:243] Waited 246 ms on lock 0x7b54002f2954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:45.207129 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":566918016}
I20260709 13:02:45.210443  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 561) took 2908 ms. Trace:
I20260709 13:02:45.210907  1735 rpcz_store.cc:276] 0709 13:02:42.302440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.308133 (+  5693us) service_pool.cc:224] Handling call
0709 13:02:43.377621 (+1069488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:43.378469 (+   848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.210188 (+1831719us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400404050. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.210342 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2459158400}
I20260709 13:02:45.212469  5190 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 13:02:45.212960  5190 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.213418  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.214171  2316 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:45.213975  5190 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.215394  5574 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:45.215590  5190 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:45.216153  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.216066  5574 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.216877  1585 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:45.212954  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 820) took 1042 ms. Trace:
I20260709 13:02:45.228188  2310 rpcz_store.cc:276] 0709 13:02:44.170380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.170594 (+   214us) service_pool.cc:224] Handling call
0709 13:02:44.200907 (+ 30313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:44.230080 (+ 29173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.212649 (+982569us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b54004e61d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:45.212879 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57834240}
I20260709 13:02:45.149421  5500 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.150866  5540 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:45.247246  5540 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.247869  5540 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.177829  5551 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:45.260120  4965 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.265483  5574 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.271557  5431 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:45.272146  5431 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.273092  5431 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.274283  5431 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:45.275607  1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 536) took 2313 ms (client timeout 1992 ms). Trace:
W20260709 13:02:45.276176  1586 rpcz_store.cc:269] 0709 13:02:42.961786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.962042 (+   256us) service_pool.cc:224] Handling call
0709 13:02:45.275188 (+2313146us) spinlock_profiling.cc:243] Waited 6.52 ms on lock 0x7b5400443754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.275454 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14989056}
I20260709 13:02:45.278263  1587 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.278673  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.279549  1586 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:45.281828  1592 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:45.282611  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.284461  4966 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:45.285195  4966 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.145272  5505 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.286270  4966 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.287714  5505 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.284529  5482 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:45.291067  5482 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.175275  5504 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:45.292286  5482 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.292915  5504 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:45.295472  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2423) took 3035 ms. Trace:
I20260709 13:02:45.298278  1742 rpcz_store.cc:276] 0709 13:02:42.260125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.260260 (+   135us) service_pool.cc:224] Handling call
0709 13:02:44.256597 (+1996337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:44.406996 (+150399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.288992 (+881996us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540039e4d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:45.295176 (+  6184us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b540039e4d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.295387 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4920009856}
I20260709 13:02:45.178514  5373 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.169358  5326 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:45.211349  5591 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.309019  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.315824  1585 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:45.311944  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:45.322167  1735 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:45.310196  1661 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:45.321640  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.327260  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.328631  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.332463  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.317986  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.334045  1585 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:45.309968  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.335292  1586 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:45.335510  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.337903  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.338168  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.338757  1586 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:45.367383  1557 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.960s (SENT)
I20260709 13:02:45.368880  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.368893  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.370440  1737 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:45.371196  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.371469  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.371939  1740 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:45.319737  4906 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.372171  1737 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:45.377921  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.344960  1584 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:45.402420  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.403028  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:45.400100  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.403741  1584 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.402051  1740 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:45.405874  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.406291  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.378542  1734 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:45.401139  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.344563  1587 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 13:02:45.407578  1587 pending_rounds.cc:85] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:02:45.407830  1587 pending_rounds.cc:107] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 13:02:45.408063  1587 raft_consensus.cc:2889] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:45.402925  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.408790  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2460) took 1677 ms. Trace:
I20260709 13:02:45.409137  1587 rpcz_store.cc:276] 0709 13:02:43.731558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.969575 (+238017us) service_pool.cc:224] Handling call
0709 13:02:43.969860 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.277728 (+1307868us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b58008fbad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:02:45.408537 (+130809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.408725 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2998870400}
I20260709 13:02:45.326148  5505 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.412228  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.324620  5482 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.410637  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.413702  1742 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:45.422802  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.319693  5501 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.339802  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.426285  1585 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:45.378422  5373 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.424279  1587 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
W20260709 13:02:45.433887  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66573.7us
I20260709 13:02:45.405833  5392 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.419986  1584 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:45.380187  5591 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.320591  5502 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260709 13:02:45.405212  5574 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:45.431121  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.422482  5405 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:45.431275  5505 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.440073  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.404808  5504 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.201356  5539 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.428982  5551 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.429049  4965 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.432765  5326 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.441781  5597 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.000s
I20260709 13:02:45.412094  4906 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.453487  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.442881  5574 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.429030  5412 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.444420  5604 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.439419  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.506825  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.507751  1584 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:45.459172  1587 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:45.440601  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.437831  5373 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.436441  5480 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 13:02:45.442857  5392 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.463797  5578 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:45.469636  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.502612  5574 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:45.502043  5597 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:45.461586  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.528667  1585 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:45.514905  5560 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.476780  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 154166us
I20260709 13:02:45.505856  5610 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.487342  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:45.472903  5584 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:45.502148  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48798.8us
I20260709 13:02:45.537843  1586 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:45.539368  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.488220  4965 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.486297  5539 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:45.509126  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.490634  5326 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.561604  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.562577  1661 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:02:45.519119  1740 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:45.512915  5583 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.004s
I20260709 13:02:45.564164  5583 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:45.458135  5602 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.565028  5602 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.566645  1590 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:45.501693  5502 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:45.567423  5502 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.531836  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.569252  1585 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:45.570142  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2521) took 1122 ms. Trace:
I20260709 13:02:45.570501  1585 rpcz_store.cc:276] 0709 13:02:44.447362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.531489 (+1084127us) service_pool.cc:224] Handling call
0709 13:02:45.570064 (+ 38575us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.571650  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.572362  1585 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:45.572876  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2520) took 1125 ms. Trace:
I20260709 13:02:45.481019  5477 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.444562  5591 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:45.515420  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74878.4us
I20260709 13:02:45.576318  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:45.577373  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.579483  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.580863  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.488384  5551 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.469638  5522 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.554665  1479 maintenance_manager.cc:643] P e6b16382849d482ba544fc5ee58e0b0a: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 7.401s	user 0.204s	sys 0.122s Metrics: {"bytes_written":30185,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":322418,"dirs.run_cpu_time_us":635,"dirs.run_wall_time_us":88963,"drs_written":1,"lbm_read_time_us":535,"lbm_reads_lt_1ms":4,"lbm_write_time_us":124679,"lbm_writes_1-10_ms":2,"lbm_writes_10-100_ms":4,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":315702,"threads_started":2,"update_count":10,"wal-append.queue_time_us":240813}
I20260709 13:02:45.580955  5405 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:45.583006  5405 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.583446  5405 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:45.583657  5551 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.584270  5551 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:45.584838  5551 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:45.542655  1587 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:45.585566  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2513) took 1156 ms. Trace:
I20260709 13:02:45.585892  1587 rpcz_store.cc:276] 0709 13:02:44.428869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.439772 (+1010903us) service_pool.cc:224] Handling call
0709 13:02:45.585478 (+145706us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.586877  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.587627  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2529) took 1028 ms. Trace:
I20260709 13:02:45.587929  1587 rpcz_store.cc:276] 0709 13:02:44.559303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.586564 (+1027261us) service_pool.cc:224] Handling call
0709 13:02:45.587569 (+  1005us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.588783  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.519210  5595 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:45.591884  1480 maintenance_manager.cc:419] P e6b16382849d482ba544fc5ee58e0b0a: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7463 bytes on disk
I20260709 13:02:45.591789  5405 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.595598  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.596395  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:45.598707  1479 maintenance_manager.cc:643] P e6b16382849d482ba544fc5ee58e0b0a: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.006s	user 0.002s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":537,"lbm_reads_lt_1ms":4}
W20260709 13:02:45.601068  5605 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 13:02:45.597211  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.603061  5584 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.603368  1584 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.603775  5584 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.604035  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.604300  5584 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:45.606889  5612 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:02:45.607261  1660 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
W20260709 13:02:45.608057  5601 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.531s	user 0.005s	sys 0.000s
I20260709 13:02:45.608343  5505 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.608727  1660 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:45.608937  5505 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.609540  5505 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.611586  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.611901  5505 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.612553  5505 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:45.612901  5505 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.613030  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.613837  1665 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:02:45.614092  5606 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:02:45.615044  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.615653  1663 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:45.589360  1587 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:45.512661  5604 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.618247  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2519) took 1171 ms. Trace:
I20260709 13:02:45.618795  1587 rpcz_store.cc:276] 0709 13:02:44.446749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.588544 (+1141795us) service_pool.cc:224] Handling call
0709 13:02:45.618081 (+ 29537us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.620074  5604 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.620615  5604 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.547252  5610 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.605247  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:45.620762  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.622623  5373 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.622901  1585 rpcz_store.cc:276] 0709 13:02:44.447067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.571332 (+1124265us) service_pool.cc:224] Handling call
0709 13:02:45.572817 (+  1485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.623581  5610 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.624205  1585 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:45.624867  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2462) took 1863 ms. Trace:
I20260709 13:02:45.622742  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2518) took 1176 ms. Trace:
I20260709 13:02:45.626937  1587 rpcz_store.cc:276] 0709 13:02:44.446460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.620412 (+1173952us) service_pool.cc:224] Handling call
0709 13:02:45.622666 (+  2254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.533555  5611 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 13:02:45.633792  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53002.1us
I20260709 13:02:45.621712  5583 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:45.624123  5412 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:45.624734  5503 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.001s
I20260709 13:02:45.535640  5560 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.613306  5505 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:45.627625  5405 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:45.636127  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 96863.8us
I20260709 13:02:45.644340  1587 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:45.621061  5604 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.625218  1585 rpcz_store.cc:276] 0709 13:02:43.761893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.623704 (+1861811us) service_pool.cc:224] Handling call
0709 13:02:45.624000 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.624632 (+   632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.624813 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.635159  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.675958  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 545) took 1899 ms. Trace:
I20260709 13:02:45.680146  5583 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.674324  4906 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.675639  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.747457  5578 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.755189  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2358) took 2822 ms. Trace:
I20260709 13:02:45.756410  5560 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.755664  1664 rpcz_store.cc:276] 0709 13:02:42.932576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.932736 (+   160us) service_pool.cc:224] Handling call
0709 13:02:42.933288 (+   552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:42.933742 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:42.933778 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:43.393598 (+459820us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:43.416945 (+ 23347us) log.cc:844] Serialized 26 byte log entry
0709 13:02:43.615567 (+198622us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:43.615619 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:43.615709 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:45.753088 (+2137379us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54001da4a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52dbe 00007fcdedf5d930 00007fcdedf5d8d8 00007fcdedf5d858 00007fcdedf5d813 00007fcdedf5ca9c 00007fcdf3e48626 00007fcdf3e42dd8 00007fcdedf997cb 00007fcdedf9c2c0
0709 13:02:45.754736 (+  1648us) raft_consensus.cc:1685] finished
0709 13:02:45.754924 (+   188us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.755113 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3987,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":134,"spinlock_wait_cycles":77716096,"thread_start_us":101785,"threads_started":1,"wal-append.queue_time_us":144180}
I20260709 13:02:45.709808  1584 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:45.689687  5616 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.679715  5600 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.533s	user 0.006s	sys 0.004s
I20260709 13:02:45.670225  5021 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:45.643718  5613 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 13:02:45.675468  5584 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:45.782307  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107232us
I20260709 13:02:45.732923  1587 rpcz_store.cc:276] 0709 13:02:43.776132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.643797 (+1867665us) service_pool.cc:224] Handling call
0709 13:02:45.644163 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.654651 (+ 10488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.675725 (+ 21074us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.733834  5431 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.736157  4966 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:45.733152  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98141.6us
I20260709 13:02:45.758769  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2528) took 1200 ms. Trace:
I20260709 13:02:45.787966  1584 rpcz_store.cc:276] 0709 13:02:44.558633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.596967 (+1038334us) service_pool.cc:224] Handling call
0709 13:02:45.758701 (+161734us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.760661  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.788327  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.763273  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.690711  5480 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.782085  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:45.782274  2310 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:45.643522  5503 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.783527  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.707816  5540 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.748171  5106 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:45.789494  1742 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:45.789832  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.792025  1584 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:45.806169  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2463) took 2008 ms. Trace:
I20260709 13:02:45.806676  1584 rpcz_store.cc:276] 0709 13:02:43.797947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.791620 (+1993673us) service_pool.cc:224] Handling call
0709 13:02:45.791888 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.805716 (+ 13828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.806042 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.808179  1584 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:45.796336  2314 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
W20260709 13:02:45.799306  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.799597  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:45.804308  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:45.810861  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.812336  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:45.814119  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.780058  5600 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:45.784601  4906 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.803123  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.784894  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.821276  2312 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:45.821331  1742 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:45.796944  5405 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:45.823480  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:45.810099  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 549) took 2011 ms. Trace:
I20260709 13:02:45.825415  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.825773  1584 rpcz_store.cc:276] 0709 13:02:43.798592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.807655 (+2009063us) service_pool.cc:224] Handling call
0709 13:02:45.808008 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.809738 (+  1730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.809953 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.795171  5618 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:45.828227  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.828186  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:45.829383  2310 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:45.832477  2312 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:45.829867  1591 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:45.802186  5584 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:45.834630  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2464) took 1563 ms. Trace:
I20260709 13:02:45.803104  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:45.830904  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.870880  1742 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.871065  2309 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:45.871403  1740 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 13:02:45.815585  5412 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:45.872135  5412 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.872592  5412 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:45.830283  5610 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:45.875180  5610 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.875594  5610 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.797253  4966 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.831346  5106 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 13:02:45.836421  5542 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:02:45.835680  4906 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.879937  4966 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.881161  4906 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:45.851536  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.884893  5591 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.887279  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2465) took 1615 ms. Trace:
I20260709 13:02:45.887849  5502 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.888626  5405 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.889635  5405 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.891252  5505 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.891877  5405 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.892573  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.893549  5505 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.894086  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:45.895179  5553 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:45.895569  5553 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.895916  5553 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:45.897871  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.898747  5540 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.854624  5604 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.856550  5503 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.831897  5373 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:45.846822  5600 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:45.904011  5373 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:45.904433  5600 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:45.904631  5373 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:45.904934  5600 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:45.857899  5619 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:45.858189  2315 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:45.822170  5617 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:45.860257  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 546) took 2083 ms. Trace:
I20260709 13:02:45.860714  4965 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:45.817625  5611 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:45.779913  5616 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.818228  5597 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:45.861425  5584 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.831528  5618 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:45.867393  5621 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:02:45.853096  5574 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.796762  5431 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.922384  5619 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:45.906020  5021 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:45.924283  5561 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:45.924431  5431 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.924822  5561 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.832095  5392 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.887655  1588 rpcz_store.cc:276] 0709 13:02:44.271787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.886383 (+1614596us) service_pool.cc:224] Handling call
0709 13:02:45.886756 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.886954 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.886981 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.887024 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:45.887055 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.887107 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.887217 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:45.925406  5021 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.934659  5021 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.935480  5602 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:45.937079  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.938067  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.938485  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:45.926990  5616 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.927106  5574 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:45.942560  5462 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.905902  4966 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:45.945981  4966 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:45.946916  4966 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:45.949388  5584 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.951579  5617 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:45.952474  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1757) took 1483 ms (client timeout 1770 ms). Trace:
I20260709 13:02:45.952600  5617 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:45.953053  1736 rpcz_store.cc:269] 0709 13:02:44.469189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.007141 (+537952us) service_pool.cc:224] Handling call
0709 13:02:45.952242 (+945101us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b54002030d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.952397 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59631360}
I20260709 13:02:45.954768  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:45.848524  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 563) took 3537 ms (client timeout 2183 ms). Trace:
I20260709 13:02:45.845101  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.956054  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.846017  2314 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:45.847780  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.956977  1736 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:45.957350  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.957868  1737 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 6: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:45.958653  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:45.960661  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 570) took 3033 ms (client timeout 1799 ms). Trace:
W20260709 13:02:45.961108  1668 rpcz_store.cc:269] 0709 13:02:42.927604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.927753 (+   149us) service_pool.cc:224] Handling call
0709 13:02:45.960483 (+3032730us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b540030b954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.960589 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1613819648}
I20260709 13:02:45.849844  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.930017  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.962915  1588 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:45.963366  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 574) took 2474 ms. Trace:
I20260709 13:02:45.963712  1666 rpcz_store.cc:276] 0709 13:02:43.488741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.488892 (+   151us) service_pool.cc:224] Handling call
0709 13:02:43.493433 (+  4541us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:43.635965 (+142532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.963114 (+2327149us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:45.963307 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":262885632}
I20260709 13:02:45.964920  5493 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:45.955745  1741 rpcz_store.cc:269] 0709 13:02:42.316260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.316392 (+   132us) service_pool.cc:224] Handling call
0709 13:02:45.848323 (+3531931us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540012ded4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.848446 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2982723840}
I20260709 13:02:45.966183  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2525) took 1491 ms. Trace:
I20260709 13:02:45.966547  1586 rpcz_store.cc:276] 0709 13:02:44.474532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.339555 (+865023us) service_pool.cc:224] Handling call
0709 13:02:45.966007 (+626452us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400036bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.966112 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":246758272}
I20260709 13:02:45.967386  2314 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:45.967571  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:45.969216  5411 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:45.969884  5411 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.972641  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.973325  2310 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:45.974960  5241 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:45.975389  5241 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.956936  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.976658  1739 raft_consensus.cc:2368] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:45.977064  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.977466  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:45.977766  1738 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:45.978219  1739 raft_consensus.cc:2368] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:45.978642  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.979069  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.979354  1738 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:45.968341  2314 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:45.980572  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2444) took 2369 ms. Trace:
I20260709 13:02:45.981047  2314 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:45.980934  1592 rpcz_store.cc:276] 0709 13:02:43.611371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:43.612314 (+   943us) service_pool.cc:224] Handling call
0709 13:02:45.281618 (+1669304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.282298 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.980361 (+698063us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400443750. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.980485 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3838484224}
I20260709 13:02:45.982245  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:45.982602  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.983165  2310 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 5.
I20260709 13:02:45.983539  1592 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:45.984167  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2470) took 1703 ms. Trace:
W20260709 13:02:45.867184  1590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2466) took 1595 ms (client timeout 1846 ms). Trace:
I20260709 13:02:45.910542  1585 rpcz_store.cc:276] 0709 13:02:43.776445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.649747 (+1873302us) service_pool.cc:224] Handling call
0709 13:02:45.745887 (+ 96140us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:45.746330 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.746365 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:45.747041 (+   676us) log.cc:844] Serialized 26 byte log entry
0709 13:02:45.780525 (+ 33484us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:45.780574 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.780682 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:45.860006 (+ 79324us) raft_consensus.cc:1685] finished
0709 13:02:45.860034 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.860190 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1616,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":25411,"thread_start_us":33005,"threads_started":1,"wal-append.queue_time_us":111127}
I20260709 13:02:45.883028  5477 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:45.998373  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.914721  1742 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:45.917661  5611 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:45.918433  5618 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:45.846086  5543 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 13:02:45.918329  5597 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:45.898904  5560 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 13:02:45.798517  5480 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:45.962318  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 112601us
I20260709 13:02:46.110641  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:45.984568  1592 rpcz_store.cc:276] 0709 13:02:44.281146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.983085 (+1701939us) service_pool.cc:224] Handling call
0709 13:02:45.983380 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.983904 (+   524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.984105 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:45.984731  1590 rpcz_store.cc:269] 0709 13:02:44.272066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:44.984505 (+712439us) service_pool.cc:224] Handling call
0709 13:02:45.866899 (+882394us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:45.867090 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1313739520}
W20260709 13:02:45.993568  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:45.994107  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:46.112510  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.115181  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:45.994602  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.115885  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.996511  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.997869  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.117297  1739 raft_consensus.cc:2368] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:45.786975  5583 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.112036  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.124727  1592 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 5.
I20260709 13:02:46.124883  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.000537  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:46.126214  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125788us
I20260709 13:02:46.000824  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:46.005558  5616 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.004873  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.136443  1585 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:46.136587  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.137632  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:46.013754  2313 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:02:46.137638  1742 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:46.016306  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.139468  2312 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:46.031147  1588 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:46.035187  2314 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 13:02:46.141073  2314 pending_rounds.cc:85] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 7
I20260709 13:02:46.141409  2314 pending_rounds.cc:107] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 13:02:46.143162  2314 raft_consensus.cc:2889] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:46.036204  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.975778  5241 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:46.147342  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.033906  5611 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.042918  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 542) took 3079 ms. Trace:
I20260709 13:02:46.103511  5597 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.045188  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.158051  2315 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:46.045521  5373 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:46.050734  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2517) took 1604 ms (client timeout 1923 ms). Trace:
W20260709 13:02:46.159541  1591 rpcz_store.cc:269] 0709 13:02:44.446193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.171294 (+725101us) service_pool.cc:224] Handling call
0709 13:02:46.050492 (+879198us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b540031ea54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:46.050649 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1503012608}
I20260709 13:02:46.051004  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:46.052695  5610 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.055428  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.033725  5602 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:46.067569  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.175216  1736 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:46.033298  5548 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:02:46.070812  5561 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.071599  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:46.180435  1738 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:46.100888  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.102742  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.109057  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.033130  5581 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.121s
I20260709 13:02:45.929090  5326 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:46.004285  4685 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.123605  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:45.837260  1584 rpcz_store.cc:276] 0709 13:02:44.271496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.833799 (+1562303us) service_pool.cc:224] Handling call
0709 13:02:45.834113 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.834306 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.834338 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.834382 (+    44us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:45.834420 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:45.834473 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:45.834573 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.147279  4965 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:46.133677  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:46.146432  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:46.041728  5522 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:46.147784  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:46.123783  5503 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
I20260709 13:02:46.150830  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:45.937704  5619 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:46.154189  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 328855us
I20260709 13:02:46.207214  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:46.157193  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2471) took 1875 ms. Trace:
I20260709 13:02:46.208107  1590 rpcz_store.cc:276] 0709 13:02:44.281433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.131978 (+1850545us) service_pool.cc:224] Handling call
0709 13:02:46.156680 (+ 24702us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.156875 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.156897 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:46.156941 (+    44us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:46.156971 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.157036 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.157138 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.163303  1588 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:46.209872  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2474) took 1878 ms. Trace:
I20260709 13:02:46.210309  1588 rpcz_store.cc:276] 0709 13:02:44.332052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.140867 (+1808815us) service_pool.cc:224] Handling call
0709 13:02:46.162960 (+ 22093us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.209536 (+ 46576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.209811 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.164041  1587 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:46.167595  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.169363  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.116096  1586 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:46.182654  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.212797  1741 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:46.214484  1741 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 13:02:46.216275  1741 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:46.182854  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.153591  1593 rpcz_store.cc:276] 0709 13:02:42.963686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:42.969967 (+  6281us) service_pool.cc:224] Handling call
0709 13:02:42.970251 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:42.978199 (+  7948us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:43.244039 (+265840us) log.cc:844] Serialized 14 byte log entry
0709 13:02:43.469911 (+225872us) log.cc:844] Serialized 14 byte log entry
0709 13:02:45.130370 (+1660459us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b180027b6e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:45.290267 (+159897us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b180027b6e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:45.290404 (+   137us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:45.290472 (+    68us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:45.290535 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.042573 (+752038us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:46.042644 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.042833 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5509,"prepare.queue_time_us":423358,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":2704,"raft.queue_time_us":123416,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":8616,"spinlock_wait_cycles":42893184,"thread_start_us":550829,"threads_started":3,"wal-append.queue_time_us":345993,"wal-append.run_cpu_time_us":1603,"wal-append.run_wall_time_us":1001617}
I20260709 13:02:46.184110  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.184957  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.182237  1737 raft_consensus.cc:2376] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11. Re-sending same reply.
I20260709 13:02:46.147400  5454 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:46.147238  5539 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:46.252357  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:46.132242  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 131522us
I20260709 13:02:46.254307  1557 leader_election.cc:400] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9.
I20260709 13:02:46.230777  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.191895  2317 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 5.
I20260709 13:02:46.231333  1664 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:46.192749  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.257131  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.257392  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.257916  1664 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.258503  1665 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:46.259632  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.260378  1665 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:46.193306  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.262440  1662 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:46.193609  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.266412  1666 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:46.194077  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.267153  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 6.
I20260709 13:02:46.147585  5549 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:46.196558  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50252us
I20260709 13:02:46.149294  2313 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:02:46.198833  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.198920  2312 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:46.255543  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:46.273034  4967 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:46.273641  4967 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.273828  5326 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.274078  4967 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:46.154255  5411 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.274434  5326 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.226748  1741 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:46.228819  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:46.276093  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47385us
I20260709 13:02:46.276918  5326 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.277598  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.156010  5412 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.156026  5023 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260709 13:02:46.191129  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2509) took 1763 ms. Trace:
I20260709 13:02:46.255470  1668 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:46.283727  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.279223  1557 leader_election.cc:400] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
I20260709 13:02:46.300678  1584 rpcz_store.cc:276] 0709 13:02:44.427563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.190038 (+1762475us) service_pool.cc:224] Handling call
0709 13:02:46.190427 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.190679 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.190718 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:46.190801 (+    83us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:46.190847 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.190920 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.191053 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.302690  1557 leader_election.cc:403] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:02:46.303560  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:46.303838  1662 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:46.307363  2312 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:46.305246  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2512) took 1876 ms. Trace:
I20260709 13:02:46.309115  1584 rpcz_store.cc:276] 0709 13:02:44.428565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.304154 (+1875589us) service_pool.cc:224] Handling call
0709 13:02:46.304574 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.304822 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.304859 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:46.304926 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:46.304972 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.305042 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.305176 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.311396  1584 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 13:02:46.314072  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45039.5us
I20260709 13:02:46.315634  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.315786  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.315685  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.154536  5493 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 13:02:46.195464  5602 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:46.317075  1667 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:46.351214  1660 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:46.356402  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.358278  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.359061  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:46.312136  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2524) took 1850 ms. Trace:
I20260709 13:02:46.331461  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 548) took 2539 ms. Trace:
I20260709 13:02:46.360044  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.203426  5190 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:46.360069  1587 rpcz_store.cc:276] 0709 13:02:43.792515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.791365 (+1998850us) service_pool.cc:224] Handling call
0709 13:02:45.791728 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.211436 (+419708us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.331125 (+119689us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:46.331377 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":855840640}
I20260709 13:02:46.350783  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.154886  5616 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.362416  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.366955  1587 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:46.281795  5539 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.201850  4965 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.253476  5498 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:46.378168  1590 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:46.318842  5411 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.363713  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.317134  1557 leader_election.cc:400] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:46.364611  1586 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:46.370915  1736 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:46.366400  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:46.361255  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 233033us
I20260709 13:02:46.313767  5503 consensus_queue.cc:1243] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:46.374920  1740 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:02:46.345216  5604 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:46.279487  5326 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:46.278556  5373 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.368443  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:46.383143  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:46.399374  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.384865  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.360177  1584 rpcz_store.cc:276] 0709 13:02:44.461489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.310539 (+1849050us) service_pool.cc:224] Handling call
0709 13:02:46.311004 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.311829 (+   825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.312069 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:46.401072  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:46.403723  1742 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:46.367916  5561 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.318992  5540 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:46.319056  5522 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:46.344872  5542 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:46.350705  5477 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:46.201190  4685 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.375675  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.414544  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.415335  4685 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:46.415627  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.415961  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.417897  5625 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:02:46.388257  1557 leader_election.cc:403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:46.394137  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.420468  1662 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 9 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:02:46.421245  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.394897  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.398530  1665 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:02:46.391018  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.430008  1659 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:46.430263  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.431555  5542 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 12
I20260709 13:02:46.397905  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:46.408787  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:46.381034  5539 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.427305  5480 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.427850  1662 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:46.388818  5190 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:46.413303  5477 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.385826  4965 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.445407  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.480943  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.481140  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.481560  1666 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:46.403107  5326 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.446313  1736 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:02:46.447144  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 130146us
W20260709 13:02:46.392953  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 259434us
W20260709 13:02:46.453122  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 326267us
I20260709 13:02:46.396102  5591 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:46.465219  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.481895  1667 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
W20260709 13:02:46.445910  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48119.6us
I20260709 13:02:46.416010  4685 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:46.492264  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.489144  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:46.491292  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:46.495693  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:46.492877  2318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.063s	sys 0.062s
I20260709 13:02:46.494154  5591 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.501345  5591 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.502650  5023 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:46.504626  5023 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:46.482195  5477 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:46.509016  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.509102  1736 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:46.511050  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.409356  5604 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.516743  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.482568  5454 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:46.522915  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.487807  5326 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.530815  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.531461  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:46.532238  1736 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:46.490521  5412 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:46.525746  5548 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:46.493628  4685 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:46.538722  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 169268us
I20260709 13:02:46.505092  5493 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:46.543412  4685 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:46.543598  5493 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:46.544237  5493 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:46.553689  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:46.542301  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 404773us
I20260709 13:02:46.527942  5454 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:46.512228  5023 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.550498  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:46.526239  5634 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:46.582916  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:46.575199  1736 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:02:46.571084  1664 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:46.596307  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.597100  2314 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:46.538280  5549 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:46.600616  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.601548  1736 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:46.568140  5610 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:46.605743  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.606348  1667 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.607311  4685 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.603888  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.615731  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.617704  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.623939  4685 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:46.626343  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.574270  5583 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.636225  1664 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:46.629541  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:46.629794  4685 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:46.632776  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:46.637291  4685 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.637341  2314 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:46.638098  4685 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.639178  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:46.640270  1736 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:46.643133  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:46.509855  5602 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.658215  1667 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:46.659268  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2510) took 2231 ms. Trace:
W20260709 13:02:46.659859  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 137045us
I20260709 13:02:46.605046  5549 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:46.670336  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:46.668736  1593 rpcz_store.cc:276] 0709 13:02:44.427941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.218797 (+1790856us) service_pool.cc:224] Handling call
0709 13:02:46.219162 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.219435 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:46.530740 (+311305us) log.cc:844] Serialized 14 byte log entry
0709 13:02:46.658866 (+128126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:46.658931 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:46.658981 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.659058 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.659202 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":421744,"thread_start_us":438598,"threads_started":2}
I20260709 13:02:46.673879  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.675033  1593 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 13:02:46.675592  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:46.677460  1666 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:46.679865  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 184272us
I20260709 13:02:46.629909  5021 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:46.681869  1736 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:46.631698  5599 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:46.668123  5540 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:46.557309  5190 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.646979  5504 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:46.701026  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.671743  5583 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:46.702636  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:46.640360  5645 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 13:02:46.708575  1593 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:46.694638  5241 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.532s	user 0.014s	sys 0.000s
I20260709 13:02:46.685357  5021 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:46.704923  5553 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.809s	user 0.016s	sys 0.001s
W20260709 13:02:46.709825  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 127012us
I20260709 13:02:46.750308  1736 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:46.704972  5583 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:46.668747  5638 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
W20260709 13:02:46.716270  5627 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260709 13:02:46.764403  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 131742us
I20260709 13:02:46.699965  4967 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.781558  5604 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:46.783735  5628 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.274s	user 0.001s	sys 0.000s
I20260709 13:02:46.784529  5616 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:46.767246  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:46.787014  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.748229  5021 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:46.765023  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 165028us
W20260709 13:02:46.799186  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 221703us
I20260709 13:02:46.757220  5462 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:46.761758  5583 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:46.770000  5626 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.273s	user 0.001s	sys 0.001s
I20260709 13:02:46.841562  5599 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:46.845163  5023 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:46.846139  5631 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 13:02:46.846256  5562 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:46.759817  5648 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.097s
W20260709 13:02:46.847215  5646 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 13:02:46.756605  5553 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:46.748060  5241 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.979497  5553 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:46.786965  5581 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:46.980706  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2477) took 2598 ms. Trace:
I20260709 13:02:46.981173  1591 rpcz_store.cc:276] 0709 13:02:44.382455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.160804 (+1778349us) service_pool.cc:224] Handling call
0709 13:02:46.161280 (+   476us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:46.177258 (+ 15978us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 13:02:46.177305 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:46.177966 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 13:02:46.558007 (+380041us) raft_consensus.cc:1645] Marking committed up to 2
0709 13:02:46.558049 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.558127 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.980436 (+422309us) raft_consensus.cc:1685] finished
0709 13:02:46.980459 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:46.980627 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15322,"raft.queue_time_us":16385,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":9423,"thread_start_us":243579,"threads_started":1,"wal-append.queue_time_us":586639}
I20260709 13:02:46.983366  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:46.984555  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:46.984587  5412 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:46.986052  1668 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:46.764961  5651 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 13:02:46.859136  5629 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.249s	user 0.001s	sys 0.001s
I20260709 13:02:46.923252  5638 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:46.707887  5454 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:46.987797  5454 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:46.946733  5648 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:02:46.947927  5657 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:46.990264  5657 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:46.990661  5657 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:46.992522  4685 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:46.997213  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 230081us
W20260709 13:02:46.955194  5639 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:02:46.923146  5023 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.000396  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 213502us
I20260709 13:02:46.971292  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:47.004179  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:46.860690  5636 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
W20260709 13:02:46.859570  5630 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.305s	user 0.000s	sys 0.001s
W20260709 13:02:46.851270  5632 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.328s	user 0.002s	sys 0.000s
I20260709 13:02:47.027586  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.999787  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:47.006019  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:46.973033  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:47.009243  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 339010us
I20260709 13:02:47.010053  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:46.860340  5635 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
W20260709 13:02:47.032948  5561 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.000s
I20260709 13:02:47.033641  1664 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:47.036758  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2472) took 2755 ms. Trace:
I20260709 13:02:47.037995  5574 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:47.037225  1592 rpcz_store.cc:276] 0709 13:02:44.281718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.125604 (+1843886us) service_pool.cc:224] Handling call
0709 13:02:46.125923 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:46.126285 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.126318 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:46.126797 (+   479us) log.cc:844] Serialized 26 byte log entry
0709 13:02:46.448332 (+321535us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:46.448394 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.448492 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.756854 (+308362us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b4800080358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:02:47.036382 (+279528us) raft_consensus.cc:1685] finished
0709 13:02:47.036411 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.036682 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5914,"raft.run_cpu_time_us":4534,"raft.run_wall_time_us":619888,"spinlock_wait_cycles":24342784,"thread_start_us":235878,"threads_started":1,"wal-append.queue_time_us":453766}
I20260709 13:02:47.039230  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.039800  1592 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:46.985265  1591 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:47.042181  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.030072  1586 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:47.043771  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.044894  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.030827  5656 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:47.031302  5662 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:02:47.011606  5604 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.029230  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:47.057544  1593 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:47.046067  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:47.062669  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.010421  5540 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:47.051167  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.065649  1586 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.047544  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:47.062973  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:47.015633  5241 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.023808  4967 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:47.054553  5591 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.553s	user 0.012s	sys 0.000s
W20260709 13:02:47.060279  5600 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.155s	user 0.022s	sys 0.019s
I20260709 13:02:47.053490  5412 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:47.054589  5480 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:02:47.079155  5493 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.534s	user 0.010s	sys 0.000s
I20260709 13:02:47.012400  5023 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.064558  5656 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:47.113850  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2473) took 2831 ms. Trace:
I20260709 13:02:47.147329  1585 rpcz_store.cc:276] 0709 13:02:44.281994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.137150 (+1855156us) service_pool.cc:224] Handling call
0709 13:02:46.137475 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:46.137905 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:46.137947 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:46.149269 (+ 11322us) log.cc:844] Serialized 26 byte log entry
0709 13:02:46.466667 (+317398us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:46.466736 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.466833 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.113553 (+646720us) raft_consensus.cc:1685] finished
0709 13:02:47.113575 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.113781 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1314,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":10757,"thread_start_us":262514,"threads_started":1,"wal-append.queue_time_us":400129}
W20260709 13:02:47.075536  5411 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.677s	user 0.013s	sys 0.003s
I20260709 13:02:47.059674  5106 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 5 FOLLOWER]: Advancing to term 6
I20260709 13:02:47.229146  5411 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11.
W20260709 13:02:47.085951  5618 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.972s	user 0.027s	sys 0.047s
W20260709 13:02:47.227543  4965 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.730s	user 0.026s	sys 0.017s
W20260709 13:02:47.127874  5611 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.948s	user 0.009s	sys 0.005s
I20260709 13:02:47.059144  5599 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 13:02:47.230443  5611 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.234391  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 585) took 1259 ms. Trace:
I20260709 13:02:47.235200  5611 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.236322  5599 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:47.235849  5657 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.236791  5599 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:47.237582  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:02:47.139161  5597 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.955s	user 0.004s	sys 0.014s
W20260709 13:02:47.135710  5542 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.704s	user 0.004s	sys 0.012s
I20260709 13:02:47.246151  5542 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:47.245993  5597 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.248342  5597 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.149175  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.147017  5493 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.157366  4967 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:47.250685  5493 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.112498  5481 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
W20260709 13:02:47.165740  5584 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.216s	user 0.013s	sys 0.010s
I20260709 13:02:47.251533  5584 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.253767  5584 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.255611  1741 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:47.256915  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2488) took 1361 ms. Trace:
I20260709 13:02:47.170642  5581 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:47.258407  5581 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.258724  5637 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.259259  5637 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:47.176865  5661 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 13:02:47.237807  1661 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:47.261936  1661 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:47.262003  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2469) took 1499 ms. Trace:
I20260709 13:02:47.179960  5023 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.262914  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:47.262605  1735 rpcz_store.cc:276] 0709 13:02:45.762271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.763525 (+  1254us) service_pool.cc:224] Handling call
0709 13:02:45.987067 (+223542us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:45.987602 (+   535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.987643 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:46.021795 (+ 34152us) log.cc:844] Serialized 52 byte log entry
0709 13:02:46.452716 (+430921us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:46.452779 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.452948 (+   169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.147802 (+694854us) raft_consensus.cc:1685] finished
0709 13:02:47.147827 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.261523 (+113696us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54000e1d50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.261912 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":948,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":65841,"spinlock_wait_cycles":512832256,"thread_start_us":412439,"threads_started":1,"wal-append.queue_time_us":458023}
I20260709 13:02:47.274845  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 708) took 1498 ms. Trace:
I20260709 13:02:47.070188  5659 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20260709 13:02:47.275861  5659 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:02:47.276461  5659 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:47.188014  1586 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:47.087165  5662 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20260709 13:02:47.195639  5656 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.196868  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2471) took 1433 ms. Trace:
I20260709 13:02:47.278854  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.279753  1586 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:47.278630  1734 rpcz_store.cc:276] 0709 13:02:45.763040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.764241 (+  1201us) service_pool.cc:224] Handling call
0709 13:02:45.889729 (+125488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:45.890125 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:45.890154 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:45.890652 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 13:02:46.184714 (+294062us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:46.184758 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:46.184827 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:46.922743 (+737916us) raft_consensus.cc:1685] finished
0709 13:02:46.922769 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.196483 (+273714us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540014ebd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.196782 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4036,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":241777024,"thread_start_us":291128,"threads_started":1,"wal-append.queue_time_us":500331}
I20260709 13:02:47.115254  5600 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.280959  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.235868  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2511) took 2807 ms. Trace:
I20260709 13:02:47.281683  1590 rpcz_store.cc:276] 0709 13:02:44.428255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.278453 (+1850198us) service_pool.cc:224] Handling call
0709 13:02:46.278907 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.390075 (+111168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.235625 (+845550us) spinlock_profiling.cc:243] Waited 99 ms on lock 0x7b54003b10d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:47.235805 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227650816}
I20260709 13:02:47.282861  5600 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.229858  5618 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.234822  1742 rpcz_store.cc:276] 0709 13:02:45.974951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.336487 (+361536us) service_pool.cc:224] Handling call
0709 13:02:46.336810 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.404100 (+ 67290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.234101 (+830001us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b5400404a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:47.234321 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153045888}
I20260709 13:02:47.284268  1733 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:47.241894  5599 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.243889  1486 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:47.245131  1490 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 236) took 38859 ms (client timeout 15000 ms). Trace:
W20260709 13:02:47.140955  5539 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.653s	user 0.003s	sys 0.008s
I20260709 13:02:47.258073  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:47.175179  5326 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.633s	user 0.009s	sys 0.009s
I20260709 13:02:47.265348  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:47.257263  1741 rpcz_store.cc:276] 0709 13:02:45.895542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.314949 (+419407us) service_pool.cc:224] Handling call
0709 13:02:46.315261 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.256079 (+940818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.256678 (+   599us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:47.256851 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2162082048}
I20260709 13:02:47.268247  1661 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.323200  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:47.323915  1668 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:47.324291  1661 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:47.185540  5549 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.003s
I20260709 13:02:47.269236  1585 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:47.183477  5462 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:47.270351  5637 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.272806  5633 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 13:02:47.286578  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.031s	sys 0.061s
I20260709 13:02:47.275360  2318 rpcz_store.cc:276] 0709 13:02:45.776138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:45.776542 (+   404us) service_pool.cc:224] Handling call
0709 13:02:45.776885 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:45.777136 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:45.915253 (+138117us) log.cc:844] Serialized 14 byte log entry
0709 13:02:47.269422 (+1354169us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b1800392080. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:02:47.274364 (+  4942us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:47.274443 (+    79us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:47.274521 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.274618 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.274773 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21765,"prepare.queue_time_us":369802,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":206411,"raft.run_cpu_time_us":5491,"raft.run_wall_time_us":50275,"spinlock_wait_cycles":5656960,"thread_start_us":898465,"threads_started":3,"wal-append.queue_time_us":626355}
W20260709 13:02:47.342263  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.031s	sys 0.061s
I20260709 13:02:47.286439  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:47.342662  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 654465us
I20260709 13:02:47.347359  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:47.289831  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:47.245132  5477 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.709s	user 0.016s	sys 0.015s
I20260709 13:02:47.351649  1584 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:47.302215  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:47.369335  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 67244.2us
I20260709 13:02:47.303515  1586 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:47.373581  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2542) took 1049 ms. Trace:
I20260709 13:02:47.374101  1586 rpcz_store.cc:276] 0709 13:02:46.324449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.280662 (+956213us) service_pool.cc:224] Handling call
0709 13:02:47.373478 (+ 92816us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:47.375562  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.376397  1586 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.058627  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.314857  2313 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:47.380791  2313 raft_consensus.cc:606] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:47.381295  2313 raft_consensus.cc:614] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:47.315644  5507 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:47.316303  5599 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.317032  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1784) took 1411 ms. Trace:
I20260709 13:02:47.318303  1739 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:47.331436  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:47.339401  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.080610  5660 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:47.349910  5462 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:47.313642  5548 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.746s	user 0.016s	sys 0.005s
W20260709 13:02:47.355207  1659 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.018s	sys 0.029s
W20260709 13:02:47.407357  1659 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.018s	sys 0.029s
W20260709 13:02:47.358832  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69345.5us
I20260709 13:02:47.365635  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:47.365075  1585 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:47.071475  5604 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.230338  4965 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.275426  5583 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.304739  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:47.344198  5326 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.372750  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:47.387015  5507 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:47.300729  1592 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:47.396374  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:47.399192  1733 rpcz_store.cc:276] 0709 13:02:45.905683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.100513 (+194830us) service_pool.cc:224] Handling call
0709 13:02:47.316803 (+1216290us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.316962 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2536321792}
I20260709 13:02:47.350106  5633 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:47.277402  5659 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.344374  5539 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.478202  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.086639  5591 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.419391  4685 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.419469  1590 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:47.419770  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:47.421069  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:47.421555  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:47.481518  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:47.312345  1490 rpcz_store.cc:269] 0709 13:02:08.394154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:08.434127 (+ 39973us) service_pool.cc:224] Handling call
0709 13:02:47.244975 (+38810848us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:39.723814 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:39.750177 (+ 26363us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:39.750252 (+    75us) write_op.cc:435] Acquired schema lock
0709 13:02:39.750290 (+    38us) tablet.cc:662] Decoding operations
0709 13:02:39.751789 (+  1499us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:39.751900 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:39.751941 (+    41us) tablet.cc:685] Acquiring locks for 10 operations
0709 13:02:39.753210 (+  1269us) tablet.cc:701] Row locks acquired
0709 13:02:39.753270 (+    60us) write_op.cc:260] PREPARE: finished
0709 13:02:39.805518 (+ 52248us) write_op.cc:270] Start()
0709 13:02:40.171124 (+365606us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b54002d0e28. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf43999e1 00007fcdf439864c 00007fcdf1a28624 00007fcdf1a13286 00007fcdf1a12d58 00007fcdf1a179f1 00007fcdf1a179a9 00007fcdf1a17939 00007fcdf1a17901 00007fcdf1a16b2d 000000000038f354
0709 13:02:40.171431 (+   307us) write_op.cc:276] Timestamp: P: 1783602159753427 usec, L: 0
0709 13:02:40.171461 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 13:02:40.172412 (+   951us) log.cc:844] Serialized 4024 byte log entry
0709 13:02:40.648855 (+476443us) op_driver.cc:464] REPLICATION: finished
0709 13:02:40.819866 (+171011us) write_op.cc:301] APPLY: starting
0709 13:02:40.819971 (+   105us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:40.820946 (+   975us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:40.820988 (+    42us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:41.028490 (+207502us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:41.128095 (+ 99605us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0709 13:02:41.128154 (+    59us) write_op.cc:312] APPLY: finished
0709 13:02:41.128578 (+   424us) log.cc:844] Serialized 77 byte log entry
0709 13:02:41.128781 (+   203us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:41.128938 (+   157us) write_op.cc:454] Released schema lock
0709 13:02:41.129010 (+    72us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:41.129478 (+   468us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:41.129537 (+    59us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9006,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":156126,"apply.run_cpu_time_us":6403,"apply.run_wall_time_us":309579,"num_ops":10,"prepare.queue_time_us":415726,"prepare.run_cpu_time_us":11361,"prepare.run_wall_time_us":669573,"raft.queue_time_us":214446,"raft.run_cpu_time_us":14783,"raft.run_wall_time_us":163030,"replication_time_us":457365,"spinlock_wait_cycles":12967680,"thread_start_us":667035,"threads_started":4,"wal-append.queue_time_us":227399,"wal-append.run_cpu_time_us":36106,"wal-append.run_wall_time_us":1772719}]]}
W20260709 13:02:47.428522  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 123838us
I20260709 13:02:47.402667  5477 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.398979  5266 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.352888  5638 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 13:02:47.400020  5542 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.403784  5660 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.350142  5549 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.367955  5412 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:47.518791  1741 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:47.433977  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 386556us
I20260709 13:02:47.399824  5653 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:47.400605  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69306.5us
I20260709 13:02:47.399189  5373 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:47.405813  5493 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:47.402210  5021 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.000s
I20260709 13:02:47.533182  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:47.382331  1593 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:47.503584  1588 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:47.388353  5663 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:47.439211  5616 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:47.534112  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:47.511385  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:47.487164  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:47.549180  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:47.433239  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 385778us
I20260709 13:02:47.550712  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:47.515610  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:47.422952  5548 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:47.553004  4685 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:02:47.532881  5634 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.921s	user 0.023s	sys 0.000s
I20260709 13:02:47.479460  5411 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
W20260709 13:02:47.434044  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 61136.8us
I20260709 13:02:47.477857  5574 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:47.405647  5480 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:47.505761  5507 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.487687  5657 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.553666  4685 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:47.552141  5663 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:47.552565  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72292.8us
I20260709 13:02:47.520283  5659 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:47.517606  5660 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:47.564785  2312 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
I20260709 13:02:47.545701  5498 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 13:02:47.559998  5611 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:47.537889  5493 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.545442  5616 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.557492  5656 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:47.575350  5574 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:47.518273  5266 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.557304  4967 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:47.533221  5637 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:02:47.517549  5542 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.533313  5561 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:02:47.564985  5634 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:47.537568  5653 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.560597  5619 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:47.580613  5480 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.609120  5480 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.610119  4965 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.600224  5574 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.611155  5539 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.596597  5507 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:47.617638  5477 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.617980  5591 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.618466  4967 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:47.619827  4967 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.621727  5266 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.623193  5326 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.625885  5583 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.588908  4685 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.533216  5373 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.545032  5021 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.567081  5602 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:47.572383  5548 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:47.579962  5618 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.431205  5581 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.594635  5493 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:47.597343  5498 consensus_queue.cc:1243] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:02:47.596349  5616 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:47.646668  5671 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:47.620807  5653 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.647416  5671 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:47.648257  5677 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:02:47.644168  5663 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.650164  5655 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:47.650632  5655 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:47.651441  5655 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.653136  5655 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:47.639331  5619 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:47.572968  5633 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.604383  5656 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:47.663573  5604 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.665650  5542 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.733956  1486 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 7, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:47.820393  5604 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:47.821791  5583 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:47.822082  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:47.821449  1486 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 8, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:47.822700  5583 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:47.513767  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 455859us
I20260709 13:02:47.461753  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:47.512073  1585 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:47.751030  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2538) took 1428 ms. Trace:
I20260709 13:02:47.830447  1588 rpcz_store.cc:276] 0709 13:02:46.322186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.415369 (+ 93183us) service_pool.cc:224] Handling call
0709 13:02:47.750063 (+1334694us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.750568 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2499948928}
I20260709 13:02:47.831779  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.832604  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:47.832971  1585 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.824715  1486 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 5 to 9, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 9 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:47.833761  1588 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:47.751658  1587 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 6.
W20260709 13:02:47.751060  1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2550) took 1421 ms (client timeout 1838 ms). Trace:
I20260709 13:02:47.491752  1589 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:02:47.759289  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2481) took 2833 ms. Trace:
I20260709 13:02:47.528774  1592 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.548779  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:47.847232  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:47.848315  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 299535us
W20260709 13:02:47.532320  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.502152  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:47.554172  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:47.559157  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:47.875098  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 315921us
I20260709 13:02:47.771014  5267 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:47.877259  5602 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:47.771422  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2400) took 1161 ms. Trace:
I20260709 13:02:47.878047  5267 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.878264  1659 rpcz_store.cc:276] 0709 13:02:46.609957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.610404 (+   447us) service_pool.cc:224] Handling call
0709 13:02:46.610932 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:46.611304 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:47.407884 (+796580us) log.cc:844] Serialized 14 byte log entry
0709 13:02:47.770420 (+362536us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:47.770561 (+   141us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:47.770709 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.770883 (+   174us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.771164 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":820637,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":14061,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":206,"thread_start_us":1128639,"threads_started":2,"wal-append.queue_time_us":235104}
I20260709 13:02:47.771095  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2478) took 1932 ms. Trace:
I20260709 13:02:47.776880  2311 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:47.779233  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 714) took 1268 ms. Trace:
I20260709 13:02:47.882619  5659 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:47.438062  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:47.486601  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:47.685987  5619 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.782192  1734 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:47.598158  5412 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.666795  5549 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.659927  5581 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.636992  5373 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:47.681149  5656 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.981256  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 387) took 1381 ms. Trace:
W20260709 13:02:47.983379  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 90847.5us
I20260709 13:02:47.983510  2311 rpcz_store.cc:276] 0709 13:02:46.599687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.599837 (+   150us) service_pool.cc:224] Handling call
0709 13:02:46.632867 (+ 33030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.881932 (+1249065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.980499 (+ 98567us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400610154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:47.981172 (+   673us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2628379904}
W20260709 13:02:47.654698  5675 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:02:47.986150  5616 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.686985  5633 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:47.685212  5660 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:48.024741  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.025810  5617 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:48.026765  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:48.027596  5660 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:47.790169  2316 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:47.794325  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 716) took 1265 ms. Trace:
I20260709 13:02:48.032083  2317 rpcz_store.cc:276] 0709 13:02:46.529134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.529332 (+   198us) service_pool.cc:224] Handling call
0709 13:02:46.549394 (+ 20062us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.023969 (+474575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:47.024003 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:47.052523 (+ 28520us) log.cc:844] Serialized 26 byte log entry
0709 13:02:47.404847 (+352324us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:47.404920 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.405005 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:47.781413 (+376408us) raft_consensus.cc:1685] finished
0709 13:02:47.781462 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:47.794035 (+ 12573us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56597,"raft.queue_time_us":240486,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":25428,"thread_start_us":825979,"threads_started":2,"wal-append.queue_time_us":496627}
I20260709 13:02:48.037712  5267 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:47.837544  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 7, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:48.041836  5106 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.812s	user 0.007s	sys 0.009s
I20260709 13:02:48.042528  5106 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 13:02:47.794582  5671 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:47.800474  5021 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:47.802721  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 270302us
I20260709 13:02:47.803655  1663 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:47.763427  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.812463  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:47.818652  5599 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.051226  5599 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:47.837898  1584 rpcz_store.cc:269] 0709 13:02:46.329105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.408475 (+ 79370us) service_pool.cc:224] Handling call
0709 13:02:47.750530 (+1342055us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b540025c954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.750819 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1921030528}
I20260709 13:02:48.054522  1735 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:48.055409  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:48.056752  1584 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:47.542047  1661 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:02:48.058921  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1776) took 1887 ms. Trace:
I20260709 13:02:47.852660  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:47.843544  1590 rpcz_store.cc:276] 0709 13:02:44.925444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.282831 (+2357387us) service_pool.cc:224] Handling call
0709 13:02:47.419274 (+136443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.486581 (+ 67307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.758579 (+271998us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540031c9d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.758994 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313049984}
I20260709 13:02:48.059900  5663 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:48.060890  5021 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:47.854221  1586 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:48.062181  5651 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:48.062682  5651 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:47.863775  5662 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:02:47.865949  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 404115us
W20260709 13:02:48.063002  5431 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 13:02:48.063482  5662 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.064150  5651 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.865093  5670 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:48.065745  5670 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.886111  1666 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:48.066116  5651 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:48.068029  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:47.821444  5604 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:47.926483  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 49400.4us
I20260709 13:02:47.882548  2312 rpcz_store.cc:276] 0709 13:02:46.510498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.510692 (+   194us) service_pool.cc:224] Handling call
0709 13:02:47.778501 (+1267809us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.778836 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2395741056}
I20260709 13:02:47.942459  5602 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:47.883234  5659 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:47.955206  5583 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:47.889259  5548 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:47.972779  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.078557  5604 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:48.079133  5548 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:47.972523  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:47.978349  5549 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.082346  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:48.083278  2313 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:48.082902  5549 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:48.084859  1593 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.004s
I20260709 13:02:47.596915  5611 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.085402  1593 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:48.086203  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2497) took 1087 ms. Trace:
I20260709 13:02:48.086529  1593 rpcz_store.cc:276] 0709 13:02:46.998706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.058359 (+ 59653us) service_pool.cc:224] Handling call
0709 13:02:48.086117 (+1027758us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:48.087929  5611 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:48.088196  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:48.088647  5659 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:48.089039  2316 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:48.091089  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:48.095264  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 541091us
I20260709 13:02:48.096159  5645 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:48.096735  5645 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.097175  5645 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.099676  5671 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:47.989979  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:47.989423  5462 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.567s	user 0.017s	sys 0.005s
I20260709 13:02:48.046455  1591 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 6.
I20260709 13:02:48.058743  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:47.848332  1741 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:02:47.881670  1739 rpcz_store.cc:276] 0709 13:02:45.838276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.205303 (+367027us) service_pool.cc:224] Handling call
0709 13:02:47.318031 (+1112728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.399491 (+ 81460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.770439 (+370948us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400107cd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:47.770779 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2558091648}
W20260709 13:02:48.061398  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 209090us
I20260709 13:02:48.066231  5670 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.059463  1661 rpcz_store.cc:276] 0709 13:02:46.171251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.171371 (+   120us) service_pool.cc:224] Handling call
0709 13:02:46.180530 (+  9159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:47.237698 (+1057168us) spinlock_profiling.cc:243] Waited 201 ms on lock 0x7b54002e5020. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfdf1c0 00007fcdedfe0231 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c
0709 13:02:48.058083 (+820385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.058539 (+   456us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":462835456}
I20260709 13:02:48.066568  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:48.076118  1592 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:48.043179  1486 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:48.077237  5602 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:48.082086  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 109948us
I20260709 13:02:48.106911  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.107401  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.108569  5583 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.109156  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.109490  1667 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:47.977234  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2478) took 3566 ms. Trace:
I20260709 13:02:48.113034  5539 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:48.114125  1585 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:48.110513  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.114892  5616 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:48.115698  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.119730  5477 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:02:48.127342  5599 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.088253  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.096884  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:48.129141  5574 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.518s	user 0.013s	sys 0.007s
I20260709 13:02:48.130362  5548 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:48.130681  5549 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.191052  2316 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:48.195519  1588 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:48.189181  1589 rpcz_store.cc:276] 0709 13:02:44.411067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:46.175506 (+1764439us) service_pool.cc:224] Handling call
0709 13:02:46.175931 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:47.843154 (+1667223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:47.976246 (+133092us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540031e554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:47.976967 (+   721us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3025484544}
W20260709 13:02:48.201853  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 143556us
I20260709 13:02:48.204386  1664 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:48.209877  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.211700  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:48.209940  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.215037  1666 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:48.218190  1586 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:48.218430  1585 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:48.212412  1587 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:48.215822  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:48.224014  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:48.232626  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 166506us
I20260709 13:02:48.188211  5462 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.215688  5651 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:48.243419  1584 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:48.244240  5549 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:02:48.247537  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.228128  5599 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.208914  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
W20260709 13:02:48.249888  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.250849  1659 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:02:48.253610  2312 raft_consensus.cc:606] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:48.263434  2312 raft_consensus.cc:614] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:48.264775  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:48.228929  5583 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.271826  5412 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.272523  5412 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.275182  5412 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.275992  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.276799  1659 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:48.277386  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.278259  1667 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:48.282100  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:48.283316  5655 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:48.283629  5431 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:48.284031  5655 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:48.284822  5431 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:48.285333  5431 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:48.285252  5655 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.286738  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2543) took 1961 ms. Trace:
I20260709 13:02:48.286849  5431 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.287209  1591 rpcz_store.cc:276] 0709 13:02:46.325022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:47.062395 (+737373us) service_pool.cc:224] Handling call
0709 13:02:48.286477 (+1224082us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b540025ce54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:48.286650 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2254778112}
I20260709 13:02:48.288393  5431 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:48.292143  5373 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.289669  5542 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:48.291610  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2534) took 2917 ms. Trace:
I20260709 13:02:48.289680  5106 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.296243  5599 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.241791  5651 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.234239  5539 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.216167  5574 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.298651  4685 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:48.323498  4685 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.323900  4685 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:48.284763  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2484) took 3253 ms. Trace:
I20260709 13:02:48.266160  5662 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.308307  1584 rpcz_store.cc:276] 0709 13:02:45.374526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.083614 (+2709088us) service_pool.cc:224] Handling call
0709 13:02:48.084085 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.244025 (+159940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.291308 (+ 47283us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54004001d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:48.291536 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":365851520}
I20260709 13:02:48.301867  1588 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:48.309574  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2492) took 1959 ms. Trace:
I20260709 13:02:48.332864  1591 rpcz_store.cc:276] 0709 13:02:46.340878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.288281 (+1947403us) service_pool.cc:224] Handling call
0709 13:02:48.288868 (+   587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.289289 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:48.289320 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:48.289378 (+    58us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:48.289411 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.289478 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.300106 (+ 10628us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:48.251111  5611 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:48.318868  5651 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.319983  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.301990  5504 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.324412  1586 rpcz_store.cc:276] 0709 13:02:45.031350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.062144 (+3030794us) service_pool.cc:224] Handling call
0709 13:02:48.211293 (+149149us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.219401 (+  8108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.284487 (+ 65086us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540033aed0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:48.284665 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341910400}
W20260709 13:02:48.288691  1557 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.042s (SENT)
I20260709 13:02:48.331245  1740 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:48.329336  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82473.1us
W20260709 13:02:48.337450  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 240895us
I20260709 13:02:48.337725  1666 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:48.311100  5412 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.341423  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 554) took 2342 ms. Trace:
I20260709 13:02:48.349365  1588 rpcz_store.cc:276] 0709 13:02:45.990366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.223844 (+2233478us) service_pool.cc:224] Handling call
0709 13:02:48.249933 (+ 26089us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.332437 (+ 82504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.332697 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:48.342954  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.339425  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.352380  5477 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:48.353144  5477 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:48.353214  5454 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:48.354420  5477 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.356139  5267 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:48.357173  5267 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:48.307389  5493 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.357594  5267 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:48.358491  5583 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.358678  5267 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.359628  5493 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.208914  1486 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 8, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 8 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:48.360220  5267 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.362820  5373 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.362620  5504 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.365235  5599 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.365649  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:48.366919  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78315.6us
I20260709 13:02:48.368032  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:48.368994  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:48.371698  5574 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.313140  1587 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:48.381261  1589 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:48.372098  5462 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:48.410759  1665 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:48.380262  1664 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:48.380378  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.361430  1486 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 5 to 6, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 6 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:48.385438  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:48.386986  5021 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:48.387509  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:48.388043  5583 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:48.378221  5106 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.390347  5498 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:48.391121  5549 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:48.393096  5480 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:48.385470  5599 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 13:02:48.394550  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:48.374248  5542 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.398813  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.405652  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.406635  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2490) took 2411 ms. Trace:
W20260709 13:02:48.407382  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78970.9us
I20260709 13:02:48.417971  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:48.438295  1589 rpcz_store.cc:276] 0709 13:02:45.994854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.223072 (+2228218us) service_pool.cc:224] Handling call
0709 13:02:48.255508 (+ 32436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.405801 (+150293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.406376 (+   575us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400443c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:48.406567 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270590720}
I20260709 13:02:48.420645  1659 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:48.430147  1667 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:48.423506  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:48.441323  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.423959  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:48.418025  5504 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:48.447860  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62587.1us
I20260709 13:02:48.448854  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:48.457616  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47058.1us
W20260709 13:02:48.459223  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64858.5us
I20260709 13:02:48.463492  1588 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:48.471952  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84602.7us
I20260709 13:02:48.433864  5662 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:48.440590  5480 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.471276  1586 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:48.475399  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.480872  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.485258  1589 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:48.485934  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76810.1us
I20260709 13:02:48.476612  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:48.488576  5611 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:48.489333  5611 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:48.489709  5691 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.002s
I20260709 13:02:48.490628  5611 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.490193  1486 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 5, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195), now has a pending config: VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193), VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194), VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195), VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 5 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:48.493677  5667 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:48.494177  5667 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.494561  5667 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:48.480799  5548 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:02:48.499012  5698 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:02:48.499269  5542 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.500147  1584 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.451701  5412 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.502717  5412 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.481127  5662 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:48.518805  1587 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:48.472705  5604 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:48.503794  5662 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:48.527487  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46561.8us
I20260709 13:02:48.504643  5480 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:48.503739  5106 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.527354  5504 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:48.532533  5633 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:48.534729  1587 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:48.535322  5412 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.537043  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:48.511579  5498 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:48.573460  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2495) took 1559 ms. Trace:
I20260709 13:02:48.570055  4685 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.574070  1587 rpcz_store.cc:276] 0709 13:02:46.987437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.533964 (+1546527us) service_pool.cc:224] Handling call
0709 13:02:48.534348 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.546200 (+ 11852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.546475 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:48.609872  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72959.2us
I20260709 13:02:48.579530  5504 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:48.543725  5660 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:48.617069  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:48.617458  1591 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 13:02:48.626543  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2554) took 2160 ms. Trace:
I20260709 13:02:48.626909  1591 rpcz_store.cc:276] 0709 13:02:46.466239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.504097 (+2037858us) service_pool.cc:224] Handling call
0709 13:02:48.571874 (+ 67777us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:48.626209 (+ 54335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.626481 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:48.607138  5663 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:48.620049  5659 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.531s	user 0.015s	sys 0.000s
I20260709 13:02:48.631981  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2482) took 3632 ms. Trace:
I20260709 13:02:48.576807  5412 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:48.636620  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92456.6us
I20260709 13:02:48.621662  5106 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:48.642082  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112278us
I20260709 13:02:48.624054  5574 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.585994  5583 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
W20260709 13:02:48.646926  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 170461us
I20260709 13:02:48.632860  1590 rpcz_store.cc:276] 0709 13:02:44.994546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.061182 (+3066636us) service_pool.cc:224] Handling call
0709 13:02:48.061797 (+   615us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.062321 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:48.197192 (+134871us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.617878 (+420686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:48.617947 (+    69us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:48.626336 (+  8389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.626432 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.626579 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10422,"prepare.queue_time_us":311918,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":21012,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":1980,"thread_start_us":501063,"threads_started":2}
I20260709 13:02:48.637173  5659 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.674589  5611 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:48.670708  5106 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:48.713887  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 326384us
I20260709 13:02:48.712776  4685 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.711637  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:48.713472  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.716967  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:48.714500  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:48.710198  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:48.731868  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.645336  5660 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.737346  1593 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:48.736523  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:48.739733  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82061.7us
W20260709 13:02:48.743441  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79375.5us
I20260709 13:02:48.746460  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:48.712546  5633 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:48.711438  5498 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:48.752010  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 383122us
I20260709 13:02:48.757830  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.701140  5574 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.756778  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.756999  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2501) took 1552 ms. Trace:
I20260709 13:02:48.765234  1590 rpcz_store.cc:276] 0709 13:02:47.204280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.691591 (+1487311us) service_pool.cc:224] Handling call
0709 13:02:48.740098 (+ 48507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.746606 (+  6508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:48.746636 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:48.756639 (+ 10003us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:48.756714 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.756797 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.756929 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:48.733704  5655 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:48.767931  4685 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:48.768535  4685 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:48.738994  5611 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:48.684742  5663 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.768409  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.793385  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.796747  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:48.797432  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.798450  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:48.809481  1662 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:02:48.802685  5716 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 13:02:48.803606  5703 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 13:02:48.804344  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.799015  5710 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.797128  5702 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:48.805356  1742 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:48.814251  1666 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:48.823757  1665 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:48.827674  5493 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:48.832602  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 555) took 2755 ms. Trace:
I20260709 13:02:48.769740  5655 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:48.852752  5493 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.843030  1585 rpcz_store.cc:276] 0709 13:02:46.076964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.246968 (+2170004us) service_pool.cc:224] Handling call
0709 13:02:48.297733 (+ 50765us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.298171 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:48.570120 (+271949us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.832148 (+262028us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:48.832219 (+    71us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:48.832275 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.832361 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.832530 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7606,"prepare.queue_time_us":100256,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":35,"thread_start_us":463989,"threads_started":3,"wal-append.queue_time_us":181323}
W20260709 13:02:48.842916  1557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.679s (SENT) [suppressed 11 similar messages]
I20260709 13:02:48.673743  1486 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:48.870267  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:48.794270  5477 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:02:48.886943  2317 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:48.836094  5602 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
W20260709 13:02:48.856778  5670 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.001s
I20260709 13:02:48.898139  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:48.905560  4685 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:48.912853  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 193351us
I20260709 13:02:48.843039  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:48.912458  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:48.922219  2312 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:48.823031  5504 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.945704  1585 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:48.947849  1486 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 4, term changed from 4 to 7, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:48.920699  5702 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:48.914263  5670 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:48.939556  5671 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.752s	user 0.012s	sys 0.001s
I20260709 13:02:48.928344  5602 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:48.951107  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:48.952039  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2491) took 2611 ms. Trace:
I20260709 13:02:48.973927  1592 rpcz_store.cc:276] 0709 13:02:46.340549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.295898 (+1955349us) service_pool.cc:224] Handling call
0709 13:02:48.296485 (+   587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:48.577745 (+281260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:48.577795 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:48.621639 (+ 43844us) log.cc:844] Serialized 26 byte log entry
0709 13:02:48.770503 (+148864us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:48.770568 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.770649 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:48.951765 (+181116us) raft_consensus.cc:1685] finished
0709 13:02:48.951788 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.951952 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":43709,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":64826,"thread_start_us":363316,"threads_started":2,"wal-append.queue_time_us":271326}
I20260709 13:02:48.986083  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2498) took 1835 ms. Trace:
I20260709 13:02:48.899520  5710 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.990178  5728 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:48.990909  5728 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.992228  5710 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:48.986505  1591 rpcz_store.cc:276] 0709 13:02:47.150244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.633025 (+1482781us) service_pool.cc:224] Handling call
0709 13:02:48.676142 (+ 43117us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.704665 (+ 28523us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:02:48.747620 (+ 42955us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.958579 (+210959us) log.cc:844] Serialized 14 byte log entry
0709 13:02:48.985629 (+ 27050us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:48.985691 (+    62us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:48.985738 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:48.985892 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:48.986023 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26758,"prepare.queue_time_us":40417,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":105,"thread_start_us":148398,"threads_started":1}
I20260709 13:02:48.993786  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:48.853608  5498 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.995821  5670 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.954494  5655 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.998754  1584 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:48.997226  5655 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:48.984589  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.004621  5373 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:49.005090  5373 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.005453  5373 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.007017  5702 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:49.007315  5267 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.006s
I20260709 13:02:49.009646  1733 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:49.015605  5539 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.016244  5711 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.016920  5711 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.017063  5712 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:49.017738  5702 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.018497  5702 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.019541  5711 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:49.020711  5718 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 13:02:49.021035  5702 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.021433  5659 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:48.966064  5021 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:49.026818  5021 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.029088  5021 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:49.030378  1486 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:02:49.034885  5701 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 13:02:49.036398  5611 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.038218  5412 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:49.041018  1586 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.000s
I20260709 13:02:49.041484  1586 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:49.046728  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2499) took 1853 ms. Trace:
I20260709 13:02:48.971566  1662 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:02:49.032557  1486 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:48.909288  5106 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:48.993247  5728 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.058369  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
W20260709 13:02:49.041486  5651 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.613s	user 0.007s	sys 0.008s
W20260709 13:02:49.065800  1589 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.540s	user 0.008s	sys 0.003s
I20260709 13:02:48.993835  5720 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:49.069796  1589 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
I20260709 13:02:49.068748  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:49.071991  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:49.083117  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 115944us
I20260709 13:02:49.083519  2316 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:49.083915  5724 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:49.084589  5724 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.086391  5720 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.088867  5720 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.089707  5713 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:49.090400  5713 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.090263  1486 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:02:49.093648  5712 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.094254  5574 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:49.094174  5712 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.094926  5574 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:49.096148  5574 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:49.096547  5574 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:49.096680  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:49.097193  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.097947  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:49.098758  1665 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:49.099011  5549 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.091061  5670 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:49.085386  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:49.106395  1663 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:02:49.083423  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:49.101388  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.107069  2312 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:49.109692  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.110360  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:49.089650  4967 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:49.113901  2317 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
W20260709 13:02:49.119298  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 276434us
I20260709 13:02:49.120033  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.051396  5671 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:02:49.099502  5549 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.120056  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.114611  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.118600  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.124907  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.122294  1665 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:49.096750  5712 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.131031  2313 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:49.136417  1585 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:49.131256  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:49.140210  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73697.8us
I20260709 13:02:49.140834  2311 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:49.141820  5728 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:49.142256  5728 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.142616  5728 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.143529  4967 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:49.051087  5633 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.140264  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.146538  2316 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:49.109365  5574 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.149468  5663 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:49.150388  5574 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:49.136416  5660 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.151046  5549 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.152318  5656 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:49.137051  5670 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.154218  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 561) took 1070 ms. Trace:
I20260709 13:02:49.155333  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:49.157846  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72572.4us
I20260709 13:02:49.158425  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:49.159078  5480 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.003s
I20260709 13:02:49.159472  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:49.159627  5480 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.164748  5267 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:02:49.165088  5662 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.618s	user 0.008s	sys 0.001s
I20260709 13:02:49.165371  5267 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:49.166095  5493 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.166443  5267 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.154978  1593 rpcz_store.cc:276] 0709 13:02:48.083965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.087895 (+  3930us) service_pool.cc:224] Handling call
0709 13:02:49.154010 (+1066115us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54003b15d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:49.154153 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1236992768}
I20260709 13:02:49.173265  1586 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:49.178041  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:49.161187  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:49.111680  5504 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.188115  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:49.080365  5651 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.180126  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:49.224661  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51860.5us
I20260709 13:02:49.185698  1667 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:49.181939  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.191650  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.125080  5713 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:49.230489  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 57402.8us
I20260709 13:02:49.268518  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.283156  1589 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:49.284149  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2503) took 2016 ms. Trace:
I20260709 13:02:49.284266  1662 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:02:49.284559  1589 rpcz_store.cc:276] 0709 13:02:47.267444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.131035 (+1863591us) service_pool.cc:224] Handling call
0709 13:02:49.282911 (+151876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.283803 (+   892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.284070 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:49.152747  5656 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:49.178187  5712 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.296818  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.300091  1667 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:49.298542  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.301144  1586 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:49.190758  4967 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:49.273016  5659 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:49.308825  2311 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:49.316876  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.184609  5662 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.314291  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:49.319608  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:49.320304  1589 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:49.317965  2310 raft_consensus.cc:606] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:49.323814  2310 raft_consensus.cc:614] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:49.324913  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:49.325713  5719 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.326325  5719 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:49.324954  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 163880us
I20260709 13:02:49.331488  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:49.332209  1662 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 6.
I20260709 13:02:49.325016  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:49.285573  5702 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:02:49.335441  2315 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:49.170532  5602 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.315276  5659 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.358465  5709 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.368247  1593 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:02:49.323468  5667 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.828s	user 0.013s	sys 0.003s
I20260709 13:02:49.388742  1590 raft_consensus.cc:2368] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:49.303069  5504 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.397053  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.399864  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:49.337579  5583 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.339145  4967 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:49.404026  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.312182  5712 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:49.406678  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 643541us
I20260709 13:02:49.391037  5659 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:49.430187  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.429112  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.393261  5670 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:49.453636  1593 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:49.467988  1733 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:02:49.468379  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2502) took 2239 ms. Trace:
W20260709 13:02:49.474653  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 142795us
I20260709 13:02:49.478536  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.485873  5431 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:49.486440  5431 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:49.487468  5431 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.489035  5431 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:49.495846  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:49.499529  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:49.468909  1592 rpcz_store.cc:276] 0709 13:02:47.228533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.111575 (+1883042us) service_pool.cc:224] Handling call
0709 13:02:49.177043 (+ 65468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.308019 (+130976us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:02:49.324180 (+ 16161us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.467161 (+142981us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:49.467945 (+   784us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:49.468027 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.468137 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.468295 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":14476,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"thread_start_us":76703,"threads_started":1}
I20260709 13:02:49.503521  1742 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:49.503960  1742 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:49.506618  1741 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:49.507788  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:49.463575  5691 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:02:49.504774  1742 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.516392  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:49.535177  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.535599  1742 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:49.468258  5710 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:49.518256  5701 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:02:49.395291  5667 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.540645  2314 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:49.468130  5611 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:49.546924  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:49.532429  5645 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.329s	user 0.011s	sys 0.004s
I20260709 13:02:49.401240  5709 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.507313  5480 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:49.542865  1593 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:49.555202  5731 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:49.540403  5655 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:02:49.583773  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 184006us
W20260709 13:02:49.561260  5733 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260709 13:02:49.562224  5732 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 13:02:49.540488  5548 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:49.558634  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 168370us
I20260709 13:02:49.547600  5710 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.557184  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.461364  5656 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.502090  5583 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.595455  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:49.558651  5725 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.605033  5656 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:49.605388  5737 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 13:02:49.605647  5721 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 13:02:49.606160  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:49.607424  5706 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 13:02:49.591779  5548 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:49.608543  1632 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:49.609115  5667 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:49.580279  5651 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.609962  1735 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:49.610720  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:49.611294  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:49.645574  2316 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 13:02:49.612607  5504 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.613268  5662 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:49.588903  5655 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:49.650772  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44705.3us
I20260709 13:02:49.597853  5731 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:49.591503  5717 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.658808  2312 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.544s	user 0.011s	sys 0.000s
I20260709 13:02:49.659403  2312 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:49.635751  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:49.619325  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:49.640902  1592 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:02:49.643977  1632 leader_election.cc:403] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:02:49.646312  1590 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [6.4-10.6]
I20260709 13:02:49.646916  5548 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:49.661389  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:49.665582  1739 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:49.642359  1589 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:49.557576  5645 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:49.651769  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:49.675694  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2504) took 2395 ms. Trace:
W20260709 13:02:49.675982  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:49.654359  5412 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:49.798209  4685 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:49.798471  2313 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.517s	user 0.013s	sys 0.007s
W20260709 13:02:49.798715  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147011us
I20260709 13:02:49.798919  2313 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
W20260709 13:02:49.797788  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 130495us
I20260709 13:02:49.804495  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:49.759305  1742 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:02:49.656751  5655 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:49.757473  1734 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 13:02:49.761914  1590 rpcz_store.cc:276] 0709 13:02:47.276114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.617069 (+2340955us) service_pool.cc:224] Handling call
0709 13:02:49.617652 (+   583us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:49.663138 (+ 45486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.663394 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:49.762634  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:49.677665  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69883.8us
I20260709 13:02:49.768519  1668 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:49.754371  5267 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:49.769348  5462 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:49.662344  5663 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:49.769910  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:49.771296  1589 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:49.771883  5373 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.766s	user 0.014s	sys 0.004s
I20260709 13:02:49.777666  5717 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.779033  5731 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.784122  5724 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:49.785852  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 392) took 1009 ms. Trace:
I20260709 13:02:49.640923  5725 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.788095  5710 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:49.791678  1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2565) took 1479 ms (client timeout 1628 ms). Trace:
I20260709 13:02:49.793974  1588 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:49.794267  1593 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:49.799379  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 323476us
W20260709 13:02:49.887905  1584 rpcz_store.cc:269] 0709 13:02:48.312096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.335804 (+ 23708us) service_pool.cc:224] Handling call
0709 13:02:49.791467 (+1455663us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:49.791612 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53234432}
I20260709 13:02:49.889781  1591 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:49.889974  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2505) took 2613 ms. Trace:
I20260709 13:02:49.891623  1593 rpcz_store.cc:276] 0709 13:02:47.276448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.793217 (+2516769us) service_pool.cc:224] Handling call
0709 13:02:49.794085 (+   868us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.889661 (+ 95576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.889860 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:49.867117  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:49.894047  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2557) took 2531 ms. Trace:
I20260709 13:02:49.894531  1589 rpcz_store.cc:276] 0709 13:02:47.362831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.754936 (+2392105us) service_pool.cc:224] Handling call
0709 13:02:49.771039 (+ 16103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.893669 (+122630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.893974 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:49.896130  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91748.2us
I20260709 13:02:49.888931  4685 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:49.890460  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:49.974462  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2559) took 2611 ms. Trace:
I20260709 13:02:49.047176  1587 rpcz_store.cc:276] 0709 13:02:47.203347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.643583 (+1440236us) service_pool.cc:224] Handling call
0709 13:02:48.757533 (+113950us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:48.842598 (+ 85065us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:48.853247 (+ 10649us) log.cc:844] Serialized 14 byte log entry
0709 13:02:49.046345 (+193098us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:49.046405 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:49.046450 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.046521 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.046665 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1015,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":6797,"raft.queue_time_us":115,"thread_start_us":192621,"threads_started":1}
I20260709 13:02:49.762568  5651 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:49.797390  5633 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 13:02:49.794215  5645 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:49.983409  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 116404us
I20260709 13:02:49.979521  1591 rpcz_store.cc:276] 0709 13:02:47.363313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.799031 (+2435718us) service_pool.cc:224] Handling call
0709 13:02:49.889597 (+ 90566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.899034 (+  9437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.974373 (+ 75339us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:49.995954  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:49.810184  5660 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:49.999470  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 111280us
I20260709 13:02:50.001756  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 560) took 2172 ms. Trace:
I20260709 13:02:50.003515  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2507) took 2253 ms. Trace:
I20260709 13:02:50.007551  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:50.007755  1593 rpcz_store.cc:276] 0709 13:02:47.750730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.892599 (+2141869us) service_pool.cc:224] Handling call
0709 13:02:49.988694 (+ 96095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.989022 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:49.989065 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.003133 (+ 14068us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:50.003201 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.003297 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.003449 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.007100  1589 rpcz_store.cc:276] 0709 13:02:47.828953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.895378 (+2066425us) service_pool.cc:224] Handling call
0709 13:02:49.987975 (+ 92597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.001255 (+ 13280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.001305 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.001376 (+    71us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:50.001420 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.001521 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.001675 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.012035  1742 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:49.810153  5480 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:49.999940  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2561) took 2636 ms. Trace:
I20260709 13:02:50.031483  1590 rpcz_store.cc:276] 0709 13:02:47.363753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.890943 (+2527190us) service_pool.cc:224] Handling call
0709 13:02:49.894936 (+  3993us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.990411 (+ 95475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:49.990441 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:49.999603 (+  9162us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:49.999661 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.999728 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.999856 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:49.986941  5462 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:50.020852  1591 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:50.040934  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2564) took 1741 ms. Trace:
I20260709 13:02:50.041448  1591 rpcz_store.cc:276] 0709 13:02:48.300007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.992818 (+1692811us) service_pool.cc:224] Handling call
0709 13:02:50.020555 (+ 27737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.040613 (+ 20058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.040866 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.019251  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2560) took 2655 ms. Trace:
I20260709 13:02:50.042341  1584 rpcz_store.cc:276] 0709 13:02:47.363531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.890726 (+2527195us) service_pool.cc:224] Handling call
0709 13:02:49.986297 (+ 95571us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.998599 (+ 12302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:49.998650 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:49.998733 (+    83us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:50.018862 (+ 20129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.018991 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.019167 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:49.991127  5651 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:49.993813  5633 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:50.045866  1739 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:02:49.894837  5412 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.023448  5723 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:49.892496  5373 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:50.094762  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64242.3us
I20260709 13:02:50.024684  5724 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:50.000548  5454 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.091218  4685 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.166308  1589 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:02:50.219403  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 564) took 1591 ms. Trace:
I20260709 13:02:50.220834  1589 rpcz_store.cc:276] 0709 13:02:48.627771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.083891 (+1456120us) service_pool.cc:224] Handling call
0709 13:02:50.157844 (+ 73953us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.218983 (+ 61139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.219316 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.157150  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2570) took 1778 ms. Trace:
I20260709 13:02:50.037297  5498 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:50.223508  1590 rpcz_store.cc:276] 0709 13:02:48.378578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.032732 (+1654154us) service_pool.cc:224] Handling call
0709 13:02:50.086151 (+ 53419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.156704 (+ 70553us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.156737 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.156803 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:50.156841 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.156913 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.157075 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.097455  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:50.033152  5480 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.089082  5633 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:50.230587  5574 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.080s	user 0.010s	sys 0.005s
I20260709 13:02:50.231604  5412 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:49.967651  2309 rpcz_store.cc:276] 0709 13:02:48.775826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.814912 (+ 39086us) service_pool.cc:224] Handling call
0709 13:02:48.921703 (+106791us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:49.001099 (+ 79396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:49.040330 (+ 39231us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:49.084999 (+ 44669us) log.cc:844] Serialized 26 byte log entry
0709 13:02:49.232600 (+147601us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:49.252923 (+ 20323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:49.293605 (+ 40682us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:49.785240 (+491635us) raft_consensus.cc:1685] finished
0709 13:02:49.785273 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:49.785451 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":49224,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":92105,"threads_started":1,"wal-append.queue_time_us":698156}
I20260709 13:02:50.230947  5462 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.232020  1584 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 9 index: 10. (index mismatch)
I20260709 13:02:50.243139  1590 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:50.280503  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2517) took 1117 ms. Trace:
I20260709 13:02:50.280928  1590 rpcz_store.cc:276] 0709 13:02:49.163011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.232508 (+1069497us) service_pool.cc:224] Handling call
0709 13:02:50.233171 (+   663us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.280179 (+ 47008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.280430 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:50.085824  5651 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:50.047223  5709 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.167459  5583 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.283334  1739 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.242308  5480 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:50.025933  5660 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:50.244928  5728 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.102s	user 0.011s	sys 0.004s
I20260709 13:02:50.094040  5723 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:50.273157  5599 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:50.290808  5756 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:50.319761  5754 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:50.320448  5754 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:50.320977  5754 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:50.238829  5454 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:50.323593  5759 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.433s	user 0.000s	sys 0.001s
I20260709 13:02:50.309188  5739 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:50.324352  5454 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.325064  5739 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:50.326853  5660 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:50.326516  5739 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.316682  5728 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.330502  5728 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:50.330855  5748 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.341s	user 0.000s	sys 0.000s
W20260709 13:02:50.332149  5751 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.640s	user 0.002s	sys 0.000s
I20260709 13:02:50.334478  1741 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:50.317256  5723 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:50.335014  5599 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.337160  5599 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.338910  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:50.341392  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 566) took 1193 ms. Trace:
I20260709 13:02:50.341849  1592 rpcz_store.cc:276] 0709 13:02:49.148282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.532251 (+383969us) service_pool.cc:224] Handling call
0709 13:02:50.341201 (+808950us) spinlock_profiling.cc:243] Waited 93.2 ms on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:50.341326 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214362368}
I20260709 13:02:50.343616  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:50.344085  5723 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.346410  5723 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:50.349160  5431 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.860s	user 0.017s	sys 0.002s
W20260709 13:02:50.358348  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.372021  5725 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.373514  5717 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.373828  1734 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:50.358781  5659 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.898s	user 0.010s	sys 0.010s
I20260709 13:02:50.375793  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.376614  2313 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 6 FOLLOWER]: Advancing to term 7
W20260709 13:02:50.359503  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.236999  5724 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:50.370937  5548 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.708s	user 0.010s	sys 0.000s
I20260709 13:02:50.409739  1734 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:50.411216  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.414491  1592 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:02:50.400485  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.415583  1586 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:50.401094  5373 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.406107  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:50.393819  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.416500  1735 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:02:50.416813  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:50.401751  5604 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.806s	user 0.013s	sys 0.004s
I20260709 13:02:50.425395  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:50.430696  1741 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:50.402173  5431 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.439424  1740 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.442682  2316 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:50.444280  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:50.436198  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 428968us
I20260709 13:02:50.448408  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:50.446374  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 394452us
W20260709 13:02:50.452152  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.459434  2313 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:50.402017  5659 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.434860  5604 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:50.461561  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:50.461710  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.489614  1586 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:50.490147  5549 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.490749  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.491426  1586 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
W20260709 13:02:50.491678  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.492475  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.492547  5723 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.492794  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:50.493211  1586 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:50.486717  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.411542  5548 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.490869  5549 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.528287  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.504132  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.503878  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.517859  1585 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:50.512742  2313 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:50.528200  2318 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:50.568341  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2521) took 1298 ms. Trace:
I20260709 13:02:50.515075  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:02:50.569383  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54441.5us
I20260709 13:02:50.568799  1590 rpcz_store.cc:276] 0709 13:02:49.269934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.291312 (+1021378us) service_pool.cc:224] Handling call
0709 13:02:50.310148 (+ 18836us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.310790 (+   642us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:50.406930 (+ 96140us) log.cc:844] Serialized 14 byte log entry
0709 13:02:50.567972 (+161042us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.568035 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:50.568085 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.568162 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.568282 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":98658,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"thread_start_us":159159,"threads_started":1,"wal-append.queue_time_us":119956}
I20260709 13:02:50.518620  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.572023  5659 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.574576  5480 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.575403  5549 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.576192  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 567) took 1427 ms. Trace:
I20260709 13:02:50.576578  1584 rpcz_store.cc:276] 0709 13:02:49.148609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.080730 (+932121us) service_pool.cc:224] Handling call
0709 13:02:50.130356 (+ 49626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.279830 (+149474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.575938 (+296108us) spinlock_profiling.cc:243] Waited 92.6 ms on lock 0x7b54001c4154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:50.576129 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213030144}
I20260709 13:02:50.577243  5548 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.577592  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.484603  5663 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260709 13:02:50.578961  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:50.525997  2309 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:02:50.503643  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.580209  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.580611  1738 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:50.581720  1738 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:50.582067  1738 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:50.570740  2316 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:50.588239  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.589524  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:50.590520  1660 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:50.525929  5373 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.588523   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:02:50.592082  5373 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.594924   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:02:50.514290  5599 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:50.595487  5373 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.514111  5728 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.596536  5724 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.596791  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2500) took 1706 ms. Trace:
I20260709 13:02:50.596921  5728 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.597172  1740 rpcz_store.cc:276] 0709 13:02:48.890762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.891027 (+   265us) service_pool.cc:224] Handling call
0709 13:02:48.929876 (+ 38849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.446006 (+1516130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.596524 (+150518us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b5400435b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:50.596715 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3471389440}
W20260709 13:02:50.599259  5655 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.668s	user 0.012s	sys 0.000s
I20260709 13:02:50.599644  5728 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.532277  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.601043  5604 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:50.591229  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:50.601758  5656 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.964s	user 0.008s	sys 0.004s
I20260709 13:02:50.601645  5604 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:50.602013  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:50.602703  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.603569  1592 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:50.603902  5604 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:50.604337  1584 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:50.607596  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:50.591777  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:50.551643  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2493) took 3741 ms. Trace:
I20260709 13:02:50.609038  1588 rpcz_store.cc:276] 0709 13:02:46.810292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.481678 (+1671386us) service_pool.cc:224] Handling call
0709 13:02:49.793786 (+1312108us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:49.891053 (+ 97267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.551327 (+660274us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54002d0ad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:50.551546 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2947607296}
I20260709 13:02:50.589322  2310 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:50.621060  1664 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:50.622355  1737 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:50.579324  5774 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:02:50.633517  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2508) took 2757 ms. Trace:
I20260709 13:02:50.639415  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:50.640698  1587 rpcz_store.cc:276] 0709 13:02:47.876479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.985128 (+2108649us) service_pool.cc:224] Handling call
0709 13:02:50.035813 (+ 50685us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.612798 (+576985us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.612847 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.633091 (+ 20244us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:50.633142 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.633209 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.633435 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120487,"raft.queue_time_us":210569,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":3704,"thread_start_us":417105,"threads_started":1}
I20260709 13:02:50.488459  5602 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:50.441959  5431 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:50.651670  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50209.4us
I20260709 13:02:50.652853  2316 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:50.504962  5725 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.577263  5480 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:50.610597  1738 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.654910  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:50.651837  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:50.659588  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.661424  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:50.484306  5717 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.156s
I20260709 13:02:50.668900  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:50.670935  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 178258us
I20260709 13:02:50.631335  5454 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:50.639662  5574 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:50.493307  5723 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.639178  5714 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:50.677563  1738 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:50.678743  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.661777  1735 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:50.541229  5651 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.651402  5493 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.591289  5754 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.694118  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:50.599674  5655 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.636930  5548 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 13:02:50.689589  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.703300  2316 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:50.682245  5730 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:50.657012  5713 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.690284  1590 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 13 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:02:50.690379  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.684903  5604 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.687592  5373 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:50.684726  5762 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:50.722154  2310 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:50.662537  5602 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.674857  5725 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.681972  5660 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.027s
W20260709 13:02:50.733582  5707 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.999s	user 0.012s	sys 0.008s
I20260709 13:02:50.726615  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.734086  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.734256  5431 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.734525  2316 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:50.736102  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:50.687289  5498 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 13:02:50.707273  5574 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 13:02:50.738377  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:50.740150  5373 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:50.740617  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.701267  5493 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.748905  5714 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:02:50.760655  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.727738  5713 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.770931  2309 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:02:50.770931  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2516) took 1608 ms. Trace:
I20260709 13:02:50.735255  5604 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:50.783787  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.775875  1589 rpcz_store.cc:276] 0709 13:02:49.162780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.237456 (+1074676us) service_pool.cc:224] Handling call
0709 13:02:50.281657 (+ 44201us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:50.296838 (+ 15181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.296896 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:50.376117 (+ 79221us) log.cc:844] Serialized 26 byte log entry
0709 13:02:50.605890 (+229773us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:50.605933 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.605999 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:50.770669 (+164670us) raft_consensus.cc:1685] finished
0709 13:02:50.770700 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.770854 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1004,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"thread_start_us":170126,"threads_started":1,"wal-append.queue_time_us":278957}
I20260709 13:02:50.735236  5762 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.755307  4967 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:02:50.790300  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.15475e+06us
I20260709 13:02:50.792023  5712 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:50.793587  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1797) took 2517 ms (client timeout 2091 ms). Trace:
W20260709 13:02:50.826323  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:50.830767  2309 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:02:50.746209  5633 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.840237  2314 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 7: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:50.847213  5651 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:50.866918  5745 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.939s	user 0.003s	sys 0.000s
I20260709 13:02:50.868202  5719 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.797374  4967 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:50.910602  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 222631us
W20260709 13:02:50.798617  1668 rpcz_store.cc:269] 0709 13:02:48.276280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:48.324780 (+ 48500us) service_pool.cc:224] Handling call
0709 13:02:50.793284 (+2468504us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b54001a9854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:50.793458 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3255554944}
W20260709 13:02:50.918213  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:50.918655  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 332806us
W20260709 13:02:50.920320  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0353443640000251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
I20260709 13:02:50.925486  5667 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:50.930974  5667 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.931900  5633 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.932161  5756 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:50.932643  5756 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:50.933635  5667 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.934259  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:50.935143  5667 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:50.935783  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:50.935729  5667 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:50.930850  4967 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:50.923062  5655 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.937049  5667 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:50.923303  5754 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.940980  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 563) took 2573 ms. Trace:
I20260709 13:02:50.941455  1593 rpcz_store.cc:276] 0709 13:02:48.368005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.009534 (+1641529us) service_pool.cc:224] Handling call
0709 13:02:50.042499 (+ 32965us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:50.419823 (+377324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.419873 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:50.435019 (+ 15146us) log.cc:844] Serialized 26 byte log entry
0709 13:02:50.731528 (+296509us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:50.731581 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:50.731690 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:50.940609 (+208919us) raft_consensus.cc:1685] finished
0709 13:02:50.940634 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:50.940884 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53390,"raft.queue_time_us":230541,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":11369,"thread_start_us":512960,"threads_started":2,"wal-append.queue_time_us":343573}
I20260709 13:02:50.949923  1632 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:50.951176  5373 consensus_queue.cc:1243] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:50.929414  5539 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:50.948714  5493 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.966097  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:50.968060  2315 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:50.970638  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:50.971098  5602 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.971691  2309 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:50.972318  5723 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.968470  5762 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:50.951742  5784 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:50.977768  1738 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:02:50.982138  1738 pending_rounds.cc:85] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:02:50.980100  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:50.964058  5713 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:50.989900  5602 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:50.994561  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:50.995030  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.995267  1664 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:50.995939  1588 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:50.996686  1587 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 13:02:50.997054  1587 pending_rounds.cc:85] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 4
I20260709 13:02:50.995959  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:50.995059  1589 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:02:50.997306  1587 pending_rounds.cc:107] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 13:02:50.998519  1590 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:50.998971  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:50.998980  1587 raft_consensus.cc:2889] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:50.982942  1738 pending_rounds.cc:107] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:02:51.000536  1738 raft_consensus.cc:2889] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:50.998364  1589 pending_rounds.cc:85] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:02:51.003326  1589 pending_rounds.cc:107] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:02:51.003549  1589 raft_consensus.cc:2889] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:51.004168  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:50.998588  5723 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 13:02:51.005901  1588 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:51.006644  5784 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:51.006697  5454 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.005985  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:51.007397  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.023028  1668 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:51.022238  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.000649  5660 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.006698  5723 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:51.026027  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:51.070191  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.070219  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:51.070976  1593 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:51.072189  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.033527  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.073709  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:51.070358  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 44404.1us
I20260709 13:02:51.075518  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:51.075598  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.076481  1665 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:51.078325  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:51.079289  1556 connection.cc:256] RPC call timeout handler was delayed by -1.0609488019999844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260709 13:02:51.079456  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.080435  1664 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:02:51.078711  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:51.008358  5719 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:51.079715  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:51.080113  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.022873  5784 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.024729  5602 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:51.090333  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.090334  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.022249  5412 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:51.025205  5454 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:51.029057  5714 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.069694  5670 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.077678  5754 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:51.033751  5660 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.112869  1667 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 13:02:51.030319  5713 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260709 13:02:51.109380  5764 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:51.115025  5412 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.109692  5784 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:51.027143  5655 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.089s
I20260709 13:02:51.028239  5723 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:51.144114  5604 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:51.144747  5604 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:51.144809  5723 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.145874  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:51.146317  5723 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:51.146464  1666 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:51.117158  1557 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.679s (SENT)
I20260709 13:02:51.147081  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:51.147676  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:51.069937  5493 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:02:51.118016  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:51.148945  1659 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:51.149215  5493 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:51.118757  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.093910  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.082643  5504 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:51.070319  5498 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 13:02:51.151979  5498 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:51.152326  5498 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:51.122619  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.153082  1662 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:02:51.113178  5659 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:51.153800  5659 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:51.154227  5659 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:51.156059  5574 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:51.156613  5574 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:51.157738  5574 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.166272  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.167201  1735 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 7: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:51.167574  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.168128  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.111354  5602 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:51.079712  5656 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 13:02:51.170323  5656 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:51.170696  5656 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:51.171769  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.193387  1741 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:51.143524  5787 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:02:51.149631  5493 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:51.153258  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2522) took 1862 ms (client timeout 1944 ms). Trace:
I20260709 13:02:51.168203  1736 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
W20260709 13:02:51.197736  1585 rpcz_store.cc:269] 0709 13:02:49.291062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:49.478212 (+187150us) service_pool.cc:224] Handling call
0709 13:02:51.153035 (+1674823us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400393fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:51.153191 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2387692288}
I20260709 13:02:51.118919  5549 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:51.172690  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.172420  5602 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.171553  5656 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.173285  5584 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:51.173467  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.174229  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.175114  5412 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.215819  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.180318  1668 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:51.182765  1740 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:51.183720  1666 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:51.093669  5783 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:51.185119  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.187489  1664 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 13:02:51.136531  5764 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:51.188395  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54330.8us
I20260709 13:02:51.109237  5768 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.100s
I20260709 13:02:51.121565  5754 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:51.123034  5670 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.132443  5714 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.191169  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.192404  5702 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:02:51.153110  5498 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.172863  5784 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.197765  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:51.283645  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.283814  1667 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 13 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 13:02:51.204716  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:51.285149  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.280543  1590 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:51.208959  1659 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:51.213150  1662 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:02:51.216717  1593 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:51.179741  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:51.268327  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.287923  1592 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:51.218197  5731 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.199950  5663 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.210739  5549 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.270856  1740 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:51.301450  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.300424  1661 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:51.304049  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.280076  5754 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:51.214012  5651 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.282377  5764 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:51.302853  5549 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:51.312605  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 176645us
W20260709 13:02:51.296502  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 217106us
W20260709 13:02:51.297168  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 298317us
I20260709 13:02:51.308848  1733 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:51.307426  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 204723us
I20260709 13:02:51.300124  4967 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:51.307969  5602 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:51.278132  5783 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.327822  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.311982  5412 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:51.292491  5713 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 13:02:51.321661  5762 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:51.323143  1740 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:51.340783  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 223656us
W20260709 13:02:51.341629  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.299907  5731 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.324901  5711 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:51.306957  5663 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.325150  5656 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:51.331529  5412 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:51.334283  5504 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:02:51.384578  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 290771us
I20260709 13:02:51.324152  5584 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
W20260709 13:02:51.392297  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63417.6us
I20260709 13:02:51.383653  1592 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
W20260709 13:02:51.385272  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 158504us
W20260709 13:02:51.389652  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 70345.8us
W20260709 13:02:51.393687  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67231.8us
W20260709 13:02:51.395781  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.396030  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:51.313844  5651 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.380163  5762 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:51.388437  5412 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:51.421536  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 101487us
W20260709 13:02:51.427572  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:51.428344  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.431659  5731 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:51.432870  5731 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:51.433961  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.434284  5739 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:51.434764  5739 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:51.435360  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.435770  5739 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.436502  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.437206  5739 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:51.438611  1737 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 13:02:51.444952  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43478.6us
I20260709 13:02:51.438778  5663 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.439270  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.392340  5431 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:51.443701  1734 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:02:51.444111  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:51.387509  5723 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 13:02:51.449105  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.452572  1738 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:51.454651  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.381527  5655 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 13:02:51.379485  5792 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.094s
W20260709 13:02:51.461437  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.444187  5702 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.464180  5549 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.466153  5723 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:51.466946  5802 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:02:51.470206  5789 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 13:02:51.470644  5784 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:51.471755  5793 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:02:51.472311  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:51.473063  5791 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:02:51.454576  5431 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:51.484056  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:51.497880  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 170140us
I20260709 13:02:51.502174  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:51.504598  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:51.519778  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:51.587780  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 103988us
I20260709 13:02:51.619580  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:51.621886  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 225932us
I20260709 13:02:51.628718  5663 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:51.630286  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:51.631544  5663 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:51.631600  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:51.632786  5663 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:51.633425  5663 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:51.634032  5663 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:51.508333  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
W20260709 13:02:51.657271  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 240469us
I20260709 13:02:51.496429  5802 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:51.664767  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.656431  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.666716  1738 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:51.503556  5811 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:51.584642  5431 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:51.584024  5784 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.681841  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.683024  1736 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:51.690316  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:51.690845  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.621958  5714 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.601715  5803 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:51.660081  5797 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:51.498905  5702 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.675391  5802 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:51.695516  5542 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:51.697166  5784 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:51.697674  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 109405us
I20260709 13:02:51.703759   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:02:51.703612  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 487937us
W20260709 13:02:51.702947  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 203353us
I20260709 13:02:51.746435  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:51.668989  5768 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 13:02:51.749577  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:51.504791  5504 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.654034  5798 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.091s
I20260709 13:02:51.777756  5549 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.699430  5713 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:51.764882  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.783981  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:51.793488  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.796932  1738 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:51.792836  1737 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:51.792750  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:51.804445  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.810405  1666 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:02:51.816299  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.817391  1738 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:51.769945  5797 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:51.675108  5762 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:51.832227  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 175869us
W20260709 13:02:51.833575  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49630.1us
I20260709 13:02:51.835855  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.839114  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.847877  1589 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:51.827557  5807 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.691955  5811 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.830713  5783 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:51.858167  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.860139  1589 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:02:51.858974  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94808.2us
I20260709 13:02:51.860925  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:51.861193  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80323.9us
I20260709 13:02:51.861598  1592 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:51.865872  5821 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.838552  5725 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.866557  5821 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.868110  5826 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:02:51.868968  5821 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:51.869766  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:51.870543  1666 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:51.872345  5655 raft_consensus.cc:2764] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 13:02:51.874473  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.843822  5815 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:51.837886  5818 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:51.861979  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:51.888273  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.884661  1592 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:51.892190  1589 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 13:02:51.874265  5659 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.720s	user 0.016s	sys 0.000s
I20260709 13:02:51.894362  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2513) took 2732 ms. Trace:
I20260709 13:02:51.897907  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:51.898084  5797 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:51.899580  5454 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.778s	user 0.010s	sys 0.001s
I20260709 13:02:51.900046  5454 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.813300  5810 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 13:02:51.856415  5651 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:51.901255  2314 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:51.897014  1591 rpcz_store.cc:276] 0709 13:02:49.161984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.080842 (+918858us) service_pool.cc:224] Handling call
0709 13:02:50.081275 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:50.433965 (+352690us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:50.434015 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:50.434075 (+    60us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:50.434122 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.894096 (+1459974us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540025e754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:51.894167 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:51.894310 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810541312}
I20260709 13:02:51.904219  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:51.912714  1738 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:51.896404  5542 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:51.875119  5821 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:51.936861  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:51.937309  5804 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.184s	user 0.000s	sys 0.001s
I20260709 13:02:51.937824  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:51.938047  5806 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 13:02:51.952687  2313 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:51.938167  5542 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.939985  5493 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.879791  5813 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:51.951258  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:51.896289  5659 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:51.860566  5807 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:51.953662  1589 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:51.996677  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:51.952732  2312 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:52.069451  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:52.062525  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:52.072643  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:52.073326  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.068949  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.074218  1738 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:51.962700  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.078770  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:51.895553  5815 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:52.079562  1588 raft_consensus.cc:2376] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7. Re-sending same reply.
I20260709 13:02:52.081036  1592 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:52.079171  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:52.081835  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.081691  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:52.080729  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118138us
I20260709 13:02:52.082541  1588 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:02:52.082998  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:52.030906  5816 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
W20260709 13:02:52.072178  5656 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.683s	user 0.007s	sys 0.007s
I20260709 13:02:52.097637  1591 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 13:02:51.904326  5714 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.183s
I20260709 13:02:52.102156  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:52.103547  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:52.088522  5651 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:52.111109  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.093677  5826 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:52.140187  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:52.077077  5813 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.107095  1666 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:02:52.106133  5754 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.000s
I20260709 13:02:52.103865  5784 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.132117  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:52.133029  5808 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.359s	user 0.000s	sys 0.002s
I20260709 13:02:51.980526  5821 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.096572  5702 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.095847  5493 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:52.031351  5802 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.131031  5651 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:52.158981  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.163201  1589 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 13:02:52.168838  5813 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.124893  5663 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.157732  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:52.174546  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 71111.9us
I20260709 13:02:52.196182  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:52.197027  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:52.197312  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.198580  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.198473  5656 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:02:52.199015  5802 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.199301  1736 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:52.193447  5651 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.176849  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.200273  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.135025  5826 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.175024  5719 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:52.184593  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.194259  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:52.208441  1736 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:52.142122  5454 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.199946  1632 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:02:52.222666  1632 leader_election.cc:403] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
W20260709 13:02:52.213541  5756 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.280s	user 0.015s	sys 0.003s
I20260709 13:02:52.215344  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 618) took 1049 ms. Trace:
W20260709 13:02:52.218392  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 356513us
I20260709 13:02:52.220849  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.229046  1738 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:52.230316  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.227658  1733 rpcz_store.cc:276] 0709 13:02:51.166070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.167875 (+  1805us) service_pool.cc:224] Handling call
0709 13:02:52.215032 (+1047157us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:52.215212 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321064192}
I20260709 13:02:52.203151  5809 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:52.242450  1586 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:52.200088  5656 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:52.165833  5754 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.163231  5431 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.232264  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.248503  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.242832  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:52.244494  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2533) took 1251 ms. Trace:
I20260709 13:02:52.244557  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.245434  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:52.246380  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 100559us
I20260709 13:02:52.218547  5826 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:52.248404  1589 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:52.251583  1586 rpcz_store.cc:276] 0709 13:02:50.992833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.238785 (+1245952us) service_pool.cc:224] Handling call
0709 13:02:52.239249 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.244125 (+  4876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.244415 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:52.254387  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.256634  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:52.259889  1708 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.538s (SENT)
I20260709 13:02:52.227139  5719 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.240984  5756 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:52.278712  5412 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.822s	user 0.013s	sys 0.001s
I20260709 13:02:52.262287  5431 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:52.279253  5412 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.200929  5817 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:02:52.283596  5821 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.239243  5815 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.292009  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.293263  1667 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:52.262492  5719 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:52.263692  5602 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.935s	user 0.011s	sys 0.003s
I20260709 13:02:52.263937  1585 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:52.298892  1735 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:52.265033  5754 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:52.258817  5809 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:52.268067  5713 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:52.268399  5498 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.263685  5663 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.296242  2315 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:52.281255  5756 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:52.300796  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.301833  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:52.304711  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 61988.5us
W20260709 13:02:52.306123  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73967.5us
I20260709 13:02:52.309788  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.263979  5711 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:52.311039  5549 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:52.311190  5711 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:52.311607  5549 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.312179  5549 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:52.313889  5805 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.314559  5805 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.315496  5574 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:52.316937  5805 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.319253  5667 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.319800  5667 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.320998  2311 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:52.321986  5549 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.322213  5667 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.323228  5714 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:52.324097  5714 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:52.325847  5656 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:52.325382  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.328101  5838 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:52.328529  5838 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.328861  5838 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:52.329766  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.299921  5837 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
W20260709 13:02:52.344257  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 751465us
I20260709 13:02:52.346912  1590 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:52.347383  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1826) took 1230 ms. Trace:
I20260709 13:02:52.360860  1661 rpcz_store.cc:276] 0709 13:02:51.117248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.122351 (+  5103us) service_pool.cc:224] Handling call
0709 13:02:52.347179 (+1224828us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540013ddd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:52.347293 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337789824}
I20260709 13:02:52.361415  1588 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:52.348213  1736 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:02:52.349037  5656 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:52.349970  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.350476  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 576) took 1150 ms. Trace:
I20260709 13:02:52.375111  1587 rpcz_store.cc:276] 0709 13:02:51.200086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.200245 (+   159us) service_pool.cc:224] Handling call
0709 13:02:51.264447 (+ 64202us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:51.276570 (+ 12123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:51.276600 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:51.293081 (+ 16481us) log.cc:844] Serialized 52 byte log entry
0709 13:02:52.349842 (+1056761us) spinlock_profiling.cc:243] Waited 9.26 ms on lock 0x7b18000cb860. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1
0709 13:02:52.350020 (+   178us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:52.350073 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.350141 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:52.350205 (+    64us) raft_consensus.cc:1685] finished
0709 13:02:52.350217 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.350419 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8679,"raft.queue_time_us":55070,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":7064,"spinlock_wait_cycles":21296512,"thread_start_us":51745,"threads_started":1,"wal-append.queue_time_us":189632}
I20260709 13:02:52.350616  5659 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.380447  1591 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:02:52.311506  5809 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.353464  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.387835  1586 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:52.307199  1733 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:02:52.355072  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45408.7us
W20260709 13:02:52.363837  1556 connection.cc:256] RPC call timeout handler was delayed by -1.3001714389999961s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:02:52.365080  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:52.396462  1588 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:52.380314  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:52.380661  1737 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:02:52.296823  5412 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.351008  5837 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:52.401943  1593 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:52.408987  5656 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:52.410112  5656 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.377677  5719 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.412075  5719 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.414278  5719 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.419322  1668 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:52.402063  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:52.422348  1662 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.402602  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.403733  1590 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:52.429543  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.422961  1589 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:52.430197  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:52.433063  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.433727  1589 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:52.439781  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.380425  5754 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.440830  1737 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:02:52.380152  5542 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:52.325858  5549 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.424911  5711 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:52.428020  5818 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.524s	user 0.013s	sys 0.000s
I20260709 13:02:52.487439  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:52.504981  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:52.441697  5764 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.116s	user 0.016s	sys 0.000s
I20260709 13:02:52.532215  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.534826  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:52.516799  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:52.465233  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.462029  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:52.538297  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.539997  1589 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:52.398777  5805 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260709 13:02:52.486088  5659 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.427842  5667 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260709 13:02:52.427968  5373 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.106s
I20260709 13:02:52.456920  5817 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:52.427659  5602 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.107s
I20260709 13:02:52.504473  5821 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election won for term 7
I20260709 13:02:52.551180  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:52.562711  1664 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:52.493901  5822 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:52.552132  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.533975  5719 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:52.537592  5714 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 13:02:52.551714  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:52.559979  1588 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:52.507594  5754 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.573195  5836 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:52.530465  5651 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.635244  5651 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:52.635489  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83888.6us
W20260709 13:02:52.635704  5844 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:02:52.568684  5659 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:02:52.570811  5817 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.637697  4967 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.638443  4967 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:02:52.639798  4967 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:52.571838  5719 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.546581  5542 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.641146  5719 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.602492  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.544930  5818 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:52.569550  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:02:52.665030  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95651.3us
I20260709 13:02:52.666642  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.626974  5711 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.563971  5826 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.632148  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.633006  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.537564  5764 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.641129  4967 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
W20260709 13:02:52.663414  5739 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.208s	user 0.010s	sys 0.010s
W20260709 13:02:52.672996  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:52.674693  1741 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:52.674700  1588 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:52.638459  5817 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.667732  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:52.670339  1591 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:52.675202  1662 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:52.696018  1589 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:52.699389  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.699704  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.699920  1589 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:52.700416  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.700464  1742 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:52.663566  5821 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:52.666237  5549 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:52.696455  1738 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:52.705889  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.685329  5493 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:52.674304  5826 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.718784  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:52.724871  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:52.731767  1557 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.631s (SENT)
I20260709 13:02:52.732633  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:52.733274  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:52.734079  1742 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:52.708941  5837 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:52.679024  5714 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 13:02:52.707772  4967 consensus_queue.cc:1243] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:02:52.719216  5817 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:52.720278  5493 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.743798  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.715168  5818 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:52.731034  5549 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 13:02:52.746519  1664 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:52.771083  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.779755  1742 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:52.780488  1739 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:52.721439  5711 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:52.747548  5830 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:52.764385  5802 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:52.813881  5818 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:52.815078  5807 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:52.809309  5431 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:52.778487  5723 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:52.736742  5837 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.753212  5659 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.746227  5817 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:52.779477  5493 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:52.826979  4967 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:52.835515  1742 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:52.762384  5838 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.851801  5837 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:02:52.860615  5574 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.545s	user 0.014s	sys 0.000s
I20260709 13:02:52.848707  5711 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:52.849279  5633 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:52.902922  5633 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.857956  5817 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:52.892735  5853 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:02:52.811239  5821 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.906684  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:52.907676  5711 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.891772  5818 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.886956  4967 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:52.924042  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 622) took 1177 ms. Trace:
I20260709 13:02:52.926366  5633 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.929023  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 856) took 1172 ms. Trace:
W20260709 13:02:52.882727  5498 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.527s	user 0.004s	sys 0.009s
I20260709 13:02:52.929608  2314 rpcz_store.cc:276] 0709 13:02:51.756239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.756804 (+   565us) service_pool.cc:224] Handling call
0709 13:02:52.928718 (+1171914us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54006e0fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:52.928914 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232950656}
I20260709 13:02:52.934964  4967 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:52.848840  5723 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:52.894398  5852 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:02:52.847492  5802 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:52.902221  5838 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:52.909113  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2546) took 1019 ms. Trace:
I20260709 13:02:52.956543  1741 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:53.167943  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:53.168738  1661 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:53.168740  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:53.171648  1591 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:02:53.134744  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.007s	sys 0.020s
W20260709 13:02:53.173846  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.018s	sys 0.020s
I20260709 13:02:53.038954  1739 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:53.174276  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:52.964751  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:53.175199  1587 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:52.924693  1667 rpcz_store.cc:276] 0709 13:02:51.747094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.747289 (+   195us) service_pool.cc:224] Handling call
0709 13:02:51.747675 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.293715 (+546040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.923622 (+629907us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b54001a81d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:52.923940 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1254397312}
W20260709 13:02:53.176357  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 211793us
I20260709 13:02:52.970191  1733 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:53.165581  1734 rpcz_store.cc:276] 0709 13:02:51.889254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.908112 (+1018858us) service_pool.cc:224] Handling call
0709 13:02:52.908472 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.908718 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.908756 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:52.908815 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:52.908861 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.908929 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:52.909046 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:53.183413  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.184456  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:53.184979  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.185156  1667 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:53.189038  1664 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:53.187000  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2545) took 1298 ms. Trace:
I20260709 13:02:53.175989  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.201560  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:53.186347  1734 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:52.967990  5796 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.538s	user 0.000s	sys 0.006s
I20260709 13:02:53.207058  1591 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:02:53.207127  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:53.195276  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2547) took 1303 ms. Trace:
I20260709 13:02:53.208050  1733 rpcz_store.cc:276] 0709 13:02:51.891912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.969173 (+1077261us) service_pool.cc:224] Handling call
0709 13:02:52.969718 (+   545us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.195012 (+225294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.195215 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:53.198557  1706 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:53.199819  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.191447  1662 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 13:02:53.210153  5826 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:53.211727  1662 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:53.203889  5792 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 13:02:53.213106  5431 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:53.214116  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:53.214600  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:53.210175  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.216013  1739 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:52.991222  5836 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:53.197221  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:53.204420  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:53.198366  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:53.200266  1742 rpcz_store.cc:276] 0709 13:02:51.888964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.862222 (+973258us) service_pool.cc:224] Handling call
0709 13:02:52.862601 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.862972 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.863004 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.876173 (+ 13169us) log.cc:844] Serialized 26 byte log entry
0709 13:02:53.179620 (+303447us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:53.179721 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.179915 (+   194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:53.185790 (+  5875us) raft_consensus.cc:1685] finished
0709 13:02:53.186707 (+   917us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.186932 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106,"raft.queue_time_us":99,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":17255,"thread_start_us":224030,"threads_started":1,"wal-append.queue_time_us":251867}
I20260709 13:02:53.219959  1740 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:53.190192  4967 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:53.220700  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.221486  1742 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:53.186151  5756 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:53.079668  5862 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.194677  1663 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:02:53.208779  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.214167  5477 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 13:02:53.228192  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.230628  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1816) took 2207 ms. Trace:
I20260709 13:02:53.231112  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.231072  1660 rpcz_store.cc:276] 0709 13:02:51.023654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.054692 (+ 31038us) service_pool.cc:224] Handling call
0709 13:02:51.055734 (+  1042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:51.072583 (+ 16849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:51.072619 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:51.080430 (+  7811us) log.cc:844] Serialized 26 byte log entry
0709 13:02:51.324614 (+244184us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:51.324673 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:51.328067 (+  3394us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:52.967064 (+1638997us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:53.230253 (+263189us) raft_consensus.cc:1685] finished
0709 13:02:53.230279 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.230536 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6742,"spinlock_wait_cycles":24896896,"thread_start_us":210299,"threads_started":1,"wal-append.queue_time_us":315425}
I20260709 13:02:53.232626  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.203519  5667 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.195894  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.242583  1741 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:53.207870  1664 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:52.989449  5670 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:53.218609  2283 connection.cc:256] RPC call timeout handler was delayed by -1.1400579109999853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
I20260709 13:02:53.221992  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:53.222088  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:52.948539  5828 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.221107  2310 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:52.940380  5754 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:53.213683  5431 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:53.225250  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:53.228188  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:53.246268  1742 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:53.219916  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:53.248291  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.248221  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:53.266973  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.267544  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:53.271157  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:53.223587  5836 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:53.223323  5792 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:02:53.221073  5602 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:53.277165  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:53.381024  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.438207  1741 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:53.254346  5412 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:02:53.443608  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 172603us
I20260709 13:02:53.277686  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:53.360136  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 142722us
I20260709 13:02:53.445945  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.405157  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.450763  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:53.352345  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 134448us
W20260709 13:02:53.278250  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56358.3us
I20260709 13:02:53.366496  1740 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 13:02:53.327724  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.454381  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:53.282419  1667 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:53.225579  5830 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:53.448374  2315 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:53.342453  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.465164  1739 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:02:53.448906  1586 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:53.467108  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.443900  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:53.255620  5815 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:53.459300  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 265061us
I20260709 13:02:53.474152  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 623) took 2286 ms. Trace:
W20260709 13:02:53.271144  5719 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.002s
I20260709 13:02:53.504992  1740 rpcz_store.cc:276] 0709 13:02:51.167450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.182210 (+ 14760us) service_pool.cc:224] Handling call
0709 13:02:51.182583 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:52.933718 (+1751135us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400137a20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52a1d 00007fcdedf391a4 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:02:53.454033 (+520315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.454311 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":814860032}
I20260709 13:02:53.505827  5719 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:53.506781  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.508796  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.277482  5702 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:53.491290  1741 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:53.514277  1740 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:53.515259  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.452070  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:53.516988  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.518682  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2550) took 1442 ms. Trace:
I20260709 13:02:53.260253  5811 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:53.510052  1666 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:02:53.519110  1742 rpcz_store.cc:276] 0709 13:02:52.076510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.517682 (+1441172us) service_pool.cc:224] Handling call
0709 13:02:53.518096 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.518318 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.518359 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:53.518406 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:53.518454 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.518511 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.518622 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:53.514281  5702 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:53.520586  5702 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:53.511433  5719 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:53.521288  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2552) took 1127 ms. Trace:
I20260709 13:02:53.530098  1742 rpcz_store.cc:276] 0709 13:02:52.394226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.520367 (+1126141us) service_pool.cc:224] Handling call
0709 13:02:53.520743 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:53.520965 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.520990 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:53.521034 (+    44us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:53.521076 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.521133 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:53.521235 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:53.265749  5864 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:53.434652  5670 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:53.592777  1740 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:53.537437  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.251574  5756 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:53.543313  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:53.535666  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 259802us
W20260709 13:02:53.617285  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:53.224628  5817 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:53.553684  5837 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.648s	user 0.013s	sys 0.005s
I20260709 13:02:53.277568  5754 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:53.490748  5659 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:53.667209  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 222532us
I20260709 13:02:53.750628  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:53.554153  5412 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:53.778209  5412 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:53.250593  5862 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:53.268282  5828 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:53.486829  5480 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:02:53.671641  1739 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:53.344017  5805 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:53.679306  5670 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.473937  5830 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:53.491186  5477 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:53.738550  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.741351  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:53.486562  5861 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:53.269819  5667 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:53.509819  5815 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:53.436478  5493 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:53.775379  5882 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:53.445878  5633 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.292s
I20260709 13:02:53.476912  5731 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:53.485860  5602 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:53.670030  5811 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.144s
I20260709 13:02:53.678148  5864 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:54.141234  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:53.993088  5866 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
W20260709 13:02:54.232625  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 778364us
I20260709 13:02:54.234225  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2544) took 2348 ms. Trace:
W20260709 13:02:54.230868  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 480369us
I20260709 13:02:54.234861  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:54.234724  1738 rpcz_store.cc:276] 0709 13:02:51.885944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.842745 (+956801us) service_pool.cc:224] Handling call
0709 13:02:52.843192 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.843651 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.843684 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.844274 (+   590us) log.cc:844] Serialized 26 byte log entry
0709 13:02:52.944639 (+100365us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:52.944687 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.944764 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.233938 (+1289174us) raft_consensus.cc:1685] finished
0709 13:02:54.233972 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.234142 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2138,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":99901,"threads_started":1,"wal-append.queue_time_us":601138}
I20260709 13:02:54.237195  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.238369  1738 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:54.238515  5817 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.229899  5548 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:54.242030  5670 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.239620  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.246821  1742 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:54.247100  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.248044  1738 raft_consensus.cc:2376] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11. Re-sending same reply.
I20260709 13:02:54.249207  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.041028  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1832) took 1695 ms. Trace:
W20260709 13:02:53.857561  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 206116us
I20260709 13:02:54.252507  1666 rpcz_store.cc:276] 0709 13:02:52.345884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.639025 (+293141us) service_pool.cc:224] Handling call
0709 13:02:52.639370 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:53.520054 (+880684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.040484 (+520430us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:54.040899 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2001689856}
I20260709 13:02:53.747851  5719 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:02:53.951686  5858 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
W20260709 13:02:54.054891  5663 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.005s	sys 0.023s
W20260709 13:02:53.991920  5863 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.527s	user 0.000s	sys 0.001s
I20260709 13:02:54.282553  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.283449  1733 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:54.043529  5854 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.208s	user 0.000s	sys 0.002s
W20260709 13:02:53.855960  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 402333us
W20260709 13:02:54.042939  5860 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.058s	sys 0.123s
I20260709 13:02:54.296682  5602 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:54.297436  5870 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.624s	user 0.000s	sys 0.002s
I20260709 13:02:54.232275  5805 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:53.825675  5825 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.474s
I20260709 13:02:54.300395  5811 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:54.301258  5667 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:54.301873  2310 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 13:02:54.439522  5860 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.976s	user 0.063s	sys 0.128s
I20260709 13:02:54.307833  5493 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.251729  5807 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.474092  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.312299  5670 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:54.316437  5857 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.470s	user 0.002s	sys 0.008s
I20260709 13:02:54.317700  5881 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:54.283365  5814 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:54.318509  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 628) took 2379 ms. Trace:
I20260709 13:02:54.306492  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2553) took 1911 ms. Trace:
I20260709 13:02:54.319948  5862 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.094957  5875 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:53.952201  5795 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:54.077487  5739 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:53.809180  5754 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:53.993180  5660 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:02:54.257722  5830 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:54.027400  5802 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:54.044217  5822 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:54.254748  5477 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:54.341219  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.012s	sys 0.021s
I20260709 13:02:54.285450  5549 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:54.264528  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:53.786784  5659 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.441335  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:54.445379  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 192253us
I20260709 13:02:53.753197  5837 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:54.453112  1661 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.267s	user 0.012s	sys 0.002s
I20260709 13:02:54.550575  1661 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:54.551250  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 632) took 1383 ms. Trace:
I20260709 13:02:54.455514  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.552986  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.500160  5882 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.553817  1739 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:54.553987  5882 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.472146  5811 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.554733  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.284312  5574 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
W20260709 13:02:54.306430  5702 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.785s	user 0.015s	sys 0.002s
I20260709 13:02:54.096856  5883 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.387s
W20260709 13:02:54.442160  5821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.043s	sys 0.082s
W20260709 13:02:54.470863  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.000s	sys 0.012s
W20260709 13:02:54.602633  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.001s	sys 0.012s
W20260709 13:02:54.603086  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 892680us
I20260709 13:02:54.273275  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2555) took 1878 ms. Trace:
W20260709 13:02:54.475384  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 238466us
I20260709 13:02:54.476678   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:02:54.250638  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.604075  1740 rpcz_store.cc:276] 0709 13:02:52.395138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.271488 (+1876350us) service_pool.cc:224] Handling call
0709 13:02:54.272216 (+   728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.272724 (+   508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.272764 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:54.272827 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:54.272873 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.272955 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.273188 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:54.639636  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.644s (SENT) [suppressed 34 similar messages]
I20260709 13:02:54.450783  5719 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:54.492228  1742 rpcz_store.cc:276] 0709 13:02:52.394562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.532047 (+1137485us) service_pool.cc:224] Handling call
0709 13:02:53.532708 (+   661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.247409 (+714701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.306147 (+ 58738us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b5400338954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:54.306400 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640749952}
I20260709 13:02:54.646363  5815 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.494225  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2540) took 2611 ms. Trace:
W20260709 13:02:54.494138  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 233703us
I20260709 13:02:54.491230  1665 rpcz_store.cc:276] 0709 13:02:51.938851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.942252 (+  3401us) service_pool.cc:224] Handling call
0709 13:02:51.942751 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.243525 (+300774us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.243590 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.253359 (+  9769us) log.cc:844] Serialized 26 byte log entry
0709 13:02:52.780393 (+527034us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:52.780454 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.317824 (+1537370us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:54.317907 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.317989 (+    82us) raft_consensus.cc:1685] finished
0709 13:02:54.318010 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.318412 (+   402us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21134,"raft.queue_time_us":81928,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":5805,"spinlock_wait_cycles":342309632,"thread_start_us":489015,"threads_started":2,"wal-append.queue_time_us":556232,"wal-append.run_cpu_time_us":1837,"wal-append.run_wall_time_us":1042890}
I20260709 13:02:54.647205  1736 rpcz_store.cc:276] 0709 13:02:51.882805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.432798 (+549993us) service_pool.cc:224] Handling call
0709 13:02:52.525673 (+ 92875us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.526108 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.526152 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.535464 (+  9312us) log.cc:844] Serialized 26 byte log entry
0709 13:02:52.874096 (+338632us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:52.874147 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.874226 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.493926 (+1619700us) raft_consensus.cc:1685] finished
0709 13:02:54.493957 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.494141 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":575,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7830,"thread_start_us":243538,"threads_started":1,"wal-append.queue_time_us":831697}
I20260709 13:02:54.270440  5861 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.651579  5861 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.651427  5802 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:02:54.477381  5667 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:54.504163  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.000s	sys 0.022s
W20260709 13:02:54.652535  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.000s	sys 0.023s
W20260709 13:02:54.638489  5431 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.274s	user 0.016s	sys 0.000s
I20260709 13:02:54.656396  5431 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.658938  5431 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.660382  1591 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:54.660671  5876 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.891s	user 0.006s	sys 0.005s
I20260709 13:02:54.661954  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.502748  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:54.490949  5814 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:54.539779  1584 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:54.493423  5872 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:54.498411  5795 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:54.479540  5807 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.682636  5872 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.683272  1662 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:54.683753  5764 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.684558  5807 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:54.562347  5886 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:02:54.497049  5723 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.538589  5373 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.686043  5373 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:54.686738  5877 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 13:02:54.688419  5373 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.570667  5754 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.508230  5670 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:54.574321  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.81906e+06us
I20260709 13:02:54.499924  5739 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:54.501761  5731 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.505781  5875 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:54.599850  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50681us
I20260709 13:02:54.599798  1667 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:02:54.635116  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.636845  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:54.637533  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.713994  1739 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:54.714386  1587 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:54.714964  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78544.8us
I20260709 13:02:54.715081  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.715648  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:54.716213  1739 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 13:02:54.716054  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:54.717202  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.718079  1587 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:54.536737  5822 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:54.719548  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.719858  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.638060  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2555) took 2090 ms. Trace:
I20260709 13:02:54.507607  5864 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.453545  5805 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:54.634321  5836 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.096s	user 0.014s	sys 0.002s
I20260709 13:02:54.641119  5477 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:54.729878  5892 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:02:54.730291  5477 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.639354  5549 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:54.652534  5667 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.735020  5549 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.630658  5702 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.735692  5837 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.646251  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.736833  1742 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:02:54.640259  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:54.515172  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.013s	sys 0.021s
I20260709 13:02:54.737484  5702 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:54.738014  1740 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:02:54.738169  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.29435e+06us
I20260709 13:02:54.738945  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.681944  5493 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.739780  1740 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:54.740257  5493 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.662936  5811 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.684471  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:54.684990  5872 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:54.689791  1557 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.659s (SENT)
I20260709 13:02:54.551564  1661 rpcz_store.cc:276] 0709 13:02:53.167451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.167597 (+   146us) service_pool.cc:224] Handling call
0709 13:02:54.551173 (+1383576us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:54.647246  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.681020  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 737) took 2247 ms. Trace:
I20260709 13:02:54.743583  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:54.692266  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52672.1us
I20260709 13:02:54.767619  1587 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:54.765048  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:54.671437  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.771376  1736 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:54.697974  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2565) took 1807 ms. Trace:
I20260709 13:02:54.773552  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:54.774426  1587 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:54.631237  5821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.043s	sys 0.082s
W20260709 13:02:54.702909  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 200452us
I20260709 13:02:54.704293  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.707696  1733 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:54.711055  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:54.710913  5731 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.711622  5548 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:54.637920  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:54.633558  5655 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.002s	sys 0.016s
I20260709 13:02:54.720657  1593 rpcz_store.cc:276] 0709 13:02:52.547327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.553280 (+  5953us) service_pool.cc:224] Handling call
0709 13:02:52.573963 (+ 20683us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.625865 (+ 51902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.625910 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:52.626559 (+   649us) log.cc:844] Serialized 26 byte log entry
0709 13:02:53.038347 (+411788us) spinlock_profiling.cc:243] Waited 7.03 ms on lock 0x7b180043f0c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:02:53.158812 (+120465us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:53.158878 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.158998 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.564931 (+1405933us) raft_consensus.cc:1685] finished
0709 13:02:54.564965 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.637987 (+ 73022us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8944,"raft.queue_time_us":48422,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":128,"spinlock_wait_cycles":16178048,"thread_start_us":120199,"threads_started":1,"wal-append.queue_time_us":638104}
I20260709 13:02:54.782632  1591 raft_consensus.cc:2376] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10. Re-sending same reply.
I20260709 13:02:54.724175  1739 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:54.725735  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:54.786477  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.728652  5875 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.700039  5739 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.681855  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.698557  5814 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.740794  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.845952  1740 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 9
I20260709 13:02:54.847420  1662 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:54.755087  5890 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 13:02:54.706761  5809 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.757591  5836 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.724260  5894 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:54.759029  5884 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:54.733297  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 408) took 1488 ms. Trace:
I20260709 13:02:54.698226  5498 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:54.772318  1590 rpcz_store.cc:276] 0709 13:02:52.890509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.917600 (+ 27091us) service_pool.cc:224] Handling call
0709 13:02:52.918134 (+   534us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:52.918710 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:02:52.918750 (+    40us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:53.195288 (+276538us) log.cc:844] Serialized 78 byte log entry
0709 13:02:53.758843 (+563555us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:53.758898 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:53.758988 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.685879 (+926891us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:54.689748 (+  3869us) raft_consensus.cc:1685] finished
0709 13:02:54.689772 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.697873 (+  8101us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":29853,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":155,"spinlock_wait_cycles":10219264,"thread_start_us":464149,"threads_started":1,"wal-append.queue_time_us":1206476}
I20260709 13:02:54.776762  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:54.768356  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50141.5us
I20260709 13:02:54.863973  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:54.775997  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86241.5us
I20260709 13:02:54.864636  1590 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:54.768718  2313 rpcz_store.cc:276] 0709 13:02:52.434042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.434395 (+   353us) service_pool.cc:224] Handling call
0709 13:02:52.498177 (+ 63782us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:52.530083 (+ 31906us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.530122 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:52.550054 (+ 19932us) log.cc:844] Serialized 52 byte log entry
0709 13:02:52.703988 (+153934us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:52.704058 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:52.704154 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.315350 (+1611196us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:54.662864 (+347514us) raft_consensus.cc:1685] finished
0709 13:02:54.662889 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.680858 (+ 17969us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6595,"raft.queue_time_us":33709,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":11226,"spinlock_wait_cycles":5818112,"thread_start_us":76782,"threads_started":1,"wal-append.queue_time_us":162843}
I20260709 13:02:54.782437  1592 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:54.783594  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:54.866621  1661 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:54.496876  5881 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:54.788709  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.866636  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:54.867667  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.868112  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2566) took 1420 ms. Trace:
I20260709 13:02:54.868654  1584 rpcz_store.cc:276] 0709 13:02:53.447592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.477799 (+ 30207us) service_pool.cc:224] Handling call
0709 13:02:54.868050 (+1390251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:54.768824  5764 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.842331  5883 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:02:54.794401  5542 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:54.795471  5548 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:54.833758  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:54.834813  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46561us
I20260709 13:02:54.735694  5667 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:54.786536  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60953.9us
I20260709 13:02:54.897063  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.897729  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:54.834837  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.898833  1742 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:54.899917  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.900556  1742 raft_consensus.cc:2376] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9. Re-sending same reply.
I20260709 13:02:54.767271  5815 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:54.796869  1739 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:54.863013  5814 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:54.842994  1738 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:02:54.772413  5807 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:02:54.796198  5655 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.003s	sys 0.016s
I20260709 13:02:54.781795  5663 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:54.761089  5802 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.093s
W20260709 13:02:54.869061  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:54.844560  5739 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.790220  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:54.954110  1589 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.874097  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:54.954950  1584 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:02:54.951131  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56151.9us
I20260709 13:02:54.883792  1587 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:54.886384  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:54.887522  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:54.957914  1661 raft_consensus.cc:2368] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
W20260709 13:02:54.958664  2312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 851) took 4337 ms (client timeout 3052 ms). Trace:
W20260709 13:02:54.959223  2312 rpcz_store.cc:269] 0709 13:02:50.607419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:50.733781 (+126362us) service_pool.cc:224] Handling call
0709 13:02:54.942951 (+4209170us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:54.944826 (+  1875us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2800479488}
I20260709 13:02:54.898854  1665 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:54.901444  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.897835  1591 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:02:54.904078  1557 connection.cc:256] RPC call timeout handler was delayed by -2.1083854349999456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1849s for a response
I20260709 13:02:54.961470  1742 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260709 13:02:54.962459  1557 connection.cc:256] RPC call timeout handler was delayed by -2.0390854349999472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1926s for a response
I20260709 13:02:54.962743  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:54.963316  1557 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.788s (SENT)
I20260709 13:02:54.963910  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:54.964576  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:02:54.842048  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54159.7us
I20260709 13:02:54.797087  5830 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.897107  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:54.844902  4967 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 13:02:54.834085  5719 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.860541  2314 rpcz_store.cc:276] 0709 13:02:53.245002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.256312 (+ 11310us) service_pool.cc:224] Handling call
0709 13:02:53.355430 (+ 99118us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:53.874060 (+518630us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:53.874107 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:53.890138 (+ 16031us) log.cc:844] Serialized 52 byte log entry
0709 13:02:54.340861 (+450723us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:54.340905 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.340997 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:54.732841 (+391844us) raft_consensus.cc:1685] finished
0709 13:02:54.732981 (+   140us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.733216 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88262,"raft.queue_time_us":312997,"spinlock_wait_cycles":75419904,"thread_start_us":830771,"threads_started":2,"wal-append.queue_time_us":604494}
I20260709 13:02:54.979107  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:54.979730  2309 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:54.966890  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:54.971918  1740 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:54.973654  2313 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:54.976526  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:54.981380  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:54.981976  1661 raft_consensus.cc:2368] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:54.948686  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2559) took 2506 ms. Trace:
I20260709 13:02:54.976295  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:54.982542  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.974213  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.862780  5809 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.978363  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:54.978698  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:54.965824  1557 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.536s (SENT)
I20260709 13:02:54.978404  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:54.983150  1740 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:54.982808  1739 rpcz_store.cc:276] 0709 13:02:52.442374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.947467 (+2505093us) service_pool.cc:224] Handling call
0709 13:02:54.947947 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.948209 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.948248 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:54.948309 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:54.948356 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.948437 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:54.948600 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:54.988948  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:54.989756  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.990249  1739 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:54.983994  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.985067  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:54.991775  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:54.991931  1742 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:54.987228  1667 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:54.947022  5884 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:54.780584  5795 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:54.895283  5764 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:54.994829  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.030042  1661 raft_consensus.cc:2368] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:55.030143  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:54.996843  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 99826.5us
I20260709 13:02:55.031030  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:54.976234  5702 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:54.999051  4967 consensus_queue.cc:1243] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:02:54.999469  5907 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:02:55.000415  5896 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.225s
I20260709 13:02:54.958696  5655 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.050566  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2556) took 2495 ms. Trace:
I20260709 13:02:55.053722  1588 rpcz_store.cc:276] 0709 13:02:52.551380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.687813 (+136433us) service_pool.cc:224] Handling call
0709 13:02:52.688167 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:52.762837 (+ 74670us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:52.762876 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:54.078635 (+1315759us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b2c00032118. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:54.079590 (+   955us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.457224 (+377634us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:54.458232 (+  1008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.458320 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.006258 (+547938us) raft_consensus.cc:1685] finished
0709 13:02:55.006291 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.046836 (+ 40545us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":709,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7801,"spinlock_wait_cycles":39934208,"thread_start_us":439981,"threads_started":2,"wal-append.queue_time_us":762573}
I20260709 13:02:55.056013  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.056710  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:02:55.059329  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.060117  1588 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 7.
W20260709 13:02:54.998615  1557 connection.cc:256] RPC call timeout handler was delayed by -1.9247854349999898s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
I20260709 13:02:55.000499  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.067589  2313 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:02:55.002818  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.024714  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:55.068657  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.006708  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:54.908193  5815 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.069448  1588 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
I20260709 13:02:54.993284  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.031248  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:55.035941  5795 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:55.004271  5667 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:55.036093  5837 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:02:55.040695  1584 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
W20260709 13:02:55.104969  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0679682140000182s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
I20260709 13:02:54.978466  5663 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:55.061688  5880 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:55.063278  5830 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.039707  5882 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.114856  2315 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:55.115095  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:55.115753  2312 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 7: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:55.069947  1740 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:02:55.076972  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 169556us
I20260709 13:02:55.060635  1739 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:02:55.069564  1557 connection.cc:256] RPC call timeout handler was delayed by -1.9004750819999572s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
W20260709 13:02:55.100610  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69461.7us
W20260709 13:02:55.118079  1557 connection.cc:256] RPC call timeout handler was delayed by -1.8966666420000138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2755s for a response
I20260709 13:02:54.973683  5807 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:55.090442  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60379.6us
I20260709 13:02:55.118994  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:55.069120  1587 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:02:55.102770  2309 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:02:55.102873  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:55.052749  5896 consensus_queue.cc:1243] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:02:54.994993  5633 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:55.121410  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.121628  5633 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:55.122054  5633 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:55.122130  1740 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:55.123299  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.123874  1740 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:02:55.026378  1589 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:55.061254  5493 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.118000  1706 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:55.126349  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.102584  5714 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:54.957724  5897 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 13:02:55.127175  5897 consensus_queue.cc:1243] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:02:55.127864  1589 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:02:55.102404  5667 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:55.118569  1557 connection.cc:256] RPC call timeout handler was delayed by -1.8905854349999913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260709 13:02:55.108448  4967 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:55.173544  2314 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:55.173957  1589 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.10-9.10]
W20260709 13:02:55.169947  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51062.2us
I20260709 13:02:55.170145  1591 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 13:02:55.061394  5805 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.170758  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52845.5us
I20260709 13:02:55.066162  5655 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.172549  1557 connection.cc:256] RPC call timeout handler was delayed by -1.823085434999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260709 13:02:55.181695  1557 connection.cc:256] RPC call timeout handler was delayed by -1.8203854349999915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2169s for a response
I20260709 13:02:55.175318  1589 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: [6.4-11.7]
I20260709 13:02:55.046454  5822 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.191550  1589 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:55.048027  5548 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.175406  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:55.199486  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1804) took 3755 ms (client timeout 2053 ms). Trace:
W20260709 13:02:55.181980  1557 connection.cc:256] RPC call timeout handler was delayed by -1.6980937870000048s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260709 13:02:55.185421  1587 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:02:55.187443  5756 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:55.199980  1735 rpcz_store.cc:269] 0709 13:02:51.433461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.436250 (+  2789us) service_pool.cc:224] Handling call
0709 13:02:55.189119 (+3752869us) spinlock_profiling.cc:243] Waited 862 ms on lock 0x7b5400404554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:55.189227 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1981955456}
I20260709 13:02:55.203030  1668 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:55.174427  5667 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:55.173777  5574 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 13:02:55.207950  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:55.213148  1739 raft_consensus.cc:2368] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
W20260709 13:02:55.209196  1557 connection.cc:256] RPC call timeout handler was delayed by -1.2110854349999727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2846s for a response
W20260709 13:02:55.220257  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1254058209999585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
W20260709 13:02:55.220783  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1192854349999379s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2948s for a response
W20260709 13:02:55.221221  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0228058209999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260709 13:02:55.221632  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0228058209999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
I20260709 13:02:55.176700  5714 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.177421  4967 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.210005  1661 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:55.210171  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.116624  5864 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 13:02:55.240821  5864 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:55.225417  1660 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:55.224385  5663 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.243602  5815 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.231616  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:55.236048  5910 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 13:02:55.235910  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.235937  5477 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:55.237102  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.210179  5830 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.237782  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:55.238763  5894 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:55.213804  5756 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:55.210125  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2583) took 3300 ms (client timeout 2463 ms). Trace:
I20260709 13:02:55.252843  1736 raft_consensus.cc:2410] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:02:55.261636  1588 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:55.268749  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.273254  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:55.283641  1585 rpcz_store.cc:269] 0709 13:02:51.888150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:51.888272 (+   122us) service_pool.cc:224] Handling call
0709 13:02:55.188261 (+3299989us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b54002701d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:55.188392 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840200576}
I20260709 13:02:55.251637  5837 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.287067  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 628) took 2108 ms. Trace:
I20260709 13:02:55.287825  5837 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.288654  5882 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:55.287459  1735 rpcz_store.cc:276] 0709 13:02:53.179002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.286059 (+2107057us) service_pool.cc:224] Handling call
0709 13:02:55.286500 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.286703 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:55.286729 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.286792 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:55.286833 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.286903 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.287007 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.290390  1735 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:55.291158  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 629) took 2111 ms. Trace:
I20260709 13:02:55.291594  1735 rpcz_store.cc:276] 0709 13:02:53.179551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.289800 (+2110249us) service_pool.cc:224] Handling call
0709 13:02:55.290153 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.290853 (+   700us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.291091 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.241321  5864 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.302131  5830 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.245422  4967 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:55.234467  5574 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:02:55.307389  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 132088us
I20260709 13:02:55.299220  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:55.304731  5756 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:55.311832  5913 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 13:02:55.268484  5836 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:55.316351  1739 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:55.394316  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1827) took 2400 ms. Trace:
I20260709 13:02:55.381644  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.394726  1739 rpcz_store.cc:276] 0709 13:02:52.993987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.275948 (+2281961us) service_pool.cc:224] Handling call
0709 13:02:55.316160 (+ 40212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.394058 (+ 77898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.394252 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.395574  2314 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:55.320412  1591 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
W20260709 13:02:55.397092  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.314229  1666 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:55.398650  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:55.403318  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2558) took 2971 ms (client timeout 1601 ms). Trace:
W20260709 13:02:55.405325  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.408110  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:55.424293  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 126014us
I20260709 13:02:55.425153  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:55.403892  1737 rpcz_store.cc:269] 0709 13:02:52.432132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:52.563049 (+130917us) service_pool.cc:224] Handling call
0709 13:02:55.389869 (+2826820us) spinlock_profiling.cc:243] Waited 2.18 s on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:55.403199 (+ 13330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5024043392}
I20260709 13:02:55.430512  1666 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:55.377058  5909 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.306s	user 0.000s	sys 0.001s
I20260709 13:02:55.432449  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.435966  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.454747  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 635) took 2250 ms. Trace:
W20260709 13:02:55.455112  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 342684us
I20260709 13:02:55.283921  5542 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.146s
I20260709 13:02:55.455313  1735 rpcz_store.cc:276] 0709 13:02:53.204515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.292563 (+2088048us) service_pool.cc:224] Handling call
0709 13:02:55.292934 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.293219 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:55.293837 (+   618us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.454379 (+160542us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.454432 (+    53us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:55.454475 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.454559 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.454690 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":124,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"thread_start_us":160180,"threads_started":1}
I20260709 13:02:55.456341  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:55.380360  5916 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 13:02:55.457085  1588 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:55.275039  5754 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.158s
I20260709 13:02:55.464759  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:55.457446  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.423480  5828 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.356925  5842 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 13:02:55.436625  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.462654  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.471729  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:02:55.467811  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:55.283478  5880 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.147s
I20260709 13:02:55.460363  2309 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:55.428990  5731 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.473973  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.476171  1737 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:02:55.369324  5574 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.478374  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53313.5us
I20260709 13:02:55.481925  1666 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:55.485543  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:55.312163  5917 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.294s
I20260709 13:02:55.468448  5836 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:55.469437  5882 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:55.495476  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.502803  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:55.381932  5864 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.505358  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.506377  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.433403  5702 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:55.430433  5894 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 13:02:55.473977  5842 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:55.514659  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:55.562359  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66971.2us
W20260709 13:02:55.564287  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54447.6us
I20260709 13:02:55.381064  5928 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.126s
I20260709 13:02:55.503541  5922 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 13:02:55.377416  5714 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.305877  5477 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:55.581607  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75358.1us
I20260709 13:02:55.620117  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.620255  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:55.459359  5633 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.626173  5633 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:55.626684  5574 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:55.627676  5939 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:55.628312  5939 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:55.595813  5822 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:55.502135  5828 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:55.565973  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 637) took 2341 ms. Trace:
I20260709 13:02:55.644461  1739 rpcz_store.cc:276] 0709 13:02:53.225003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.494087 (+2269084us) service_pool.cc:224] Handling call
0709 13:02:55.565392 (+ 71305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.565611 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:55.565638 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.565684 (+    46us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:55.565717 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.565777 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.565911 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.645920  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.560173  5862 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:55.509027  5807 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:55.610865  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.653488  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.565243  5815 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:55.560365  5655 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.574702  5821 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:55.615778  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 493634us
I20260709 13:02:55.665875  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.617280  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.620923  2318 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:55.614959  5730 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:02:55.618294  5929 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.108s	user 0.002s	sys 0.000s
I20260709 13:02:55.623494  5830 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.623776  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.433713  5883 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.131s
I20260709 13:02:55.597237  5542 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:55.610163  5864 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:55.598704  5454 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:55.629323  5939 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.650873  5926 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 13:02:55.654405  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 252286us
I20260709 13:02:55.695680  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.707304  1739 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:02:55.699867  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76168.4us
I20260709 13:02:55.695690  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.699467  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2564) took 3029 ms. Trace:
I20260709 13:02:55.708698  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.711560  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:55.711864  1557 leader_election.cc:403] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:02:55.712576  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.665670  5828 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.701716  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.688479  5730 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:55.708691  1738 rpcz_store.cc:276] 0709 13:02:52.669901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.947352 (+2277451us) service_pool.cc:224] Handling call
0709 13:02:54.979378 (+ 32026us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:55.030491 (+ 51113us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:55.107809 (+ 77318us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.340253 (+232444us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.343124 (+  2871us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.343325 (+   201us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:55.363433 (+ 20108us) log.cc:844] Serialized 52 byte log entry
0709 13:02:55.363611 (+   178us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:55.363813 (+   202us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.393689 (+ 29876us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.393779 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.699137 (+305358us) raft_consensus.cc:1685] finished
0709 13:02:55.699173 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.699385 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56483,"prepare.queue_time_us":175455,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":13366,"raft.queue_time_us":83732,"raft.run_cpu_time_us":504,"raft.run_wall_time_us":56572,"spinlock_wait_cycles":189813888,"thread_start_us":176905,"threads_started":2,"wal-append.queue_time_us":367910}
I20260709 13:02:55.724395  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.741286  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.700870  5454 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:55.742380  1659 raft_consensus.cc:2368] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:55.692095  5862 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.723328  5921 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:55.723474  5754 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:55.715265  1665 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:55.738484  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:55.698756  5542 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.731376  5702 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:55.710119  5830 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 13:02:55.712903  5807 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:55.716634  5880 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:02:55.744717  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.739532  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.690558  5821 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.756042  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.723724  5883 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.763851  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.765379  1738 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:55.770742  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.777419  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 638) took 2244 ms. Trace:
I20260709 13:02:55.746035  5882 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:55.772024  5922 consensus_queue.cc:1243] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 7.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:02:55.777951  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.661435  5842 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.781996  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.742956  5861 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.764274  5542 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:55.780128  1739 rpcz_store.cc:276] 0709 13:02:53.533924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.751164 (+2217240us) service_pool.cc:224] Handling call
0709 13:02:55.776843 (+ 25679us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.777057 (+   214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:55.777086 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.777133 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:02:55.777167 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.777227 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.777351 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.792614  1739 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:55.778648  5711 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:55.750157  5921 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:55.755915  5837 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:55.782938  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2567) took 2505 ms. Trace:
I20260709 13:02:55.795878  1738 rpcz_store.cc:276] 0709 13:02:53.277413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.741574 (+2464161us) service_pool.cc:224] Handling call
0709 13:02:55.765191 (+ 23617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.782643 (+ 17452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.782863 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.787919  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:55.771659  5807 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:55.800118  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:55.800819  1593 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:55.794024  1739 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:55.801676  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:55.799117  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74825.2us
I20260709 13:02:55.803165  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:55.804971  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 636) took 2587 ms. Trace:
I20260709 13:02:55.786134  5865 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:55.807957  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.743239  5477 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.790252  5922 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.810715  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:55.812187  1738 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Stepping down as leader of term 6
I20260709 13:02:55.814873  1738 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:55.798481  5814 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.849s	user 0.007s	sys 0.007s
I20260709 13:02:55.816617  1738 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.817915  1738 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 6 FOLLOWER]: Advancing to term 7
I20260709 13:02:55.813938  1739 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.821206  1739 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:55.806159  1735 rpcz_store.cc:276] 0709 13:02:53.217553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.473084 (+2255531us) service_pool.cc:224] Handling call
0709 13:02:55.473360 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.473596 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:02:55.689280 (+215684us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.804588 (+115308us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.804646 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:55.804703 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.804786 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.804912 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18819,"prepare.queue_time_us":249194,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"thread_start_us":281667,"threads_started":2,"wal-append.queue_time_us":113479}
I20260709 13:02:55.789817  5719 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:55.813974  5880 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:55.826453  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:55.799378  5921 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:55.785491  5883 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.827071  5882 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:55.806565  5807 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:55.807806  5756 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:55.799780  5730 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.833957  5865 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.850674  5730 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:55.850924  5865 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.852922  5667 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.000s
I20260709 13:02:55.805982  5842 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:55.833699  5862 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.804028  5711 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.844652  1739 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:55.848286  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.852839  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.823871  5937 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:55.794693  5861 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.851051  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 642) took 1401 ms. Trace:
I20260709 13:02:55.887797  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.858855  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.841354  1738 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 13:02:55.861706  5940 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 13:02:55.861802  5711 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.863112  5949 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:55.863626  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1846) took 1092 ms. Trace:
I20260709 13:02:55.825304  5922 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.866662  5792 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:02:55.902179  5792 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:55.903352  5861 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.903136  5792 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:55.887954  5872 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 13:02:55.891480  1662 rpcz_store.cc:276] 0709 13:02:54.771508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.771693 (+   185us) service_pool.cc:224] Handling call
0709 13:02:54.772061 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.847847 (+ 75786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.863342 (+1015495us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:55.863554 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172872064}
I20260709 13:02:55.904331  5792 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:55.905016  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.868166  1742 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:55.906185  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:55.907156  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.907336  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.881382  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1858) took 1432 ms. Trace:
I20260709 13:02:55.884663  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.890165  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 639) took 1146 ms. Trace:
I20260709 13:02:55.797739  5454 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.912370  2318 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:02:55.898080  5949 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:55.898731  5882 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:55.917004  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.899504  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.900789  5431 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.831502  5821 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:55.908263  1738 rpcz_store.cc:276] 0709 13:02:54.744311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.796855 (+1052544us) service_pool.cc:224] Handling call
0709 13:02:55.812044 (+ 15189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.889783 (+ 77739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.890091 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.928843  1738 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:55.929342  1738 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:55.909088  1739 rpcz_store.cc:276] 0709 13:02:54.448588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.792088 (+1343500us) service_pool.cc:224] Handling call
0709 13:02:55.792481 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.881058 (+ 88577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.881319 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:55.892776  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:55.884676  5944 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:55.914499  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.915386  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:02:55.887584  4967 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.524s	user 0.008s	sys 0.004s
I20260709 13:02:55.887615  1663 rpcz_store.cc:276] 0709 13:02:54.449822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.449978 (+   156us) service_pool.cc:224] Handling call
0709 13:02:54.450525 (+   547us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.690222 (+239697us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:54.690255 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:54.720595 (+ 30340us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.960369 (+239774us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:54.960417 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.383009 (+422592us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b4800210258. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:02:55.850689 (+467680us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:55.850754 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:55.850802 (+    48us) raft_consensus.cc:1685] finished
0709 13:02:55.850817 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.850987 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":505,"raft.run_cpu_time_us":2712,"raft.run_wall_time_us":681019,"spinlock_wait_cycles":560189056,"thread_start_us":187350,"threads_started":1,"wal-append.queue_time_us":277995}
I20260709 13:02:55.845767  5838 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 13:02:55.931280  1557 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:55.937410  1557 leader_election.cc:403] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:02:55.887290  5814 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:55.814436  5828 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:55.949980  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:55.891247  5811 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:55.934291  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.876549  5667 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.934423  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:55.917353  5949 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:55.950129  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.901325  5937 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:55.953773  1705 leader_election.cc:400] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:55.956629  1556 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:55.965939  1663 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:55.970826  1556 leader_election.cc:403] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:02:55.965890  1738 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.932147  5454 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:55.934845  5861 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:55.969117  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2577) took 1012 ms. Trace:
I20260709 13:02:55.933413  5822 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:02:55.960835  5835 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.433s	user 0.032s	sys 0.017s
I20260709 13:02:55.976938  1735 rpcz_store.cc:276] 0709 13:02:54.956542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.859608 (+903066us) service_pool.cc:224] Handling call
0709 13:02:55.859954 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.860175 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:55.909457 (+ 49282us) log.cc:844] Serialized 14 byte log entry
0709 13:02:55.968698 (+ 59241us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:55.968761 (+    63us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:55.968820 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:55.968903 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:55.969052 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":30882,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":98,"thread_start_us":58738,"threads_started":1}
I20260709 13:02:55.988265  1738 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:55.990586  1705 leader_election.cc:400] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:55.992959  1735 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 13:02:55.996088  1735 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:02:55.996332  1735 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 13:02:55.996593  1735 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:55.998046  2311 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:55.994962  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:55.999285  1663 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:02:56.001389  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:55.932636  5431 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.019248  5431 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:55.972574  5842 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:56.002712  5822 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:56.010279  5950 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:02:56.011118  5807 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:55.991411  5835 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.464s	user 0.033s	sys 0.017s
I20260709 13:02:56.012101  5898 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.012503  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:55.970408  5880 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 13:02:55.960158  5754 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 13:02:56.094686  2311 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:02:55.952924  4967 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.048359  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 593) took 1068 ms. Trace:
I20260709 13:02:55.938119  5944 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:55.963881  5814 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:56.089679  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.195422  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.091593  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.124643  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.135921  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141506us
I20260709 13:02:56.206861  1705 leader_election.cc:400] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:56.207355  1705 leader_election.cc:403] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:02:56.024979  5861 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:56.145993  1665 raft_consensus.cc:2368] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:55.975198  5939 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
W20260709 13:02:56.232345  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.095691  5898 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.258981  5944 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:56.262374  5952 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 13:02:56.263751  5956 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 13:02:56.147682  1738 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:56.269600  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1870) took 1278 ms. Trace:
I20260709 13:02:56.148336  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:56.270306  1738 rpcz_store.cc:276] 0709 13:02:54.991576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.928231 (+936655us) service_pool.cc:224] Handling call
0709 13:02:55.928666 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.269047 (+340381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.269463 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:56.271662  5955 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 13:02:56.272195  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:56.273209  1738 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:55.973161  5811 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.152194  1593 raft_consensus.cc:2955] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 2 to 6, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:02:56.084666  5842 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:02:56.138494  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53279.8us
I20260709 13:02:56.170923  1591 rpcz_store.cc:276] 0709 13:02:54.979732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.809561 (+829829us) service_pool.cc:224] Handling call
0709 13:02:55.838500 (+ 28939us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.838822 (+   322us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:02:55.860349 (+ 21527us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.047827 (+187478us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:56.047902 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:02:56.047980 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.048090 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:56.048291 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31187,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"thread_start_us":186971,"threads_started":1}
I20260709 13:02:56.211413  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:56.291566  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80255.6us
W20260709 13:02:56.184974  5950 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:02:56.293205  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.300557  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.298158  2311 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 13:02:56.292412  5864 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.587s	user 0.006s	sys 0.006s
I20260709 13:02:56.327327  1591 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:56.333735  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.395056  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.336642  1664 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:56.306545  5883 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:56.298929  5944 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:56.399678  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:56.404321  1663 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:56.167919  5937 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.417260  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2421) took 1181 ms. Trace:
I20260709 13:02:56.417878  1663 rpcz_store.cc:276] 0709 13:02:55.236320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.323285 (+1086965us) service_pool.cc:224] Handling call
0709 13:02:56.417165 (+ 93880us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.420469  1591 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:56.312839  5826 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.316371  5821 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:56.289036  5939 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:02:56.230473  5477 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9.
I20260709 13:02:56.331637  5917 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:56.401279  5864 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.455621  1591 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:56.325866  5922 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:56.428725  5542 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.630s	user 0.003s	sys 0.011s
W20260709 13:02:56.423820  5921 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.570s	user 0.006s	sys 0.006s
I20260709 13:02:56.473027  1663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:02:56.480863  1663 raft_consensus.cc:606] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:56.445320  5667 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:56.464625  5958 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:02:56.460502  5821 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:02:56.512282  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 211840us
I20260709 13:02:56.513306  1705 leader_election.cc:400] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:56.467679  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2621) took 1040 ms. Trace:
I20260709 13:02:56.445057  5826 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.519745  1591 rpcz_store.cc:276] 0709 13:02:55.426721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.427466 (+1000745us) service_pool.cc:224] Handling call
0709 13:02:56.455235 (+ 27769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.462196 (+  6961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.467571 (+  5375us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.521092  1705 leader_election.cc:403] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:02:56.511472  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2583) took 1398 ms. Trace:
W20260709 13:02:56.493829  5795 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.391s	user 0.010s	sys 0.006s
W20260709 13:02:56.476235  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.532745  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:02:56.507474  5807 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:56.443482  5842 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.540532  1660 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:56.465911  5864 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.473973  5917 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.559652  5969 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:02:56.560670  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.560590  5969 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:56.561882  5969 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.563962  5821 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:56.559950  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:56.544912  5964 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
I20260709 13:02:56.548370  5477 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10.
I20260709 13:02:56.568859  1663 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:56.552675  1591 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 13:02:56.686251  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 643) took 1695 ms. Trace:
W20260709 13:02:56.573756  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.006s	sys 0.006s
W20260709 13:02:56.686928  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.006s	sys 0.006s
I20260709 13:02:56.686719  1663 rpcz_store.cc:276] 0709 13:02:54.990647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.539450 (+1548803us) service_pool.cc:224] Handling call
0709 13:02:56.568589 (+ 29139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.685902 (+117313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.686184 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:56.688237  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 757131us
W20260709 13:02:56.574055  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.000s	sys 0.011s
W20260709 13:02:56.689437  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.000s	sys 0.012s
W20260709 13:02:56.689855  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 703352us
W20260709 13:02:56.628439  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.509318  5542 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:56.691250  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.692057  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132213us
I20260709 13:02:56.548770  5865 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:56.564097  5960 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
I20260709 13:02:56.508072  5921 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.715503  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.723555  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.564869  5821 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:56.748549  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 749) took 1082 ms. Trace:
I20260709 13:02:56.750856  2312 rpcz_store.cc:276] 0709 13:02:55.665731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.016744 (+351013us) service_pool.cc:224] Handling call
0709 13:02:56.017103 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.017490 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.017542 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.018056 (+   514us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.275236 (+257180us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:56.275286 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.275361 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:56.748233 (+472872us) raft_consensus.cc:1685] finished
0709 13:02:56.748261 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:56.748455 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54206,"raft.queue_time_us":7140,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":21724,"thread_start_us":256285,"threads_started":1,"wal-append.queue_time_us":291229}
I20260709 13:02:56.755018  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.759116  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.761831  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:02:56.758383  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 353095us
I20260709 13:02:56.565451  5814 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:56.685000  5807 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.714430  5667 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:56.776825  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2572) took 2082 ms. Trace:
I20260709 13:02:56.707475  4967 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:56.749886  5972 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.787405  1666 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 9: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:56.745703  5826 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:56.757582  5922 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:56.721789  5944 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:56.774119  5822 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.772433  5477 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:02:56.800472  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.644s (SENT)
W20260709 13:02:56.803201  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.804037  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.777647  5898 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.824712  5898 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.826390  5756 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:02:56.827302  5756 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:56.827939  5756 consensus_queue.cc:1243] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 11.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:02:56.828305  5454 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.856s	user 0.011s	sys 0.000s
I20260709 13:02:56.828789  5880 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:56.829321  5880 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:56.829455  5967 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:56.829921  5880 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:56.830192  5967 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:56.831482  5477 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:56.831365  5967 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.832043  5477 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.807227  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 411) took 3536 ms. Trace:
I20260709 13:02:56.807382  2316 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.10-9.10]
I20260709 13:02:56.807699  5944 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.842603  5967 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:56.797217  5972 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.844672  5880 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.813905  5821 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.817591  5667 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:56.750424  5865 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:56.788012  5807 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.818339  5976 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:56.833782  5830 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:56.822937  5542 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:56.833920  2310 rpcz_store.cc:276] 0709 13:02:53.270570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.270805 (+   235us) service_pool.cc:224] Handling call
0709 13:02:53.271185 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:54.456589 (+1185404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.806868 (+2350279us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:56.807149 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2330970368}
I20260709 13:02:56.837030  5811 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.825376  5898 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:56.845289  5792 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.940s	user 0.012s	sys 0.000s
I20260709 13:02:56.810524  5864 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.853175  2318 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:56.789845  1589 rpcz_store.cc:276] 0709 13:02:54.694523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.197176 (+502653us) service_pool.cc:224] Handling call
0709 13:02:55.197494 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.197737 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:55.210657 (+ 12920us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.565640 (+1354983us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b1800430a20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a
0709 13:02:56.776287 (+210647us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:56.776396 (+   109us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:02:56.776466 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.776570 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:56.776733 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12715,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":145779,"spinlock_wait_cycles":50688896,"thread_start_us":163356,"threads_started":2,"wal-append.queue_time_us":265725,"wal-append.run_cpu_time_us":2077,"wal-append.run_wall_time_us":1240342}
W20260709 13:02:56.833480  5882 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.902s	user 0.009s	sys 0.000s
W20260709 13:02:56.860309  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:56.862380  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138897us
I20260709 13:02:56.786656  5814 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:56.882747  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.854250  5667 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:02:56.867645  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 631) took 3687 ms (client timeout 1682 ms). Trace:
W20260709 13:02:56.874715  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 123493us
I20260709 13:02:56.898743  1557 leader_election.cc:400] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12.
I20260709 13:02:56.866966  1741 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
W20260709 13:02:56.898851  1733 rpcz_store.cc:269] 0709 13:02:53.180180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.208957 (+ 28777us) service_pool.cc:224] Handling call
0709 13:02:56.866518 (+3657561us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400435654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:56.867538 (+  1020us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2338946944}
I20260709 13:02:56.899760  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.900566  1589 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:02:56.900769  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:56.901657  1733 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:56.903681  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:56.905804  1589 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:56.908850  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.876986  5830 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:56.916364  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2584) took 1801 ms. Trace:
I20260709 13:02:56.896528  5882 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:56.905241  5667 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:56.926194  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:02:56.928750  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.928701  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:56.929471  1591 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
W20260709 13:02:56.929692  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 167966us
W20260709 13:02:56.917932  5702 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.150s	user 0.013s	sys 0.003s
I20260709 13:02:56.911603  5978 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:56.935732  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.916919  1738 rpcz_store.cc:276] 0709 13:02:55.114554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.274949 (+1160395us) service_pool.cc:224] Handling call
0709 13:02:56.275730 (+   781us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.276000 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:56.276769 (+   769us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.915903 (+639134us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:56.915982 (+    79us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:56.916045 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.916138 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:56.916290 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":199,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":30068,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":35959,"thread_start_us":638651,"threads_started":2,"wal-append.queue_time_us":565835}
I20260709 13:02:56.932660  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.930238  1589 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:56.940333  1733 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:56.941013  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 641) took 1821 ms. Trace:
I20260709 13:02:56.941650  1733 rpcz_store.cc:276] 0709 13:02:55.120021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.925812 (+1805791us) service_pool.cc:224] Handling call
0709 13:02:56.940058 (+ 14246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.940757 (+   699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.940958 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:56.940975  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.947117  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.949888  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.878621  5976 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:56.965276  1733 pending_rounds.cc:85] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:02:56.965852  1733 pending_rounds.cc:107] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:02:56.966171  1733 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:56.966820  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.917213  5754 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 13:02:56.968802  1738 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:02:56.938014  5477 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.938853  5702 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:56.971285  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:56.972347  1738 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 13:02:56.980472  1738 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:02:56.986718  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.977068  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.003110  1591 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:56.945261  5972 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:56.969946  5982 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 13:02:57.009788  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:56.928097  5977 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260709 13:02:57.003559  5921 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.010977  1589 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:02:57.004680  5792 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.016454  5880 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.017365  5807 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:56.972121  5814 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.019354  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:57.019945  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.023820  1591 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:02:57.024199  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 88554.4us
I20260709 13:02:57.025995  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.026022  1557 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13.
I20260709 13:02:57.027701  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:56.968467  5830 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.041786  5830 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.050120  5805 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.051216  5878 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:57.067307  5878 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:57.070106  5983 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:02:57.073975  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.074977  1589 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:02:57.075161  5904 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.441s	user 0.000s	sys 0.004s
W20260709 13:02:57.075443  5949 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.080s	user 0.006s	sys 0.006s
I20260709 13:02:57.075870  5949 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.077286  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2556) took 4681 ms. Trace:
I20260709 13:02:57.077733  1734 rpcz_store.cc:276] 0709 13:02:52.395474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.273935 (+1878461us) service_pool.cc:224] Handling call
0709 13:02:54.274424 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:54.323297 (+ 48873us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:54.357916 (+ 34619us) log.cc:844] Serialized 14 byte log entry
0709 13:02:54.492142 (+134226us) log.cc:844] Serialized 14 byte log entry
0709 13:02:54.492665 (+   523us) log.cc:844] Serialized 14 byte log entry
0709 13:02:54.492835 (+   170us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:54.495697 (+  2862us) log.cc:844] Serialized 26 byte log entry
0709 13:02:54.495874 (+   177us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:54.495914 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:54.495990 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.015099 (+2519109us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b18004811e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:57.074159 (+ 59060us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:57.077046 (+  2887us) raft_consensus.cc:1685] finished
0709 13:02:57.077070 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.077222 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48572,"prepare.queue_time_us":202856,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":1858,"raft.queue_time_us":47869,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":8340,"spinlock_wait_cycles":22344576,"thread_start_us":165708,"threads_started":2,"wal-append.queue_time_us":187310}
I20260709 13:02:57.079326  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2412) took 2156 ms. Trace:
I20260709 13:02:57.079757  1665 rpcz_store.cc:276] 0709 13:02:54.922714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.763498 (+840784us) service_pool.cc:224] Handling call
0709 13:02:57.079126 (+1315628us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54001a9854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.079265 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":841851776}
I20260709 13:02:57.080780  1734 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:02:57.081991  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.064113  1591 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:02:57.087282  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:57.088793  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.095137  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.096733  1740 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:02:57.113922  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2590) took 1304 ms. Trace:
I20260709 13:02:57.114336  1740 rpcz_store.cc:276] 0709 13:02:55.809275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.096070 (+1286795us) service_pool.cc:224] Handling call
0709 13:02:57.096366 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.113572 (+ 17206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.113848 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.116106  1740 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:57.116743  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2591) took 1146 ms. Trace:
I20260709 13:02:57.080824  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.117100  1740 rpcz_store.cc:276] 0709 13:02:55.969955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.115284 (+1145329us) service_pool.cc:224] Handling call
0709 13:02:57.115691 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.116481 (+   790us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.116685 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.119371  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.101693  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:57.119988  1665 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:57.089582  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2586) took 1471 ms. Trace:
I20260709 13:02:57.120793  1734 rpcz_store.cc:276] 0709 13:02:55.610201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.079942 (+1469741us) service_pool.cc:224] Handling call
0709 13:02:57.080340 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.081247 (+   907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.081459 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.100808  1705 leader_election.cc:400] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 7 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8.
I20260709 13:02:57.082835  1662 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 13:02:57.123202  1734 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 13:02:57.123486  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.123906  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2441) took 1340 ms (client timeout 1644 ms). Trace:
W20260709 13:02:57.124300  1659 rpcz_store.cc:269] 0709 13:02:55.783709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.094816 (+1311107us) service_pool.cc:224] Handling call
0709 13:02:57.123847 (+ 29031us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.125329  1740 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:57.127373  1740 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 13:02:57.127373  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:57.129293  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.128316  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.132157  1665 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:02:57.083662  5976 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.134217  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.135051  5922 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.136983  5922 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:57.087657  5898 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.137427  5922 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:57.085613  5949 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.137889  5922 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.097574  5659 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.097708  5792 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:57.086077  5878 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.146114  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.120666  5814 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.155488  1591 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:57.128825  5972 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:02:57.157346  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.159051  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:57.160759  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:57.161013  5878 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 13:02:57.171167  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2585) took 1573 ms. Trace:
I20260709 13:02:57.171612  1733 rpcz_store.cc:276] 0709 13:02:55.598116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.942525 (+1344409us) service_pool.cc:224] Handling call
0709 13:02:56.965009 (+ 22484us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:56.978572 (+ 13563us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.978607 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:57.009458 (+ 30851us) log.cc:844] Serialized 52 byte log entry
0709 13:02:57.063629 (+ 54171us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:02:57.063674 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.063834 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.170894 (+107060us) raft_consensus.cc:1685] finished
0709 13:02:57.170919 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.171099 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12220,"raft.queue_time_us":345,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":25152,"thread_start_us":53779,"threads_started":1,"wal-append.queue_time_us":124016}
I20260709 13:02:57.131525  5480 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:02:57.176546  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:56.522559  1735 rpcz_store.cc:276] 0709 13:02:55.112996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.003865 (+890869us) service_pool.cc:224] Handling call
0709 13:02:56.004167 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.004401 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:02:56.019754 (+ 15353us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.508726 (+488972us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:56.508792 (+    66us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:56.508844 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.508968 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:56.511402 (+  2434us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15627,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":62,"raft.queue_time_us":2520,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6928,"thread_start_us":487546,"threads_started":1}
I20260709 13:02:57.141842  5984 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:02:57.160447  5972 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:57.189975  5878 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:57.190513  5878 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.191424  5878 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.192730  5878 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.182711  5972 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.157271  5976 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 13:02:57.197434  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.160935  5814 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.196307  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:57.201328  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.203378  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.180768  5898 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:57.205544  5944 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:02:57.211362  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.208696  5898 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:57.249902  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.250627  1659 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.238761  5480 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:02:57.221155  5984 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:57.221451  5944 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.258916  5814 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.259725  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 642) took 2132 ms. Trace:
I20260709 13:02:57.263454  1738 rpcz_store.cc:276] 0709 13:02:55.127545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.938116 (+1810571us) service_pool.cc:224] Handling call
0709 13:02:56.968566 (+ 30450us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:57.019259 (+ 50693us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.019293 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:57.019895 (+   602us) log.cc:844] Serialized 52 byte log entry
0709 13:02:57.143154 (+123259us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:57.143221 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.143323 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.259444 (+116121us) raft_consensus.cc:1685] finished
0709 13:02:57.259477 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.259655 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9056,"raft.queue_time_us":15432,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":22601,"thread_start_us":105307,"threads_started":1,"wal-append.queue_time_us":159466}
W20260709 13:02:57.261792  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2407) took 2428 ms (client timeout 2445 ms). Trace:
W20260709 13:02:57.267136  1667 rpcz_store.cc:269] 0709 13:02:54.833697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.992909 (+159212us) service_pool.cc:224] Handling call
0709 13:02:57.261579 (+2268670us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.261732 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1364974976}
I20260709 13:02:57.250281  5898 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.273285  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.274772  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.261389  5984 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.318113  5984 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.319509  5984 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:57.240659  5667 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.287259  5881 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.288244  5944 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.290143  1659 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:02:57.296818  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2414) took 2245 ms (client timeout 2179 ms). Trace:
I20260709 13:02:57.261729  5480 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:57.303946  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:57.310106  5948 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.490s	user 0.002s	sys 0.002s
I20260709 13:02:57.312295  5980 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:02:57.313169  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2574) took 2609 ms. Trace:
I20260709 13:02:57.313602  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.314289  5968 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:02:57.314486  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 286897us
W20260709 13:02:57.314931  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 187675us
I20260709 13:02:57.315438  5949 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:57.315734  5928 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260709 13:02:57.317037  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.301388  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 644) took 2310 ms. Trace:
W20260709 13:02:57.331102  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.293164  5922 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:57.322088  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 126083us
W20260709 13:02:57.322355  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.338666  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.339074  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.339813  1659 raft_consensus.cc:2410] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:02:57.340303  1586 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:57.331035  1663 rpcz_store.cc:276] 0709 13:02:54.991120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.724338 (+1733218us) service_pool.cc:224] Handling call
0709 13:02:56.746351 (+ 22013us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.884844 (+138493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:56.884889 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:56.908065 (+ 23176us) log.cc:844] Serialized 26 byte log entry
0709 13:02:56.997914 (+ 89849us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:56.997980 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.300822 (+302842us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:57.300901 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.300961 (+    60us) raft_consensus.cc:1685] finished
0709 13:02:57.300974 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.301306 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":992,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":9474,"spinlock_wait_cycles":317442304,"thread_start_us":66123,"threads_started":1,"wal-append.queue_time_us":94085}
W20260709 13:02:57.314376  5961 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.807s	user 0.002s	sys 0.000s
I20260709 13:02:57.325402  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.326010  1587 rpcz_store.cc:276] 0709 13:02:54.703402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.202353 (+498951us) service_pool.cc:224] Handling call
0709 13:02:55.203671 (+  1318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.203823 (+   152us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:56.808775 (+1604952us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b180043f120. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf
0709 13:02:56.808977 (+   202us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.936940 (+127963us) log.cc:844] Serialized 14 byte log entry
0709 13:02:57.312001 (+375061us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b180043f120. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdf1eb158f 00007fcdeaef0571 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd
0709 13:02:57.312574 (+   573us) log.cc:844] Serialized 14 byte log entry
0709 13:02:57.312786 (+   212us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.312833 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:57.312879 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.312963 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.313108 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":391116,"prepare.queue_time_us":211944,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":150,"raft.queue_time_us":389728,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":14691,"spinlock_wait_cycles":19389952,"thread_start_us":193783,"threads_started":3,"wal-append.queue_time_us":127872}
I20260709 13:02:57.326222  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:57.328079  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.329672  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.330603  1734 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
W20260709 13:02:57.321408  1660 rpcz_store.cc:269] 0709 13:02:55.051527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.381332 (+329805us) service_pool.cc:224] Handling call
0709 13:02:57.296524 (+1915192us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400117e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.296699 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2632715648}
I20260709 13:02:57.333926  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:02:57.335978  5957 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.085s	user 0.000s	sys 0.002s
I20260709 13:02:57.345846  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.346568  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:57.344384  1587 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:57.343529  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:57.343072  1667 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:02:57.350423  1587 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:57.351629  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.344002  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.352929  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:57.342147  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.327562  5949 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.357688  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.362426  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.364248  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.364733  1586 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:02:57.359546  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.365994  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:02:57.342147  5584 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 13:02:57.337076  5928 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:02:57.325978  5480 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.370745  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.370745  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.337386  5980 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:57.371474  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.358791  1587 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.373461  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.377807  1735 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:02:57.377982  1587 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.379073  5980 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.370285  5979 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:57.381875  5979 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:57.382268  5979 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:02:57.384430  5962 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.926s	user 0.001s	sys 0.001s
W20260709 13:02:57.374730  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.374851  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.359807  5989 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.374032  1586 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:02:57.386922  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.387024  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.366673  5373 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:02:57.411958  2315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 741) took 2475 ms (client timeout 2200 ms). Trace:
I20260709 13:02:57.355973  5982 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.413894  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.362295  5949 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.355849  5702 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:02:57.389076  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.445716  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.447707  1663 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:02:57.456809  5949 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.420253  1587 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:02:57.422420  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.458632  1738 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 13:02:57.459546  5949 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.434536  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.460553  1659 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:57.461083  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2587) took 2291 ms. Trace:
I20260709 13:02:57.461856  1659 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.462478  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2418) took 2227 ms. Trace:
I20260709 13:02:57.462791  1659 rpcz_store.cc:276] 0709 13:02:55.235136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.461296 (+2226160us) service_pool.cc:224] Handling call
0709 13:02:57.461675 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.462246 (+   571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.462422 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.461488  1585 rpcz_store.cc:276] 0709 13:02:55.169864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.813499 (+643635us) service_pool.cc:224] Handling call
0709 13:02:55.826684 (+ 13185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.908204 (+ 81520us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:02:55.934566 (+ 26362us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.430907 (+496341us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:56.430969 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:56.431022 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.460829 (+1029807us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b5400144454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:57.460885 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.461014 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":312966,"prepare.queue_time_us":26172,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":8041,"raft.queue_time_us":422926,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":319106,"spinlock_wait_cycles":186874112,"thread_start_us":495883,"threads_started":2,"wal-append.queue_time_us":313038}
I20260709 13:02:57.362445  5861 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
W20260709 13:02:57.439368  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.434737  2315 rpcz_store.cc:269] 0709 13:02:54.915010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.943670 (+ 28660us) service_pool.cc:224] Handling call
0709 13:02:57.389593 (+2445923us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54000c49d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.389736 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":291334784}
I20260709 13:02:57.354833  6003 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.388669  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:57.459404  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2448) took 1580 ms (client timeout 1574 ms). Trace:
W20260709 13:02:57.468003  1661 rpcz_store.cc:269] 0709 13:02:55.879380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.899183 (+ 19803us) service_pool.cc:224] Handling call
0709 13:02:57.459206 (+1560023us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b540013d8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.459330 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2226864896}
I20260709 13:02:57.469484  1661 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:57.470359  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2419) took 2234 ms. Trace:
I20260709 13:02:57.381340  5980 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.471592  1659 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:57.470773  1661 rpcz_store.cc:276] 0709 13:02:55.235577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.469092 (+2233515us) service_pool.cc:224] Handling call
0709 13:02:57.469346 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.470155 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.470313 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.475622  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:57.476804  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88205.4us
I20260709 13:02:57.477655  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 592) took 2501 ms. Trace:
I20260709 13:02:57.478048  1593 rpcz_store.cc:276] 0709 13:02:54.976578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.807337 (+830759us) service_pool.cc:224] Handling call
0709 13:02:55.839780 (+ 32443us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:02:55.870744 (+ 30964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:55.871010 (+   266us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:02:55.911412 (+ 40402us) log.cc:844] Serialized 616 byte log entry
0709 13:02:56.151586 (+240174us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:56.441888 (+290302us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.442222 (+   334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:56.442326 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:57.313533 (+871207us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:57.467582 (+154049us) raft_consensus.cc:1685] finished
0709 13:02:57.477399 (+  9817us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.477586 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1234758,"raft.queue_time_us":121948,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":1246144,"spinlock_wait_cycles":9728768,"thread_start_us":190819,"threads_started":1,"wal-append.queue_time_us":403074}
I20260709 13:02:57.479952  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.473254  1740 raft_consensus.cc:2410] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:02:57.411126  5584 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:57.435384  5373 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:57.486191  5373 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.487694  5764 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.491263  1665 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:57.491971  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2425) took 2254 ms. Trace:
I20260709 13:02:57.492273  1665 rpcz_store.cc:276] 0709 13:02:55.246192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.490747 (+2244555us) service_pool.cc:224] Handling call
0709 13:02:57.491075 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.491714 (+   639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.491912 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.481451  5584 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.494870  5584 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:57.459962  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.503718  1742 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 13:02:57.504114  1742 pending_rounds.cc:85] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
I20260709 13:02:57.504230  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2420) took 2242 ms. Trace:
I20260709 13:02:57.504253  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2583) took 2653 ms. Trace:
W20260709 13:02:57.502619  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 640) took 2758 ms (client timeout 1766 ms). Trace:
W20260709 13:02:57.505115  1736 rpcz_store.cc:269] 0709 13:02:54.744672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.772145 (+ 27473us) service_pool.cc:224] Handling call
0709 13:02:57.502325 (+2730180us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54003fa7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.502517 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":825291648}
I20260709 13:02:57.505645  1661 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 13:02:57.506316  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2423) took 2269 ms. Trace:
I20260709 13:02:57.504921  1588 rpcz_store.cc:276] 0709 13:02:54.850466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.286046 (+435580us) service_pool.cc:224] Handling call
0709 13:02:55.286496 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:55.469992 (+183496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.503987 (+2033995us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540025ce54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:57.504189 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385881856}
I20260709 13:02:57.506791  1661 rpcz_store.cc:276] 0709 13:02:55.236938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.476157 (+2239219us) service_pool.cc:224] Handling call
0709 13:02:57.476453 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.506063 (+ 29610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.506260 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.504739  1659 rpcz_store.cc:276] 0709 13:02:55.235950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.471233 (+2235283us) service_pool.cc:224] Handling call
0709 13:02:57.471462 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.475561 (+  4099us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.477999 (+  2438us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.510015  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 648) took 2196 ms. Trace:
I20260709 13:02:57.504339  1742 pending_rounds.cc:107] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 13:02:57.510725  1742 raft_consensus.cc:2889] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:57.510422  1665 rpcz_store.cc:276] 0709 13:02:55.313646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.493028 (+2179382us) service_pool.cc:224] Handling call
0709 13:02:57.493324 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.498137 (+  4813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.498176 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.509689 (+ 11513us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:02:57.509763 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.509830 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.509959 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.512107  1659 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:57.512605  1665 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 13:02:57.512763  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2430) took 1930 ms. Trace:
I20260709 13:02:57.513192  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2432) took 1868 ms. Trace:
I20260709 13:02:57.513170  1659 rpcz_store.cc:276] 0709 13:02:55.582502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.511578 (+1929076us) service_pool.cc:224] Handling call
0709 13:02:57.511929 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.512514 (+   585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.512707 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.513769  1665 rpcz_store.cc:276] 0709 13:02:55.645020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.511995 (+1866975us) service_pool.cc:224] Handling call
0709 13:02:57.512278 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.512965 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.513141 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.475036  5872 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.520967  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.521988  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:57.522361  5906 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.547s	user 0.000s	sys 0.006s
I20260709 13:02:57.521349  1665 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.528481  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2435) took 1827 ms. Trace:
I20260709 13:02:57.528959  1665 rpcz_store.cc:276] 0709 13:02:55.701652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.520828 (+1819176us) service_pool.cc:224] Handling call
0709 13:02:57.521160 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.528176 (+  7016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.528416 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.519203  5898 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:57.534756  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 646) took 2289 ms (client timeout 2647 ms). Trace:
W20260709 13:02:57.535653  1666 rpcz_store.cc:269] 0709 13:02:55.245308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.504332 (+259024us) service_pool.cc:224] Handling call
0709 13:02:57.534545 (+2030213us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.534689 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2795935488}
I20260709 13:02:57.536254  1736 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:57.530196  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 649) took 2039 ms. Trace:
I20260709 13:02:57.530745  1665 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:57.510635  5878 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:02:57.539616  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2437) took 1799 ms. Trace:
I20260709 13:02:57.519713  1661 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.537557  1662 rpcz_store.cc:276] 0709 13:02:55.491344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.511293 (+2019949us) service_pool.cc:224] Handling call
0709 13:02:57.529519 (+ 18226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.529868 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.529898 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.529943 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:57.529974 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.530029 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.530143 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.538770  1659 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.541451  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2434) took 1844 ms. Trace:
I20260709 13:02:57.540127  1665 rpcz_store.cc:276] 0709 13:02:55.740402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.529995 (+1789593us) service_pool.cc:224] Handling call
0709 13:02:57.530262 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.539272 (+  9010us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.539537 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.542716  1666 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:57.543553  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2446) took 1696 ms. Trace:
I20260709 13:02:57.542074  1659 rpcz_store.cc:276] 0709 13:02:55.696630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.520729 (+1824099us) service_pool.cc:224] Handling call
0709 13:02:57.538584 (+ 17855us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.541197 (+  2613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.541395 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.542256  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.544010  1666 rpcz_store.cc:276] 0709 13:02:55.846720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.536879 (+1690159us) service_pool.cc:224] Handling call
0709 13:02:57.542519 (+  5640us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.543195 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.543468 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.544759  1740 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:57.543771  1665 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:02:57.548768  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2429) took 2046 ms. Trace:
I20260709 13:02:57.549490  1662 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 6. Current term is 7. Ops: []
I20260709 13:02:57.549865  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.549752  1661 rpcz_store.cc:276] 0709 13:02:55.502456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.514443 (+2011987us) service_pool.cc:224] Handling call
0709 13:02:57.514758 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.540369 (+ 25611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.548679 (+  8310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:57.550619  1736 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:57.539656  5878 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:57.552434  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2449) took 1660 ms. Trace:
I20260709 13:02:57.518985  5972 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.528788  5872 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.560552  1662 rpcz_store.cc:276] 0709 13:02:55.892609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.541225 (+1648616us) service_pool.cc:224] Handling call
0709 13:02:57.549301 (+  8076us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.552140 (+  2839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.552366 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:57.563752  5966 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.803s	user 0.001s	sys 0.002s
I20260709 13:02:57.561604  5878 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:57.571336  5584 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.573599  5373 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.575596  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.575768  5373 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.578023  5667 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.578210  5979 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.579954  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 750) took 1555 ms. Trace:
I20260709 13:02:57.580343  2311 rpcz_store.cc:276] 0709 13:02:56.024138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.137304 (+113166us) service_pool.cc:224] Handling call
0709 13:02:56.175937 (+ 38633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.301829 (+125892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.579667 (+1277838us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:57.579882 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279614720}
I20260709 13:02:57.581676  1733 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:02:57.529629  6002 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:02:57.588167  5667 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.589242  5667 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:57.589483  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 748) took 1958 ms (client timeout 2080 ms). Trace:
I20260709 13:02:57.588222  5969 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:02:57.591444  2318 rpcz_store.cc:269] 0709 13:02:55.630993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.914775 (+283782us) service_pool.cc:224] Handling call
0709 13:02:57.589302 (+1674527us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b54003ef154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.589424 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2067319168}
I20260709 13:02:57.590440  1662 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 13:02:57.602674  5667 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.603720  5480 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.605979  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.580440  5979 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.606829  1740 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 7.
I20260709 13:02:57.620705  1736 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:02:57.621594  5972 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.622563  5480 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.618007  5584 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.629926  1740 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 13:02:57.633361  1740 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:02:57.633672  1736 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: [9.7-11.8]
I20260709 13:02:57.635366  1740 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:02:57.651065  5872 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.630026  5878 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.654301  5878 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.655969  5917 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:57.659886  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 293989us
I20260709 13:02:57.659770  5968 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:57.660611  5968 consensus_queue.cc:1243] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 9.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:02:57.661695  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 294853us
I20260709 13:02:57.662834  5882 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:57.663383  5882 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:57.663726  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.664060  1593 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.664283  5882 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.665371  5977 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:02:57.665920  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.666913  5977 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:57.667891  1587 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:57.669905  1585 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:57.671746  1587 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:02:57.673535  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.674209  1585 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.675034  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.664947  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.678902  1587 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 13:02:57.682365  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:02:57.683004  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.683550  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.684121  1739 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:57.679809  5939 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.686141  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:57.692621  5936 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.958s	user 0.005s	sys 0.000s
I20260709 13:02:57.690907  5917 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:57.694031  5917 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.694983  5792 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:57.695415  5792 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.696276  5917 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:57.696226  5792 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.697315  5792 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.704867  5970 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:57.705374  5970 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:57.706208  5970 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.707372  5970 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:57.716483  5865 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:57.719521  5959 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.572s	user 0.000s	sys 0.003s
I20260709 13:02:57.721630  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1867) took 2731 ms. Trace:
I20260709 13:02:57.722048  1742 rpcz_store.cc:276] 0709 13:02:54.990860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.906449 (+915589us) service_pool.cc:224] Handling call
0709 13:02:57.503298 (+1596849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.511432 (+  8134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.721359 (+209927us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54003fa7d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.721559 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3671997824}
I20260709 13:02:57.730461  5969 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:02:57.730690  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48737.2us
I20260709 13:02:57.731648  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:02:57.732916  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.737849  5756 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:02:57.738786  5756 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:57.739764  5756 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.743832  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 861) took 2752 ms. Trace:
I20260709 13:02:57.744274  2316 rpcz_store.cc:276] 0709 13:02:54.991026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:54.991209 (+   183us) service_pool.cc:224] Handling call
0709 13:02:56.807079 (+1815870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:56.839323 (+ 32244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.743618 (+904295us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b5400570650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.743763 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4175635200}
I20260709 13:02:57.747308  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.748116  1659 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:57.748643  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.749819  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.749897  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.750455  1736 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:02:57.750660  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2444) took 1926 ms (client timeout 2097 ms). Trace:
W20260709 13:02:57.751106  1664 rpcz_store.cc:269] 0709 13:02:55.824320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.852541 (+ 28221us) service_pool.cc:224] Handling call
0709 13:02:57.750476 (+1897935us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.750599 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1048703232}
I20260709 13:02:57.751714  1736 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 13:02:57.753034  1736 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:02:57.754456  1735 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:57.755475  1735 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.755002  1737 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:57.756330  1735 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.756887  1737 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:57.757346  1735 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:57.757910  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.759313  1737 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:02:57.760263  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.760469  5968 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:02:57.754510  1733 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:02:57.761948  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.751717  1666 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:02:57.737174  5809 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:57.763339  1736 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:57.763530  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:57.764010  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.764416  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.764279  1736 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:57.767256  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.767817  5809 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:02:57.768350  1592 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:02:57.768949  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 746) took 2153 ms. Trace:
I20260709 13:02:57.769445  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.826992  1590 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:57.827489  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.828326  1592 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:57.769762  5968 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:57.770565  5898 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.771021  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.772410  1587 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 13:02:57.773066  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.773818  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.775718  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:57.775813  5802 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:57.776608  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:57.847951  2318 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:57.777112  5826 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.777839  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.849635  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.778256  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 604358us
I20260709 13:02:57.778653  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:57.779103  5875 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:02:57.778280  5633 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.780205  5922 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:57.784219  1585 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:02:57.785882  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.786979  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:57.787766  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2431) took 2166 ms (client timeout 2171 ms). Trace:
W20260709 13:02:57.878096  1668 rpcz_store.cc:269] 0709 13:02:55.621834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:55.653073 (+ 31239us) service_pool.cc:224] Handling call
0709 13:02:57.787554 (+2134481us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.787705 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3505721600}
I20260709 13:02:57.788777  1663 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:02:57.792567  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:57.793936  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2551) took 5712 ms. Trace:
W20260709 13:02:57.772365  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.877321  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.794762  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:57.797583  1735 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 13:02:57.798022  5602 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:02:57.755637  1742 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:02:57.799299  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.801397  5939 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:57.802321  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.802673  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69856.1us
I20260709 13:02:57.811163  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:57.813898  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:57.814179  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.890947  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98460.6us
I20260709 13:02:57.815868  1737 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:57.891657  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.801774  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.820266  1736 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:02:57.832477  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.833407  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.826591  2313 rpcz_store.cc:276] 0709 13:02:55.615299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:56.016332 (+401033us) service_pool.cc:224] Handling call
0709 13:02:56.016702 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.016942 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:02:56.157121 (+140179us) log.cc:844] Serialized 14 byte log entry
0709 13:02:56.506919 (+349798us) log.cc:844] Serialized 14 byte log entry
0709 13:02:57.292255 (+785336us) spinlock_profiling.cc:243] Waited 4.55 ms on lock 0x7b180038ce60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:02:57.556516 (+264261us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:57.768433 (+211917us) spinlock_profiling.cc:243] Waited 2.48 ms on lock 0x7b180038ce60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:02:57.768593 (+   160us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:57.768646 (+    53us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:02:57.768698 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.768772 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:57.768890 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1245427,"prepare.queue_time_us":253850,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":4395,"raft.queue_time_us":1266819,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":7967,"spinlock_wait_cycles":45870592,"thread_start_us":488827,"threads_started":2,"wal-append.queue_time_us":385765}
I20260709 13:02:57.893460  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.894088  1590 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:02:57.896163  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.897035  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.846463  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.897627  1589 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:02:57.897830  1586 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:57.899025  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.899197  1584 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:02:57.900269  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:57.880641  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.880983  1735 pending_rounds.cc:85] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 3
W20260709 13:02:57.901084  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.901221  1735 pending_rounds.cc:107] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 13:02:57.901618  1735 raft_consensus.cc:2889] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:57.879819  1741 rpcz_store.cc:276] 0709 13:02:52.081662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:53.518965 (+1437303us) service_pool.cc:224] Handling call
0709 13:02:56.866695 (+3347730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:56.899938 (+ 33243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.793400 (+893462us) spinlock_profiling.cc:243] Waited 3.35 s on lock 0x7b5400435650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:57.793844 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7698513280}
W20260709 13:02:57.890072  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90868.4us
I20260709 13:02:57.838729  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.900178  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.884925  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.905323  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.905623  1592 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:02:57.905978  1665 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.906706  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:57.907533  1592 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.907974  1664 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:02:57.908571  2318 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:57.897759  1734 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 7.
W20260709 13:02:57.909723  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108001us
I20260709 13:02:57.911370  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.853811  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:02:57.914789  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47460.9us
I20260709 13:02:57.931116  1588 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:57.930823  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:57.930632  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.932461  1666 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:57.914760  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.934042  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:02:57.934684  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:57.934968  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:57.865303  5838 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.951850  1668 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 13:02:57.949440  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.941999  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:57.935848  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:02:57.854557  5454 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.923630  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:58.000646  1665 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:02:57.894486  5574 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:02:57.982947  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52221us
W20260709 13:02:57.957331  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46040us
I20260709 13:02:57.930634  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:57.948927  1664 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:58.006107  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.854646  5875 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:58.003577  5830 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:58.002319  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:02:57.971060  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:57.931382  5922 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 13:02:57.962677  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:02:58.007920  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 77441.1us
I20260709 13:02:58.007390  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:02:58.008387  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:58.142333  1586 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:58.009378  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.851629  5968 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.916690  5872 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:58.013937  1664 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:02:57.911474  5939 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:58.151705  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 255551us
I20260709 13:02:58.014511  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:58.015738  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:02:57.902778  5792 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:58.155355  5792 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:58.155872  5792 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:57.995285  5764 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.134s
I20260709 13:02:58.026189  5660 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:57.887316  5826 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.867558  5802 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.916158  5984 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:02:58.018606  1739 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:58.019281  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:58.021150  1592 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:02:58.191006  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:58.191896  1592 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:02:58.193020  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:57.862739  5865 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.141s
W20260709 13:02:58.024833  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 177820us
I20260709 13:02:58.031266  1588 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:02:57.933579  5970 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:57.995209  5861 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:02:58.025684  5883 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.134s
I20260709 13:02:57.985487  5602 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:58.055598  1736 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:02:58.034154  5882 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.151s
W20260709 13:02:58.130707  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.035480  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:58.131740  1665 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 13:02:58.131640  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 280335us
I20260709 13:02:58.232173  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:57.936517  5825 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.096s
W20260709 13:02:58.139312  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:58.139878  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 133892us
I20260709 13:02:58.238973  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:57.894966  5633 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.018426  5973 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:57.936216  5881 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.096s
I20260709 13:02:58.144320  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:58.184038  5984 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:58.153003  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:57.995792  5838 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.155359  1667 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:58.197456  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:02:58.271420  1588 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:58.263000  1590 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:57.886174  5898 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.200268  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:02:58.286326  1592 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:58.291332  1592 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 13:02:58.291962  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.204276  1666 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:02:58.138312  5830 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:02:58.010775  5454 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:57.983506  5969 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:02:58.151424  6003 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:02:58.219831  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:57.914453  5730 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:58.244484  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 209145us
I20260709 13:02:58.148368  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:58.265211  1586 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:02:58.340919  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 192664us
I20260709 13:02:58.264415  5970 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:58.344385  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:58.037042  5875 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:58.243440  4967 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
W20260709 13:02:58.320354  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 100633us
I20260709 13:02:58.341996  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:58.368449  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:58.252759  5989 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 13:02:58.213469  5861 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:58.296520  1592 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:58.279261  5973 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:02:58.348975  1588 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:02:58.392820  1588 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 13:02:58.395056  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.387254  5875 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:02:58.338985  5969 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:58.357928  5917 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:02:58.365828  5865 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:02:58.389389  5861 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:58.269685  5602 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:02:58.185081  5660 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.384065  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:58.386328  1592 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:02:58.178773  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:02:58.341385  5895 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:02:58.311094  5830 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:58.387769  6018 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:02:58.383091  5970 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:02:58.397848  6003 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:02:58.463835  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 231755us
W20260709 13:02:58.512226  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 273378us
W20260709 13:02:58.476446  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.401408  5882 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.409132  5944 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:02:58.514653  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:02:58.486486  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 102535us
I20260709 13:02:58.317132  5898 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.351456  1590 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:58.371747  5730 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.415784  5865 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:02:58.387612  5984 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:58.525820  6012 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 13:02:58.530001  5969 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.531175  5838 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:58.531524  5968 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:58.532641  5633 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.484025  5895 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:58.523618  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:02:58.542076  5970 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.511843  6017 raft_consensus.cc:2764] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 13:02:58.502138  5922 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:58.300071  5979 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.509800  6018 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:02:58.520515  5917 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:58.533440  5939 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.535167  6003 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:58.578248  5895 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:02:58.567090  5454 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:58.553711  5883 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:58.552004  5865 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:58.550264  5944 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:58.533911  5882 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.549753  5602 raft_consensus.cc:2764] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:02:58.574172  5802 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:58.591591  5826 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.583583  5633 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:02:58.577378  5981 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 13:02:58.592166  5917 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.616925  5730 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.560359  5973 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.610874  5883 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:58.619038  6022 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:02:58.620672  5898 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:58.622417  5944 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.625425  5882 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.625916  6003 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.633689  6022 consensus_queue.cc:1243] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260709 13:02:58.640300  6011 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.815s	user 0.000s	sys 0.007s
I20260709 13:02:58.634044  5883 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:58.659355  5917 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:58.649164  6008 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.497s	user 0.000s	sys 0.015s
I20260709 13:02:58.666360  5830 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:02:58.682991  6009 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.861s	user 0.003s	sys 0.015s
I20260709 13:02:58.669296  5830 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:58.698230  6013 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.874s	user 0.005s	sys 0.010s
I20260709 13:02:58.701288  5660 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.589996  5979 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.730067  5922 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:02:58.743947  5872 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.585s	user 0.005s	sys 0.006s
I20260709 13:02:58.745000  5872 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:02:58.747712  5865 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.741029  5861 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.701663  6018 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.762850  6018 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:02:58.761816  5979 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.732548  5922 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.760540  5984 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:02:58.771438  5865 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:02:58.842658  1593 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.010s	sys 0.010s
I20260709 13:02:58.952283  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 756) took 1172 ms. Trace:
W20260709 13:02:58.843849  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 475908us
W20260709 13:02:58.844841  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 321164us
I20260709 13:02:58.953052  4967 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:02:58.954352  4967 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.851883  1590 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 13:02:58.955186  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:02:58.855535  1737 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.853s	user 0.007s	sys 0.009s
W20260709 13:02:58.855954  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.551998  1667 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:02:58.857525  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1889) took 1424 ms. Trace:
W20260709 13:02:58.586371  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 433442us
I20260709 13:02:58.784317  5883 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:58.782675  5895 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.867923  1591 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:02:58.979889  1591 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:02:58.980585  5895 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.981881  1591 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 13:02:58.930896  1592 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:02:58.964358  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2613) took 1175 ms. Trace:
I20260709 13:02:58.932821  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2596) took 1185 ms. Trace:
I20260709 13:02:58.936841  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 645) took 3907 ms. Trace:
W20260709 13:02:58.944996  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:58.954099  5574 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 13:02:58.854897  1584 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:02:58.846170  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1861) took 1739 ms. Trace:
I20260709 13:02:58.982841  1590 rpcz_store.cc:276] 0709 13:02:57.781281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.539959 (+758678us) service_pool.cc:224] Handling call
0709 13:02:58.850646 (+310687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.956046 (+105400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.956524 (+   478us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:58.953094  2315 rpcz_store.cc:276] 0709 13:02:57.779341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.786199 (+  6858us) service_pool.cc:224] Handling call
0709 13:02:57.786539 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.871256 (+1084717us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:58.871335 (+    79us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:02:58.871393 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:02:58.871477 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.951620 (+ 80143us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:58.951688 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:58.951950 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28615,"spinlock_wait_cycles":2491543808}
I20260709 13:02:58.941104  5861 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:58.966630  5898 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:02:58.967614  5979 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:02:58.968885  6034 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:02:58.969261  5872 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:02:58.968916  1737 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:02:58.870961  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 692131us
I20260709 13:02:58.981925  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2468) took 1034 ms. Trace:
I20260709 13:02:58.983573  1591 pending_rounds.cc:85] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:02:58.983361  1663 rpcz_store.cc:276] 0709 13:02:55.029159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.448399 (+2419240us) service_pool.cc:224] Handling call
0709 13:02:57.788587 (+340188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:57.887804 (+ 99217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.934820 (+1047016us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x7b54001ddba0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0a02 00007fcdedfdcdd6 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:02:58.936260 (+  1440us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b540013ecd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:02:58.936574 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":799951872}
I20260709 13:02:58.983004  1739 rpcz_store.cc:276] 0709 13:02:57.747156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.751807 (+  4651us) service_pool.cc:224] Handling call
0709 13:02:57.752094 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.188462 (+436368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.932097 (+743635us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:02:58.932539 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":612440960}
I20260709 13:02:58.990469  1667 rpcz_store.cc:276] 0709 13:02:57.947144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.984565 (+ 37421us) service_pool.cc:224] Handling call
0709 13:02:58.981676 (+997111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:58.994954  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2616) took 1207 ms. Trace:
I20260709 13:02:58.983119  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:02:59.001315  1584 rpcz_store.cc:276] 0709 13:02:57.783105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.853812 (+1070707us) service_pool.cc:224] Handling call
0709 13:02:58.854364 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.988310 (+133946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:58.990448 (+  2138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:58.990967  6034 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.023942  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2607) took 1113 ms. Trace:
I20260709 13:02:59.024964  1737 rpcz_store.cc:276] 0709 13:02:57.893765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.898674 (+  4909us) service_pool.cc:224] Handling call
0709 13:02:59.004025 (+1105351us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:02:59.032086  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53169.4us
I20260709 13:02:59.010862  6034 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:02:59.036069  5602 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:02:59.041736  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:59.042884  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.005s	sys 0.003s
W20260709 13:02:59.043426  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.005s	sys 0.004s
I20260709 13:02:59.046527  5979 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.049641  5898 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.051064  5898 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:59.053925  4967 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:59.054807  5883 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:02:59.060750  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53222.2us
I20260709 13:02:59.062007  5865 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:02:59.063256  5865 consensus_queue.cc:1243] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:02:59.071593  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98036us
I20260709 13:02:59.013144  5872 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:59.076069  5872 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:58.985188  1661 rpcz_store.cc:276] 0709 13:02:57.106477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.555802 (+449325us) service_pool.cc:224] Handling call
0709 13:02:57.556222 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:57.559119 (+  2897us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.560688 (+  1569us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:57.584437 (+ 23749us) log.cc:844] Serialized 26 byte log entry
0709 13:02:57.817201 (+232764us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:02:57.817255 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.817353 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:58.648327 (+830974us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:58.844849 (+196522us) raft_consensus.cc:1685] finished
0709 13:02:58.844939 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:58.845724 (+   785us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":209,"raft.queue_time_us":3075,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":519,"spinlock_wait_cycles":4593664,"thread_start_us":210138,"threads_started":1,"wal-append.queue_time_us":416649}
I20260709 13:02:58.957579  1740 rpcz_store.cc:276] 0709 13:02:57.433141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:57.640644 (+207503us) service_pool.cc:224] Handling call
0709 13:02:57.641087 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:02:57.679410 (+ 38323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:57.679443 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:02:57.685918 (+  6475us) log.cc:844] Serialized 26 byte log entry
0709 13:02:57.825237 (+139319us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:02:57.825282 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:57.825355 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:02:58.639481 (+814126us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:02:58.856643 (+217162us) raft_consensus.cc:1685] finished
0709 13:02:58.856682 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:02:58.857148 (+   466us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35432,"raft.queue_time_us":35603,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8346,"spinlock_wait_cycles":24111872,"thread_start_us":128047,"threads_started":1,"wal-append.queue_time_us":127715}
W20260709 13:02:59.083068  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49745.1us
W20260709 13:02:59.066718  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.001s	sys 0.009s
W20260709 13:02:59.065439  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 551535us
W20260709 13:02:59.087935  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.002s	sys 0.010s
I20260709 13:02:59.037142  5602 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:59.130064  5660 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:02:59.082409  6033 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:02:59.174821  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 88216.3us
I20260709 13:02:59.177075  6033 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:02:59.199877  6033 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:02:59.200989  5602 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:59.160521  5574 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
W20260709 13:02:59.351186  6040 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 13:02:59.315860  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:02:59.320250  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:02:59.387159  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:02:59.234587  5979 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:02:59.407008  5883 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:02:59.405745  5898 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:59.411036  6018 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
W20260709 13:02:59.429309  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 113728us
I20260709 13:02:59.439678  2316 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:02:59.441609  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:59.441915  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.443128  2315 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:02:59.445227  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.413468  5883 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 13:02:59.450495  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.416136  5660 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 13:02:59.458444  2318 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:02:59.464928  5792 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.253s	user 0.002s	sys 0.022s
I20260709 13:02:59.470089  2313 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
W20260709 13:02:59.468904  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56113.6us
I20260709 13:02:59.476361  6018 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:02:59.473351  5792 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.615813  5792 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:02:59.726142  6034 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.690s	user 0.002s	sys 0.016s
I20260709 13:02:59.726856  6034 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:02:59.741097  5660 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:02:59.743253  5660 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:02:59.743933  5660 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:02:59.894081  6033 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.693s	user 0.007s	sys 0.009s
I20260709 13:02:59.894711  6033 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:59.900813  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 772958us
I20260709 13:02:59.903262  6034 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:02:59.902861  5792 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:02:59.905360  5792 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:02:59.905864  5792 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:59.907280  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:02:59.907080  5792 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.908686  5792 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:02:59.902184  6033 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:02:59.902756  5660 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.926254  5898 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:02:59.926873  5898 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:02:59.927888  5898 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:02:59.930992  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 481211us
I20260709 13:02:59.935940  5660 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:02:59.941184  1634 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.025s (SENT) [suppressed 22 similar messages]
I20260709 13:02:59.948380  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.955595  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:02:59.957232  1559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.599s	user 0.020s	sys 0.043s
W20260709 13:02:59.961861  1559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.604s	user 0.022s	sys 0.045s
I20260709 13:02:59.964249  2311 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:02:59.965320  2311 raft_consensus.cc:606] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:02:59.967408  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.970312  6034 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.971596  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.971365  6034 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.967701  2311 raft_consensus.cc:614] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:02:59.976966  6053 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:02:59.977826  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.007611  1591 pending_rounds.cc:107] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 13:02:59.980676  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:02:59.983801  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:59.983641  1591 raft_consensus.cc:2889] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:02:59.984794  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:59.984938  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:02:59.985484  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2619) took 1778 ms. Trace:
I20260709 13:02:59.985769  2318 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:02:59.985937  1591 rpcz_store.cc:276] 0709 13:02:58.207094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.980777 (+773683us) service_pool.cc:224] Handling call
0709 13:02:58.981170 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:59.985190 (+1004020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:02:59.985401 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:02:59.986863  6053 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:02:59.990427  2311 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:02:59.991580  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.650s	user 0.000s	sys 0.025s
I20260709 13:02:59.998932  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:00.003185  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.003952  5792 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:00.004565  5872 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.017584  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.676s	user 0.001s	sys 0.026s
I20260709 13:03:00.005342  5898 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.007192  6034 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.008198  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:00.010043  2315 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:00.011263  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:00.020057  2313 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:00.020285  2312 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:00.018860  5872 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.018806  5792 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.022187  6034 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:00.022804  6034 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.023470  5792 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.028419  5881 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:00.029268  5881 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.029623  5872 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.039019  6034 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.044659  5881 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.019243  5898 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.051322  5881 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:00.052088  5881 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.052719  5898 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.054841  2314 raft_consensus.cc:2410] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:03:00.048930  2313 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:00.052809  6053 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.026423  5792 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.057103  6053 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.057889  1586 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.696s	user 0.026s	sys 0.014s
I20260709 13:03:00.063169  1586 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:00.058271  5792 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.068085  1587 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:00.068553  5881 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.070122  1740 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:00.070125  5792 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.071370  1739 pending_rounds.cc:85] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:03:00.073390  1739 pending_rounds.cc:107] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 13:03:00.073716  1739 raft_consensus.cc:2889] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:00.075017  1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2654) took 2278 ms (client timeout 2226 ms). Trace:
W20260709 13:03:00.075995  1586 rpcz_store.cc:269] 0709 13:02:57.796689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.144043 (+347354us) service_pool.cc:224] Handling call
0709 13:03:00.074873 (+1930830us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:00.077087  6054 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:00.058260  6053 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.087381  5872 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.087953  6053 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.088613  6055 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:00.088424  5872 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.089579  6055 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.091892  5872 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.092231  1589 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:00.092708  5872 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:00.087595  6034 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.081967  6054 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.093679  5792 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.100332  5872 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.103963  6034 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.104229  5792 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.103375  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2655) took 2304 ms. Trace:
I20260709 13:03:00.107502  5872 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.107605  6055 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.105422  1589 rpcz_store.cc:276] 0709 13:02:57.796951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.858041 (+1061090us) service_pool.cc:224] Handling call
0709 13:02:58.858662 (+   621us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:00.092816 (+1234154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.100798 (+  7982us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003b79d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:00.101668 (+   870us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2835565952}
I20260709 13:03:00.107621  5792 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.110939  5898 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.107640  6034 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.114207  5898 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.116575  1589 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [6.4-12.7]
I20260709 13:03:00.116890  5898 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.117744  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2621) took 1747 ms. Trace:
I20260709 13:03:00.118316  1589 rpcz_store.cc:276] 0709 13:02:58.369722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.112237 (+1742515us) service_pool.cc:224] Handling call
0709 13:03:00.116081 (+  3844us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:03:00.117069 (+   988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.117317 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:00.124049  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:00.128685  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:00.130337  6054 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.138152  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.139302  1740 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:00.138278  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:00.142838  2310 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 13:03:00.143352  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.149986  1740 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:00.152854  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.154126  1740 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:03:00.154639  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:00.141918  2313 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:00.144143  1584 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:00.147639  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:00.146795  1593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.194s	user 0.013s	sys 0.016s
I20260709 13:03:00.157019  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2622) took 1782 ms. Trace:
I20260709 13:03:00.157409  1584 rpcz_store.cc:276] 0709 13:02:58.374522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.140598 (+1766076us) service_pool.cc:224] Handling call
0709 13:03:00.141228 (+   630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:00.155938 (+ 14710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.156933 (+   995us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:00.158196  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:00.156402  1593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.204s	user 0.014s	sys 0.017s
W20260709 13:03:00.158481  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.811s	user 0.013s	sys 0.008s
I20260709 13:03:00.161026  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2598) took 2386 ms. Trace:
I20260709 13:03:00.159224  1738 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
W20260709 13:03:00.161903  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.814s	user 0.014s	sys 0.009s
I20260709 13:03:00.163641  6054 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.163682  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:00.164692  1738 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.164636  6054 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.165925  1590 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:03:00.167353  6054 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.169833  6057 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:00.170008  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.170957  1590 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:00.171438  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.172102  1733 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:00.172276  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.181682  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.182897  1590 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:00.185032  1590 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:00.178331  6053 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.185742  6058 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:00.186563  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.186403  6058 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:00.187111  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.188517  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.188983  5881 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:00.189826  1733 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:00.186185  6053 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.161835  1593 rpcz_store.cc:276] 0709 13:02:57.774471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.279556 (+505085us) service_pool.cc:224] Handling call
0709 13:02:58.339122 (+ 59566us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:02:58.950294 (+611172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:02:58.950376 (+    82us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:02:58.951918 (+  1542us) log.cc:844] Serialized 52 byte log entry
0709 13:03:00.159644 (+1207726us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:00.159684 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.159762 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:00.160834 (+  1072us) raft_consensus.cc:1685] finished
0709 13:03:00.160848 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:00.160974 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1893,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":25559,"thread_start_us":1817518,"threads_started":2,"wal-append.queue_time_us":368445}
I20260709 13:03:00.189052  6058 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.201534  1740 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:03:00.210319  1736 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 13:03:00.210553  1738 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:00.210345  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.193919  6054 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.216149  6054 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.216974  6053 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:00.207150  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3035658220000528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 13:03:00.217792  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1811658220000254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
I20260709 13:03:00.218694  6054 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.224076  1733 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:00.227191  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2615) took 2444 ms. Trace:
I20260709 13:03:00.227789  1587 rpcz_store.cc:276] 0709 13:02:57.782808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.851357 (+1068549us) service_pool.cc:224] Handling call
0709 13:03:00.067717 (+1216360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:00.068559 (+   842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.226867 (+158308us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000fa350. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:00.227094 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2795216128}
I20260709 13:03:00.229564  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:00.231027  6017 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:00.231556  6017 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:00.231925  6017 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.234323  6062 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.200769  6058 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.237648  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.239194  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:00.239750  1740 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.231938  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:00.240386  1593 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:00.264449  6066 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:00.265173  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.265558  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.266040  1590 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:00.265456  6066 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.267148  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.268370  1590 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:03:00.270993  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:00.273389  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60884.7us
I20260709 13:03:00.279321  1733 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:00.277383  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.278527  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:00.263129  6053 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.281839  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:00.273360  5660 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:00.237021  6062 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.288216  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:00.289486  5660 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.298481  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.299338  6053 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.304523  6054 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:00.305397  6054 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:00.306073  1668 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:00.307596  6063 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:00.309653  6062 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.309894  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:00.312053  6066 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.306032  6054 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:00.299151  5660 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:00.317395  1587 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:00.299815  1739 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:00.321002  6032 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:00.321575  6032 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.321982  6032 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:00.323282  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:00.289877  6058 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.328490  1590 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:00.304531  6053 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.334450  6048 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:00.304680  6067 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 13:03:00.337014  6048 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.337481  6048 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:00.338418  6067 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:00.338878  6067 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.339248  6067 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:00.339481  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.000s	sys 0.010s
W20260709 13:03:00.339962  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.928s	user 0.011s	sys 0.028s
I20260709 13:03:00.344728  1740 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:00.349427  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:00.353106  6069 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:00.354512  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.358229  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.359972  1661 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:00.363394  6058 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.366542  6050 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:00.346195  6068 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.308495  6063 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.373468  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:00.378232  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.379137  1661 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:03:00.372506  6050 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.380730  6050 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.384239  6068 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:00.395498  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:00.376817  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.409868  6070 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.410640  6070 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.411146  6068 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.413008  6070 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.413571  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:00.414795  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.397658  6033 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:00.416826  6017 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.420364  6017 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.421638  6072 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.422293  6072 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.424965  6072 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.430536  6071 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:00.434100  6073 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:03:00.436407  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106669us
I20260709 13:03:00.441017  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.455089  1661 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
W20260709 13:03:00.452256  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 77163.3us
I20260709 13:03:00.457104  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:00.457564  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:00.459453  6056 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:00.459656  5881 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:00.460117  5881 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:00.460161  6056 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:00.460727  5881 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:00.454175  6072 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:00.462621  6034 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:00.463442  6034 consensus_queue.cc:1243] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:00.463979  6034 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:00.464363  6034 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:00.439652  6062 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:00.461181  6057 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:00.471109  6071 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:00.505317  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:00.507705  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.508874  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.481899  6057 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:00.481042  6063 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.481571  6062 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:00.511071  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 228646us
I20260709 13:03:00.465384  6034 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.525482  1587 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:00.512703  6055 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:00.555555  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.501986  5898 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 13:03:00.502188  6058 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 13:03:00.523180  5792 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.517696  5872 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
W20260709 13:03:00.582679  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78503.7us
I20260709 13:03:00.547371  6055 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:00.499996  6056 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.579493  6033 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:00.597554  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:00.601130  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 90000.6us
I20260709 13:03:00.606395  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2610) took 2828 ms. Trace:
I20260709 13:03:00.606853  1588 rpcz_store.cc:276] 0709 13:02:57.777493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.393842 (+616349us) service_pool.cc:224] Handling call
0709 13:02:58.394193 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.394451 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:02:58.412030 (+ 17579us) log.cc:844] Serialized 14 byte log entry
0709 13:03:00.474057 (+2062027us) log.cc:844] Serialized 14 byte log entry
0709 13:03:00.599531 (+125474us) log.cc:844] Serialized 14 byte log entry
0709 13:03:00.600252 (+   721us) log.cc:844] Serialized 14 byte log entry
0709 13:03:00.605895 (+  5643us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:00.605983 (+    88us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:00.606054 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.606144 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:00.606310 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3741,"prepare.queue_time_us":246362,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":6516,"thread_start_us":2130478,"threads_started":4,"wal-append.queue_time_us":826468,"wal-append.run_cpu_time_us":2184,"wal-append.run_wall_time_us":1024525}
I20260709 13:03:00.609236  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.611085  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.612794  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.613721  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:00.614995  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.615635  1588 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:00.616636  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:00.617324  1588 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:00.593657  6055 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:00.553469  6050 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:00.603606  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 174703us
I20260709 13:03:00.597190  6063 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:00.602557  6072 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:00.704216  6063 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:00.717639  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 120258us
W20260709 13:03:00.726184  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 139615us
I20260709 13:03:00.709923  6022 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:00.686266  5792 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.725219  6034 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:00.760954  6075 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 13:03:00.760744  6068 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:00.712078  6071 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.763638  6050 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.765424  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.766258  1591 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:03:00.767879  5792 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:00.746563  6054 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.198519  6054 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:00.748062  6072 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.754427  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.201236  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.756250  6087 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.757928  6063 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.202940  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.203476  6063 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:00.759294  6069 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:00.685714  6057 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.723398  6062 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.773852  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:00.758957  5660 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.744664  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:01.307161  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 655) took 1094 ms. Trace:
I20260709 13:03:01.307572  1742 rpcz_store.cc:276] 0709 13:03:00.212923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.722570 (+509647us) service_pool.cc:224] Handling call
0709 13:03:01.307069 (+584499us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:00.792032  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:00.753831  6033 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:00.796733  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86412.9us
W20260709 13:03:00.797873  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65065.7us
I20260709 13:03:00.804912  6091 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:00.743986  6018 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:00.734244  6022 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:00.814174  6055 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.816291  5981 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:01.313140  5981 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:01.314754  6055 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:01.314302  5981 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:00.825839  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:00.827901  1588 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:03:00.838967  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 130151us
I20260709 13:03:00.840265  6070 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:00.857254  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:00.858168  1659 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:00.839704  6067 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:00.869688  6095 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 13:03:00.869856  6084 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:03:01.318877  1662 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:01.318698  6093 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:01.319178  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:00.870694  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.320533  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.320904  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 660) took 1093 ms. Trace:
I20260709 13:03:01.320962  6069 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:01.321609  6069 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:01.321645  6070 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.315485  6022 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:01.322738  6069 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.3, Last appended by leader: 3, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.318740  5981 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:01.323664  6068 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:01.324119  6069 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:00.878381  6048 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.540s	user 0.008s	sys 0.006s
I20260709 13:03:01.327209  6101 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:01.327569  6048 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.328902  6101 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:01.006592  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.330003  6048 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.330483  6067 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.331188  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.331795  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.332558  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.009600  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.333662  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.333966  1663 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:01.018288  6097 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.334646  6072 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.335486  6072 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:01.335162  6097 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.311822  6018 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:01.336591  6018 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:01.337106  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.038437  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.326357  1587 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:01.078838  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.080566  6017 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.080940  6102 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:01.107501  6071 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:01.108250  6109 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.111779  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:01.171000  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.172802  6060 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:03:01.175336  6032 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.853s	user 0.010s	sys 0.008s
W20260709 13:03:01.179917  5881 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.718s	user 0.013s	sys 0.003s
W20260709 13:03:01.193256  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 469341us
I20260709 13:03:01.203030  6087 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:00.769304  6094 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:01.358176  6060 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:01.362419  6094 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:01.362844  6094 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:01.363751  6094 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.204159  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.365888  6048 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:01.366044  6017 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.205595  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.366549  6048 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:01.367449  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 162029us
I20260709 13:03:01.367627  6048 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.368319  6017 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.205215  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:01.212050  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 2.05087e+06us
I20260709 13:03:01.222996  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.252971  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:01.277577  6112 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.268s	user 0.005s	sys 0.000s
W20260709 13:03:01.278832  6108 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.270s	user 0.003s	sys 0.001s
I20260709 13:03:01.283167  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2611) took 3502 ms. Trace:
W20260709 13:03:01.283310  6034 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.003s
I20260709 13:03:01.346189  6091 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.357438  6102 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.389889  6091 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.391302  6109 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.391971  6102 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.392369  6091 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:01.309109  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 517217us
I20260709 13:03:01.370435  6032 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.393402  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.393705  6109 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:00.885689  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2620) took 2678 ms. Trace:
I20260709 13:03:01.054560  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.413125  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.413589  1660 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:01.412684  1586 rpcz_store.cc:276] 0709 13:02:58.207444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.078233 (+1870789us) service_pool.cc:224] Handling call
0709 13:03:00.078930 (+   697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:00.114130 (+ 35200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.114176 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:00.114235 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:00.114266 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.885400 (+771134us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b5400144454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:00.885449 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:00.885620 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75547008}
I20260709 13:03:01.339820  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.415566  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.416128  1588 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:01.416257  1586 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:01.414295  5792 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.349660  6097 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.350176  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:01.359385  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.417884  5792 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:01.419006  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69024.4us
I20260709 13:03:01.419241  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.419543  1587 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:01.420105  5792 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.420418  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.420998  1587 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 13:03:01.421307  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.366600  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.421521  6066 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:01.422081  1665 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:01.423183  6105 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:03:01.377123  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54847.6us
W20260709 13:03:01.296046  6092 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.493s	user 0.003s	sys 0.004s
I20260709 13:03:01.346926  6072 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:01.375665  1592 rpcz_store.cc:276] 0709 13:02:57.780715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.484501 (+703786us) service_pool.cc:224] Handling call
0709 13:02:58.484929 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:02:58.994748 (+509819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.282829 (+2288081us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b5400431054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:01.283092 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":980922112}
I20260709 13:03:01.381702  6071 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.382126  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.382704  1591 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:01.371694  5881 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.383098  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.434151  1589 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:01.436839  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.383433  6070 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.444581  1590 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:01.327129  6104 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260709 13:03:01.384265  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.384806  6087 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:01.362457  6060 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.385769  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.394542  6034 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:01.365435  6094 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:01.394755  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:01.458004  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:01.458379  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 63718.7us
I20260709 13:03:01.459120  2311 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:01.395318  6032 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.309967  5660 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.378542  6055 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:01.340134  1733 rpcz_store.cc:276] 0709 13:03:00.227455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.289773 (+ 62318us) service_pool.cc:224] Handling call
0709 13:03:00.316032 (+ 26259us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:00.346765 (+ 30733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.346814 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:00.386089 (+ 39275us) log.cc:844] Serialized 52 byte log entry
0709 13:03:00.582434 (+196345us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:00.598979 (+ 16545us) log.cc:844] Serialized 14 byte log entry
0709 13:03:00.599983 (+  1004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:00.600106 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.320633 (+720527us) raft_consensus.cc:1685] finished
0709 13:03:01.320654 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.320822 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13956,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2377,"raft.queue_time_us":15388,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":27136,"spinlock_wait_cycles":2304,"thread_start_us":156208,"threads_started":1,"wal-append.queue_time_us":365833}
I20260709 13:03:01.445377  5881 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.398986  6018 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.400748  6017 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.401806  6080 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.402019  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.486137  2314 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:01.402053  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:01.402418  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:01.487100  1585 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:01.403093  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 197984us
I20260709 13:03:01.404204  6063 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.405387  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1898) took 1335 ms. Trace:
I20260709 13:03:01.487906  1737 rpcz_store.cc:276] 0709 13:03:00.070078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.187477 (+1117399us) service_pool.cc:224] Handling call
0709 13:03:01.188213 (+   736us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:01.188999 (+   786us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:01.189048 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:01.194869 (+  5821us) log.cc:844] Serialized 26 byte log entry
0709 13:03:01.265363 (+ 70494us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:01.265406 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.265474 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.405128 (+139654us) raft_consensus.cc:1685] finished
0709 13:03:01.405151 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.405312 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4927,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"thread_start_us":69904,"threads_started":1,"wal-append.queue_time_us":115623}
I20260709 13:03:01.409860  6057 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:01.410432  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.392370  1491 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 5, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 5 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:01.411296  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.411509  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.414644  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.312325  6062 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:01.475090  5792 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:01.425045  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:01.425737  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.495517  1706 connection.cc:256] RPC call timeout handler was delayed by -1.9745146899999781s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
I20260709 13:03:01.425101  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.343649  1662 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:01.423907  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:01.432458  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.439833  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:01.440601  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:01.441179  1591 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:01.441473  6067 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:01.441787  1587 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:01.442520  2315 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 13:03:01.442420  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.499815  2315 pending_rounds.cc:85] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 8
I20260709 13:03:01.443377  1589 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:01.500083  2315 pending_rounds.cc:107] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 13:03:01.500377  2315 raft_consensus.cc:2889] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:01.443316  2310 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:01.479429  6032 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.422083  6066 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:01.471501  1634 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.562s (SENT)
I20260709 13:03:01.505806  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:01.506561  1634 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.670s (SENT)
W20260709 13:03:01.506613  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73936.1us
W20260709 13:03:01.396747  6100 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.586s	user 0.006s	sys 0.000s
I20260709 13:03:01.453656  6102 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:01.478484  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2657) took 1315 ms. Trace:
I20260709 13:03:01.456029  6034 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.511750  1584 rpcz_store.cc:276] 0709 13:03:00.163649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.164181 (+   532us) service_pool.cc:224] Handling call
0709 13:03:00.164631 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:00.194133 (+ 29502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.194157 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:00.210515 (+ 16358us) log.cc:844] Serialized 26 byte log entry
0709 13:03:01.328446 (+1117931us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b180047d100. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:01.459960 (+131514us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:01.460018 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.460096 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.474320 (+ 14224us) raft_consensus.cc:1685] finished
0709 13:03:01.474351 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.478389 (+  4038us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30247,"raft.queue_time_us":33942,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":7107,"spinlock_wait_cycles":7747328,"thread_start_us":152935,"threads_started":2,"wal-append.queue_time_us":1259661}
I20260709 13:03:01.513230  6034 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:01.513479  5898 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:01.514134  5898 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.514441  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.515770  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.516042  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.516746  1661 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:01.516891  5898 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.473198  6060 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:01.564073  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.360651  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 248983us
I20260709 13:03:01.446293  6022 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.496112  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.496891  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.572952  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 607) took 1362 ms. Trace:
I20260709 13:03:01.501353  1659 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:01.443622  6071 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.498947  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:01.501585  2318 raft_consensus.cc:606] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:01.501087  2310 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:01.504683  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80879.1us
I20260709 13:03:01.578497  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.505594  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 606) took 1295 ms. Trace:
W20260709 13:03:01.510927  1706 connection.cc:256] RPC call timeout handler was delayed by -1.701814689999992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260709 13:03:01.579348  1706 connection.cc:256] RPC call timeout handler was delayed by -1.580314689999966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260709 13:03:01.579806  1706 connection.cc:256] RPC call timeout handler was delayed by -1.548814689999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.219s for a response
I20260709 13:03:01.517812  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.517714  6102 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.484584  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.580744  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:01.519038  6066 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.491104  6080 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.520925  1591 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:01.521616  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:01.588343  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.588557  2312 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:01.591661  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:01.592592  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 128) took 21192 ms (client timeout 15000 ms). Trace:
I20260709 13:03:01.592645  1589 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:03:01.494724  5792 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:01.593876  5792 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.598560  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.599323  1662 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.599884  1491 catalog_manager.cc:5697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:01.601809  1491 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:01.579213  1593 rpcz_store.cc:276] 0709 13:03:00.210143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:00.349231 (+139088us) service_pool.cc:224] Handling call
0709 13:03:00.360122 (+ 10891us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:00.471981 (+111859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:00.472036 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:00.509887 (+ 37851us) log.cc:844] Serialized 26 byte log entry
0709 13:03:00.589609 (+ 79722us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:00.589857 (+   248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.498190 (+908333us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54003b10d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:01.498242 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:01.498296 (+    54us) raft_consensus.cc:1685] finished
0709 13:03:01.498310 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.505529 (+  7219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33632,"raft.queue_time_us":41475,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":9960,"spinlock_wait_cycles":152294144,"thread_start_us":78971,"threads_started":1,"wal-append.queue_time_us":285815}
I20260709 13:03:01.599349  6054 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:01.610404  6054 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:01.611307  6054 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.612584  6054 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:01.521700  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:01.494397  6063 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.522817  6034 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.615082  2310 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:01.522876  1634 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.784s (SENT)
I20260709 13:03:01.526307  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.492753  6062 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:01.509009  5881 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:01.533421  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.508931  6032 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.487648  6017 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.549571  1661 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
W20260709 13:03:01.557523  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.633431  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.634898  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:01.573396  1588 rpcz_store.cc:276] 0709 13:03:00.210522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.416848 (+1206326us) service_pool.cc:224] Handling call
0709 13:03:01.417149 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.417383 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:01.417776 (+   393us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.546058 (+128282us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.572591 (+ 26533us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:01.572650 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:01.572694 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.572767 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.572894 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":81194,"prepare.queue_time_us":4713,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":24455,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":1611,"thread_start_us":87224,"threads_started":1,"wal-append.queue_time_us":89771}
I20260709 13:03:01.512089  6055 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:01.580156  1706 connection.cc:256] RPC call timeout handler was delayed by -1.5342282240000031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30560000000000004s for a response
I20260709 13:03:01.517385  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:01.584875  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86015.1us
I20260709 13:03:01.618000  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.619527  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:01.572307  6023 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.622623  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:01.640522  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:01.625813  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.641180  1739 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:01.618834  5898 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.631089  2311 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:01.569922  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.636612  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:01.627408  2310 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:01.586696  6096 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.638553  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.643971  1585 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:01.626219  6071 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.626595  5981 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.621912  6069 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:01.625553  5872 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:01.627195  6097 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:01.627643  6062 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:01.637543  6055 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:01.647506  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 124678us
I20260709 13:03:01.687523  1667 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:01.647604  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.646502  1662 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:01.650593  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.654635  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.655356  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.694458  1659 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
W20260709 13:03:01.647948  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 126499us
I20260709 13:03:01.656229  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:01.695839  1593 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:01.656992  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:01.697003  1589 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:01.663460  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:01.698429  1738 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:01.658649  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:01.679354  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 608) took 1468 ms. Trace:
I20260709 13:03:01.700075  1586 rpcz_store.cc:276] 0709 13:03:00.210854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.416989 (+1206135us) service_pool.cc:224] Handling call
0709 13:03:01.417300 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.417591 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:01.496784 (+ 79193us) log.cc:844] Serialized 14 byte log entry
0709 13:03:01.678521 (+181737us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:01.678592 (+    71us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:01.678644 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.679143 (+   499us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:01.679293 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":20674,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":4110,"thread_start_us":186621,"threads_started":2}
I20260709 13:03:01.663722  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2666) took 1156 ms. Trace:
W20260709 13:03:01.686275  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102256us
I20260709 13:03:01.647143  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:01.702545  1585 rpcz_store.cc:276] 0709 13:03:00.507638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.378755 (+871117us) service_pool.cc:224] Handling call
0709 13:03:01.663623 (+284868us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:01.687824  1634 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.785s (SENT)
I20260709 13:03:01.706549  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:01.634152  6109 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:03:01.646996  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:01.712169  1666 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:03:01.712066  1660 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:01.643422  6033 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 13:03:01.715374  1662 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:01.716622  1662 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:01.716796  1741 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:01.718969  1660 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 13:03:01.725061  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:01.716943  1584 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:01.729120  1660 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:01.648723  6022 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.681062  6055 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:01.647361  6104 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:01.661974  5898 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.746275  5898 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.746827  5898 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:01.747282  5898 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.662593  6069 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:01.655185  6034 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.733731  1659 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [8.4-12.6]
I20260709 13:03:01.664073  6097 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:01.728091  6129 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 13:03:01.736753  1491 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 3 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:01.738013  1662 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.726174  5660 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 13:03:01.647624  6023 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.773921  1666 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:03:01.742921  6080 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.750207  1665 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:01.750892  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.728010  5881 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:01.758024  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.758659  2310 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:01.678849  5872 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:01.761924  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47277.8us
I20260709 13:03:01.749271  5898 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.667593  6071 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.676007  5981 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.775095  6135 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.711669  6096 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.775710  1667 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:01.764050  1738 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:01.866889  1735 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:01.883455  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:01.884663  1586 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 13:03:01.775089  1491 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 3 to 4, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:03:01.815459  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.769991  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:01.800786  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.836668  1659 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:01.888931  1585 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:01.837268  1662 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:01.760555  6069 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:01.892030  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54829.7us
I20260709 13:03:01.881758  1664 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:01.893915  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 136008us
I20260709 13:03:01.896131  1593 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:01.764810  6097 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:01.901468  6068 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:01.903107  5981 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:01.849427  1763 debug-util.cc:398] Leaking SignalData structure 0x7b08003cdde0 after lost signal to thread 434
W20260709 13:03:01.905167  1763 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4c80 after lost signal to thread 1525
I20260709 13:03:01.905859  6137 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:01.906898  1763 debug-util.cc:398] Leaking SignalData structure 0x7b0800362240 after lost signal to thread 1617
W20260709 13:03:01.908399  1763 debug-util.cc:398] Leaking SignalData structure 0x7b08001977a0 after lost signal to thread 1692
W20260709 13:03:01.908725  1763 debug-util.cc:398] Leaking SignalData structure 0x7b0800341920 after lost signal to thread 1693
I20260709 13:03:01.919310  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:01.920441  1763 debug-util.cc:398] Leaking SignalData structure 0x7b080010efe0 after lost signal to thread 1766
W20260709 13:03:01.922163  1763 debug-util.cc:398] Leaking SignalData structure 0x7b08002fc080 after lost signal to thread 2342
I20260709 13:03:01.773428  6022 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:01.930544  5792 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.931967  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:01.932862  5792 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:01.878006  6033 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:01.924475  1763 debug-util.cc:398] Leaking SignalData structure 0x7b080030b560 after lost signal to thread 6039
I20260709 13:03:01.882128  6017 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:03:01.928704  6023 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:01.894733  5872 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.930099  6096 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:01.867671  6127 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:01.929947  1491 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:01.930491  6068 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:01.910056  1734 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:01.867484  6134 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:01.909593  6063 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:01.940325  1634 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.080s (SENT)
I20260709 13:03:01.960624  1659 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:01.773547  6109 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.179s
W20260709 13:03:01.949586  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53924us
I20260709 13:03:01.961166  1659 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:01.968586  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 390179us
I20260709 13:03:01.968753  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:01.939704  6087 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:01.949055  1589 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:01.972433  1662 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:03:01.974457  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 335996us
I20260709 13:03:01.966918  1739 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
W20260709 13:03:01.978799  1706 connection.cc:256] RPC call timeout handler was delayed by -1.0277146899999821s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.27690000000000003s for a response
I20260709 13:03:01.946930  6127 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:01.960282  5660 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:01.958979  6068 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:01.762996  6094 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.213s
I20260709 13:03:01.989975  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:01.990720  2310 raft_consensus.cc:606] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:01.983079  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:01.973265  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1892) took 1085 ms. Trace:
I20260709 13:03:01.944347  6138 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:01.959247  6134 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:01.946485  6057 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:03:01.895764  6135 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.985461  1659 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:01.997917  1659 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:01.997045  6034 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.999260  6135 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:01.593392  1486 rpcz_store.cc:269] 0709 13:02:40.399646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:40.402989 (+  3343us) service_pool.cc:224] Handling call
0709 13:03:01.592317 (+21189328us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:02:50.716031 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:02:50.906193 (+190162us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:02:50.906265 (+    72us) write_op.cc:435] Acquired schema lock
0709 13:02:50.906301 (+    36us) tablet.cc:662] Decoding operations
0709 13:02:50.907770 (+  1469us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:02:50.907851 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:02:50.907885 (+    34us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:02:50.944222 (+ 36337us) tablet.cc:701] Row locks acquired
0709 13:02:50.944276 (+    54us) write_op.cc:260] PREPARE: finished
0709 13:02:50.947137 (+  2861us) write_op.cc:270] Start()
0709 13:02:50.962337 (+ 15200us) write_op.cc:276] Timestamp: P: 1783602170944399 usec, L: 0
0709 13:02:50.963579 (+  1242us) op_driver.cc:348] REPLICATION: starting
0709 13:02:51.003120 (+ 39541us) log.cc:844] Serialized 5889 byte log entry
0709 13:02:51.951092 (+947972us) op_driver.cc:464] REPLICATION: finished
0709 13:02:52.200771 (+249679us) write_op.cc:301] APPLY: starting
0709 13:02:52.200924 (+   153us) tablet.cc:1367] starting BulkCheckPresence
0709 13:02:52.573165 (+372241us) tablet.cc:1370] finished BulkCheckPresence
0709 13:02:52.573327 (+   162us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:02:53.526059 (+952732us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:02:53.526615 (+   556us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:02:53.526682 (+    67us) write_op.cc:312] APPLY: finished
0709 13:02:53.527427 (+   745us) log.cc:844] Serialized 129 byte log entry
0709 13:02:55.817185 (+2289758us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdeaed30ac 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b
0709 13:02:56.147820 (+330635us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:56.180015 (+ 32195us) write_op.cc:454] Released schema lock
0709 13:02:56.212238 (+ 32223us) write_op.cc:341] FINISH: Updating metrics
0709 13:02:56.318664 (+106426us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:02:56.324243 (+  5579us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1477221,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":249260,"apply.run_cpu_time_us":99867,"apply.run_wall_time_us":4078691,"cfile_cache_hit":54,"cfile_cache_hit_bytes":56457,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6218,"cfile_init":2,"lbm_read_time_us":44833,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":10,"num_ops":14,"prepare.queue_time_us":381127,"prepare.run_cpu_time_us":32425,"prepare.run_wall_time_us":651428,"raft.queue_time_us":405771,"raft.run_cpu_time_us":25518,"raft.run_wall_time_us":188536,"replication_time_us":987233,"spinlock_wait_cycles":27841792,"thread_start_us":3563377,"threads_started":5,"wal-append.queue_time_us":2966087,"wal-append.run_cpu_time_us":39173,"wal-append.run_wall_time_us":2580562}]]}
W20260709 13:03:02.000984  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:01.982905  6096 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:02.004932  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:01.978192  6054 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:01.975430  6087 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:02.006875  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69846.5us
W20260709 13:03:02.007540  1634 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.168s (SENT)
I20260709 13:03:02.002900  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:01.965932  6104 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 13:03:02.009194  1662 rpcz_store.cc:276] 0709 13:03:00.887827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.707971 (+820144us) service_pool.cc:224] Handling call
0709 13:03:01.708366 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.972937 (+264571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:01.973205 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39552}
I20260709 13:03:01.985141  5660 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:02.012928  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.023272  6138 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:01.981838  6063 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:01.968065  5881 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:03:02.020610  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.071339  1741 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:02.021031  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.072697  1739 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:02.001860  6096 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.073694  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1888) took 1187 ms. Trace:
I20260709 13:03:02.073868  6096 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:02.074127  1664 rpcz_store.cc:276] 0709 13:03:00.886603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.517159 (+630556us) service_pool.cc:224] Handling call
0709 13:03:02.073464 (+556305us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54002535d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:02.073629 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":560247040}
I20260709 13:03:02.074704  6022 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:02.075505  6022 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:02.075788  1660 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 13:03:02.075870  6022 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.076400  6022 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:02.015472  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 290511us
I20260709 13:03:02.077979  6068 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.078902  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:02.032780  6138 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:02.033147  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:01.986558  5898 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 13:03:02.081367  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:02.033860  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:01.999300  6034 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.035096  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:02.035477  6071 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:02.036979  1634 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.196s (SENT)
I20260709 13:03:02.039242  6102 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.040093  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.044281  6018 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.048139  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1895) took 1159 ms. Trace:
I20260709 13:03:02.048887  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.095757  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:02.096295  1589 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:02.049206  1584 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:02.096973  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.051540  1742 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:02.052686  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.053983  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.055897  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.099774  1588 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:02.056080  6127 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:01.984903  5981 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:03:02.057924  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.105155  1585 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.057523  5660 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.026695  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.062389  1587 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:02.064054  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.017024  5792 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:02.010344  6087 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.142689  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.144248  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.144439  6143 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:02.145247  6143 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:02.146572  6143 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.069794  6134 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.148272  6134 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:02.149788  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.152190  6023 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:02.152705  6023 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:02.153139  6023 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:02.153283  5660 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.115070  6017 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:02.086687  6063 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.154059  6017 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:02.020557  6104 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:02.085194  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.013972  6054 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:02.087375  1738 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:02.169737  1739 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:03:02.079396  6068 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.089920  2310 raft_consensus.cc:606] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:02.174129  2310 raft_consensus.cc:614] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:02.092837  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:02.094393  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 429739us
I20260709 13:03:02.050870  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.175341  1593 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.095238  1665 rpcz_store.cc:276] 0709 13:03:00.888960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.708828 (+819868us) service_pool.cc:224] Handling call
0709 13:03:01.733267 (+ 24439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.888521 (+155254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.047901 (+159380us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b5400014ad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:02.048065 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4554112}
I20260709 13:03:02.103233  1590 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
W20260709 13:03:02.110023  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46117.3us
I20260709 13:03:02.176604  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.112599  1660 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:02.116739  5872 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.090852  6062 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.117686  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.122219  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.122609  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.125279  6034 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.114619  6071 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:02.131880  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 123349us
W20260709 13:03:02.216005  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.133437  6138 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:02.070295  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73940.6us
I20260709 13:03:02.217826  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:02.219795  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.157449  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.158980  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.222378  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.223824  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.122299  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.225917  1740 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:02.227218  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.228946  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:02.164932  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.145397  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.178009  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.177135  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.177757  1660 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:02.194880  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.295166  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.198825  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.113727  5792 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.130805  5981 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:03:02.212983  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.300820  1738 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
W20260709 13:03:02.161031  6102 consensus_queue.cc:1175] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df6a18af06d543b2931837f5e6bab95f" responder_term: 9 status { last_received { term: 9 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 9 index: 10 } }
I20260709 13:03:02.176505  6135 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:02.270920  6054 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.288944  1739 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
W20260709 13:03:02.262598  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105204us
I20260709 13:03:02.312705  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.234558  1659 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:03:02.313413  1659 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 6
I20260709 13:03:02.313740  1659 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:02.313259  1585 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:02.314149  1659 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:02.287760  1733 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:03:02.112479  5881 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.102s
I20260709 13:03:02.318528  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.290478  6018 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:02.320838  1588 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:02.292837  1661 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:02.225138  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.287760  1740 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:02.297266  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 120246us
I20260709 13:03:02.337347  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.293311  6034 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 13:03:02.303246  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 83602.1us
I20260709 13:03:02.289366  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 7.
I20260709 13:03:02.314745  1738 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 13:03:02.300350  5792 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:02.321667  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 273910us
I20260709 13:03:02.328989  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.346385  1664 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:02.329816  1593 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:02.294432  6137 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.347874  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:02.301858  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 123748us
I20260709 13:03:02.335037  1663 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:02.290596  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.297920  6060 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:02.332240  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.350028  1738 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 13:03:02.354001  1665 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:02.360133  1665 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:02.356236  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.319701  6063 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.349687  1660 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.357774  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:02.332861  5872 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:02.274816  6017 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:02.392890  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 155042us
I20260709 13:03:02.304523  6097 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.398896  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:02.399825  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:02.400382  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.359781  1738 pending_rounds.cc:85] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 6
I20260709 13:03:02.401290  1738 pending_rounds.cc:107] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 13:03:02.401620  1738 raft_consensus.cc:2889] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:02.393630  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.396708  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.403257  1738 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:02.344667  6018 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:02.394789  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1897) took 1229 ms. Trace:
I20260709 13:03:02.341141  6032 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:02.420037  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.404369  1660 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.348084  6137 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.404321  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:02.374446  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.403818  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.349663  6136 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:02.412381  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.408308  1667 rpcz_store.cc:276] 0709 13:03:01.165051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.890552 (+725501us) service_pool.cc:224] Handling call
0709 13:03:01.922896 (+ 32344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.923167 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 13:03:02.178824 (+255657us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.394365 (+215541us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:02.394432 (+    67us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:02.394485 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.394566 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:02.394722 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":375584,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"thread_start_us":459956,"threads_started":2}
I20260709 13:03:02.413100  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.413424  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.373615  6054 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.423281  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.423373  1665 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.425203  1737 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:02.346170  6067 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 13:03:02.395874  6144 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:02.352720  6071 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 13:03:02.446842  1590 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:02.398644  6033 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:02.403575  6034 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:02.432425  2315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:02.431411  1585 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
W20260709 13:03:02.465783  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50557.7us
I20260709 13:03:02.432775  2310 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:02.433071  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.430969  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.472596  2311 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
W20260709 13:03:02.434481  2282 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.007s	sys 0.013s
W20260709 13:03:02.474193  2282 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.007s	sys 0.013s
I20260709 13:03:02.443850  1738 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:02.438001  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:02.431275  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.479056  1667 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:02.432873  1592 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:02.480458  2315 raft_consensus.cc:606] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:02.442714  1663 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:03:02.485386  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.489334  2315 raft_consensus.cc:614] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:02.494412  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.488072  1663 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:02.427855  6149 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:02.467329  6033 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.450233  6151 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.423615  6060 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 13:03:02.437598  6062 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.509096  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:02.538723  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.443260  6136 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:02.484297  5981 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:02.439440  6097 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:02.521535  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 117326us
I20260709 13:03:02.524798  1665 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:02.532979  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 173014us
I20260709 13:03:02.507263  6034 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:02.535959  5881 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:02.590200  5881 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:02.566167  1663 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.575896  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:02.529475  6033 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:02.583730  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.590215  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.585970  6067 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:02.563860  6104 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 13:03:02.603725  6067 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.604184  6067 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:02.590742  5881 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.544063  6136 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:02.593667  6152 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:02.598461  1663 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:02.612681  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:02.570314  6055 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.808s	user 0.012s	sys 0.004s
I20260709 13:03:02.613545  1584 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.601748  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.614172  1587 raft_consensus.cc:2376] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11. Re-sending same reply.
I20260709 13:03:02.614041  6055 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.615504  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.616695  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.616899  6055 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.603123  1706 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.617748  1740 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:02.564563  6018 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.617784  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.618816  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.601114  1589 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:02.620040  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.620344  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.620671  1589 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:02.621021  6131 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:02.621448  6131 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.621835  6131 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:02.627552  6022 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.000s
I20260709 13:03:02.620059  6055 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:02.628055  6022 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.630355  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.631250  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:02.564198  5981 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.591787  6144 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.639380  1663 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:02.539103  6057 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 13:03:02.642446  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:02.654047  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46472.6us
I20260709 13:03:02.619957  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.590215  6072 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:02.645287  5898 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:02.646469  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 297349us
I20260709 13:03:02.666654  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:02.645958  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 271604us
W20260709 13:03:02.581774  6069 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.680s	user 0.010s	sys 0.003s
I20260709 13:03:02.669576  6069 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.654834  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:02.658847  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.662748  1589 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 13:03:02.650838  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52692.5us
I20260709 13:03:02.670775  1661 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:02.591244  6151 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.674696  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.675454  1587 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:02.649225  6097 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:02.650394  5981 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.672466  6069 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:02.676838  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:02.685269  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.715556  2281 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.715890  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:02.719147  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44897.3us
I20260709 13:03:02.660081  6072 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:02.726092  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:02.731889  1734 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:03:02.676514  6151 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:02.734833  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 240542us
I20260709 13:03:02.736016  1584 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:03:02.738767  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:02.739666  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.718027  6154 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:02.745923  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.676728  6104 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
W20260709 13:03:02.736672  6134 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.585s	user 0.013s	sys 0.001s
I20260709 13:03:02.753880  6134 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:02.754344  6134 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.755079  6134 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:02.755394  6097 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:02.756151  6097 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.757843  6063 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.758502  6063 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.742517  6145 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.760038  6145 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.760763  6063 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.762296  6145 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.762995  6097 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:02.763662  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.761439  6063 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:02.730582  6174 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:02.758888  6152 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:02.739936  6137 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:02.796145  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 667) took 1365 ms. Trace:
I20260709 13:03:02.744684  6055 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 13:03:02.794292  6149 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:02.798694  6182 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.797052  1662 rpcz_store.cc:276] 0709 13:03:01.430474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.023665 (+593191us) service_pool.cc:224] Handling call
0709 13:03:02.023995 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:02.024276 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:02.301096 (+276820us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.677931 (+376835us) log.cc:844] Serialized 14 byte log entry
0709 13:03:02.795689 (+117758us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:02.795749 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:02.795807 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:02.795886 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:02.796050 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17152,"prepare.queue_time_us":319938,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":117,"raft.queue_time_us":28458,"spinlock_wait_cycles":31427072,"thread_start_us":638059,"threads_started":3,"wal-append.queue_time_us":418757}
I20260709 13:03:02.793167  6109 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:02.792330  6152 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:02.743486  6034 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.813858  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:02.814144  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68364.9us
I20260709 13:03:02.819280  2311 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:03:02.826077  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 162943us
I20260709 13:03:02.790009  6136 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.827349  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:02.828099  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:02.827864  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 161329us
I20260709 13:03:02.813992  6109 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.817801  6152 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:02.842350  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.843237  1589 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:02.812773  6182 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.793664  6072 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:02.846144  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 160970us
W20260709 13:03:02.847330  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 131446us
W20260709 13:03:02.849099  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:02.849987  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:02.838686  6096 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.764s	user 0.015s	sys 0.000s
I20260709 13:03:02.853284  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.823467  6151 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:03:02.835472  6109 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:02.861383  1660 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:02.861801  6151 consensus_queue.cc:1243] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:02.862429  6072 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.862891  6183 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:02.863471  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:02.867682  1589 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:02.867417  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.912514  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.914243  6174 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:03:02.918251  6171 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:03:02.918853  6102 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:02.919845  6177 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:03:02.921725  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:02.913085  2318 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:02.924314  1665 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:02.927414  6154 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:02.927914  6154 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:02.930975  6180 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:03:02.933882  6131 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.934299  5981 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.936184  6131 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.936543  5981 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:02.865862  6142 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:03:02.937177  6063 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:02.912005  6183 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.938812  6183 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:02.940567  2311 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:02.941918  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.942890  1586 raft_consensus.cc:2410] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 13:03:02.936930  1735 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:02.943516  6191 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:02.943742  5872 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:02.944408  5872 consensus_queue.cc:1243] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:02.850812  5881 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:02.948308  5881 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:02.948973  1668 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:02.950577  6136 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.950748  6189 raft_consensus.cc:993] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
W20260709 13:03:02.951268  6094 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 13:03:02.951460  6189 raft_consensus.cc:1081] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:02.953528  6188 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:02.954070  6188 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.858675  6055 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:02.956552  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.956776  6127 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:02.936832  6181 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 13:03:02.860579  6096 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.959432  6187 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:02.961752  1591 raft_consensus.cc:2368] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:02.944010  6191 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.959276  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:02.954403  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:02.967307  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:02.976629  6186 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:02.976786  6188 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.961272  6096 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.979110  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.979836  1665 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:02.980957  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.982203  2318 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:02.983757  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.984328  1661 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:02.990397  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.990687  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:02.993130  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.993574  2313 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:02.977592  6186 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:02.993010  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 225393us
I20260709 13:03:02.928843  6154 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:02.995064  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.996294  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.989040  6182 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:02.937232  5981 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:02.998620  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:02.979135  5881 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:02.996292  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:02.980533  6063 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:03.002357  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.006085  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.006763  1660 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.006446  1661 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
W20260709 13:03:02.991329  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80373.6us
I20260709 13:03:02.978075  6191 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.002414  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:03.010450  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.013896  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:03.018782  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.019601  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.019430  1661 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:03.020226  1664 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.021214  2309 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:03.024823  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.025642  1739 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.027690  1660 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:03.027374  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.037595  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.039371  1663 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:03:03.039652  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.039841  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1896) took 1927 ms. Trace:
I20260709 13:03:02.983368  6189 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:03.039683  1663 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:03.042851  1663 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:03.043514  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.044162  1662 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:03.040262  1666 rpcz_store.cc:276] 0709 13:03:01.112699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:01.853764 (+741065us) service_pool.cc:224] Handling call
0709 13:03:01.904003 (+ 50239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:01.904241 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:02.085718 (+181477us) log.cc:844] Serialized 14 byte log entry
0709 13:03:03.039378 (+953660us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b1800131860. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:03:03.039490 (+   112us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:03.039549 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:03.039596 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.039676 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:03.039783 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":329970,"prepare.run_cpu_time_us":1530,"prepare.run_wall_time_us":15156,"raft.queue_time_us":147411,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":25111,"spinlock_wait_cycles":9623808,"thread_start_us":611563,"threads_started":3,"wal-append.queue_time_us":421193}
I20260709 13:03:03.016957  6154 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:03.047823  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 216476us
I20260709 13:03:03.060353  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:03.061288  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 91462.5us
I20260709 13:03:03.070204  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:02.991564  6176 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:03:03.016739  6063 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:02.860883  6149 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.064966  6183 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.085812  6128 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:03.016498  5881 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:03.071617  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 106118us
I20260709 13:03:03.076188  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:03.086800  5881 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:03.043036  1740 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
I20260709 13:03:03.087513  1740 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:03.087929  1740 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:03.088385  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.089527  6054 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:03.089622  6128 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.090824  6138 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:03.091214  6138 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:03.092084  6128 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.092046  6138 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.089659  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:03.094065  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:03.094635  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.095463  6054 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:03.095882  6054 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.096009  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:03.096480  6128 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:03.097143  6128 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.096662  6054 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:03.095866  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.045806  6097 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:03.104547  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.111383  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.107036  1740 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.118436  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:03.119733  6197 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:03.045969  6161 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:03.121306  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.121145  6206 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:03.125128  2318 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:03.110039  6128 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:03.079965  6063 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:03.132830  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.134045  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.110003  6181 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:03.136615  1739 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:03.136917  1740 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.137480  1664 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:03.121356  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.076857  6096 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260709 13:03:03.123692  6145 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:03.116619  6097 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.146209  2309 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:03:03.130254  5792 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.737s	user 0.012s	sys 0.001s
I20260709 13:03:03.147575  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.124892  6136 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:03.083029  6182 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 13:03:03.149418  6097 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.109411  6055 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260709 13:03:03.166334  6193 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:03:03.167101  6067 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.003s
I20260709 13:03:03.167554  6067 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.169778  1585 raft_consensus.cc:2368] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
I20260709 13:03:03.169910  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:03.170310  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:03.171172  1590 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 13:03:03.171394  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:03.171901  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:03.172459  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:03.172991  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:03.174835  6087 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.033s	user 0.012s	sys 0.010s
I20260709 13:03:03.175246  6087 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:03.177726  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 107578us
I20260709 13:03:03.179005  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:03.145419  6145 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:03.157775  6190 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
W20260709 13:03:03.166673  6023 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.990s	user 0.009s	sys 0.006s
I20260709 13:03:03.161621  5792 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.187345  6183 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:03.188472  5792 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:03.181452  6178 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 13:03:03.185914  1659 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.153000  6195 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:03.190289  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:03.160987  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:03.191339  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:03.122164  6142 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 13:03:03.198282  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.146512  6184 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.141664  6181 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:03.160810  6188 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:03.141856  6206 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.244063  1735 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:03.213568  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.246048  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.185945  6067 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.196497  5872 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.271116  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:03.163944  6136 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:03.275255  1739 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:03.210891  6096 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:03.211377  5981 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:03.177417  6087 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.239713  6197 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:03.211650  6145 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.283917  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:03.247816  6188 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.280194  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:03.190024  6023 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.284778  1739 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:03.245393  6181 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.248793  6055 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:03.296016  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.309602  1735 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:03.296140  2317 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:03.273845  6195 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:03.320044  6195 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:03.313818  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 122577us
I20260709 13:03:03.324352  5792 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:03.277293  6136 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.325724  5792 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:03.326277  5792 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.327487  5792 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.124428  6186 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.318491  6170 raft_consensus.cc:2764] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 13:03:03.248157  6184 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.348459  6184 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.280567  6197 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:03.349997  6197 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:03.356333  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.357127  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.362270  6144 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.701s	user 0.000s	sys 0.012s
I20260709 13:03:03.362895  6144 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:03.329340  5792 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.243844  6190 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 13:03:03.278640  5981 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:03.290621  6188 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.273330  6134 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.290827  6067 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:03.417016  2311 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:03.426640  6206 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:03.419546  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 298319us
I20260709 13:03:03.438236  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.442054  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.402648  5981 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.287230  6102 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.130s
I20260709 13:03:03.458177  1736 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.462010  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.416821  6067 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.390426  5881 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.432286  6184 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.294672  6023 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:03.439558  6141 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.029s	sys 0.055s
W20260709 13:03:03.482781  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.467664  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 254257us
I20260709 13:03:03.447361  5898 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:03.502125  6109 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.589s	user 0.010s	sys 0.002s
I20260709 13:03:03.502682  6109 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.467196  6207 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:03.471660  6067 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:03.511407  5898 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.527048  5898 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.527779  1735 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:03.511878  6127 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.000s
I20260709 13:03:03.512995  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:03.514596  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.530681  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.531005  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:03.531689  1590 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.520349  6097 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.521034  6109 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:03.523053  5872 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:03.524111  6173 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.739s	user 0.003s	sys 0.000s
I20260709 13:03:03.485036  6184 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.534889  6134 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:03.535394  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2624) took 1946 ms (client timeout 1562 ms). Trace:
I20260709 13:03:03.534763  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:03.538411  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.479764  6174 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:03.533397  1587 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:03.543438  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:03.546756  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.572435  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.539115  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:03.546255  1591 rpcz_store.cc:269] 0709 13:03:01.588591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.589876 (+1001285us) service_pool.cc:224] Handling call
0709 13:03:03.535151 (+945275us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400443254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:03.535303 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":839421696}
W20260709 13:03:03.576568  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.579411  1665 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [9.6-12.7]
I20260709 13:03:03.579913  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.581079  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:03.581656  1593 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.582258  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:03.481163  6141 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.975s	user 0.030s	sys 0.055s
W20260709 13:03:03.572158  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.545852  6057 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:03.590240  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51253.4us
I20260709 13:03:03.594784  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.595183  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.595705  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.597280  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.597954  1666 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:03.598050  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.599941  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:03.601343  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.601878  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.602130  1662 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:03.603361  1667 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:03.605144  1735 raft_consensus.cc:1217] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Deduplicated request from leader. Original: 6.4->[8.5-12.7]   Dedup: 8.5->[10.6-12.7]
I20260709 13:03:03.601347  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:03.605818  1735 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [8.5-12.7]
I20260709 13:03:03.606443  1663 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:03.610486  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.614529  1735 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.615602  1591 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 13:03:03.616451  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2640) took 1012 ms. Trace:
I20260709 13:03:03.616808  1591 rpcz_store.cc:276] 0709 13:03:02.603923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.599456 (+995533us) service_pool.cc:224] Handling call
0709 13:03:03.615252 (+ 15796us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.616180 (+   928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.616387 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:03.601472  6072 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.000s
I20260709 13:03:03.620085  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.626314  1666 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.621533  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.487860  6145 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.618655  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.611374  6055 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:03.573199  6181 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.594489  6063 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.669718  1584 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:03:03.599175  6186 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:03.604629  6109 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:03.671350  1662 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 13:03:03.672276  1632 connection.cc:256] RPC call timeout handler was delayed by -3.6156278089999887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
W20260709 13:03:03.622041  6033 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.970s	user 0.002s	sys 0.013s
I20260709 13:03:03.678001  6216 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:03.700964  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79581.1us
W20260709 13:03:03.702463  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78927.9us
W20260709 13:03:03.703058  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.690222  1593 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:03.601075  5792 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:03.627727  6207 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:03.623530  6182 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:03.699003  1632 connection.cc:256] RPC call timeout handler was delayed by -3.1314278089999448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2477s for a response
I20260709 13:03:03.614886  6071 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:03.601389  6057 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.704308  5792 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.719844  5792 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:03.720690  6057 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.616914  6127 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260709 13:03:03.709199  1739 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 15. Ops: []
I20260709 13:03:03.725210  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.725860  6070 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:03.726405  6142 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.726967  6142 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.727519  6070 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:03.731287  1587 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:03.697248  6152 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.854s	user 0.011s	sys 0.003s
I20260709 13:03:03.674471  6072 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
W20260709 13:03:03.714709  6212 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
I20260709 13:03:03.666301  6096 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 13:03:03.666800  6170 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
I20260709 13:03:03.717854  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:03.732138  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2642) took 1127 ms. Trace:
I20260709 13:03:03.773269  1587 rpcz_store.cc:276] 0709 13:03:02.604467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.730695 (+1126228us) service_pool.cc:224] Handling call
0709 13:03:03.731071 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.731836 (+   765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.732071 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:03.712041  6055 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:03.765193  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:03.777813  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:03.708199  6207 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.772066  1741 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:03.709798  6182 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.734573  1666 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:03.774336  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 677) took 1574 ms. Trace:
W20260709 13:03:03.774984  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66999.1us
I20260709 13:03:03.738281  6142 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.711496  6216 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.717900  6227 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:03.784646  1659 rpcz_store.cc:276] 0709 13:03:02.200184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.350484 (+150300us) service_pool.cc:224] Handling call
0709 13:03:02.395546 (+ 45062us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.189775 (+794229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.774068 (+584293us) spinlock_profiling.cc:243] Waited 790 ms on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:03.774270 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1817345280}
I20260709 13:03:03.704388  6186 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:03.780983  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 162454us
I20260709 13:03:03.707907  6033 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.820869  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.777259  6071 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:03.738142  6188 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:03.832371  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 114654us
W20260709 13:03:03.823642  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 119881us
I20260709 13:03:03.782862  6057 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:03.774770  6136 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.783728  5881 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:03.767710  6152 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.782757  6207 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:03.786886  6182 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:03.819950  6227 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:03.786206  6127 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.864487  1592 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:03.888444  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:03.786669  6055 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:03.872452  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:03.874553  6128 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.742s	user 0.012s	sys 0.001s
I20260709 13:03:03.877094  6152 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:03.877902  6233 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:03:03.878257  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2650) took 1046 ms. Trace:
W20260709 13:03:03.874583  6220 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
W20260709 13:03:03.880826  6197 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
W20260709 13:03:03.881979  6054 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.746s	user 0.011s	sys 0.001s
I20260709 13:03:03.883544  1741 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:03:03.886387  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.886618  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.927362  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:03.890719  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:03.838526  6097 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:03.888224  6227 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:03.893791  1737 raft_consensus.cc:2410] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 13:03:03.857445  6191 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.825s	user 0.009s	sys 0.008s
I20260709 13:03:03.896639  1589 rpcz_store.cc:276] 0709 13:03:02.827081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.843816 (+ 16735us) service_pool.cc:224] Handling call
0709 13:03:03.873277 (+1029461us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54001def54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:03.873446 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25113472}
I20260709 13:03:03.880911  5981 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.898953  6057 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:03.900938  6236 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 13:03:03.904300  5881 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:03.910539  6066 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.910323  6225 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:03.835386  6071 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.895987  6128 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.865408  6184 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:03.923974  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:03.924809  1590 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:03.900684  6214 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 13:03:03.935192  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.946442  1589 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:03.956627  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2644) took 1351 ms. Trace:
I20260709 13:03:03.952502  1590 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 13:03:03.958362  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2647) took 1351 ms. Trace:
I20260709 13:03:03.958823  1590 rpcz_store.cc:276] 0709 13:03:02.607212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.951905 (+1344693us) service_pool.cc:224] Handling call
0709 13:03:03.952260 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.957949 (+  5689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.958290 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:03.959851  1588 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:03.960414  1590 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:03.961058  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2651) took 1133 ms. Trace:
I20260709 13:03:03.957006  1589 rpcz_store.cc:276] 0709 13:03:02.605332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.945777 (+1340445us) service_pool.cc:224] Handling call
0709 13:03:03.946203 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.956278 (+ 10075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.956560 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:03.962146  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.962975  1589 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:03.963553  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2652) took 1136 ms. Trace:
I20260709 13:03:03.963836  1589 rpcz_store.cc:276] 0709 13:03:02.827639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.962510 (+1134871us) service_pool.cc:224] Handling call
0709 13:03:03.962798 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.963326 (+   528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.963498 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:03.952176  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65897.3us
W20260709 13:03:03.931308  6154 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.857s	user 0.013s	sys 0.001s
I20260709 13:03:03.946147  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:03.978150  1741 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:03.964270  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:03.961344  1590 rpcz_store.cc:276] 0709 13:03:02.827382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.959828 (+1132446us) service_pool.cc:224] Handling call
0709 13:03:03.960151 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.960830 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.961009 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:03.920230  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:03.974845  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:03.929253  6181 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:03.980986  1585 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:03.983083  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:03.928861  6197 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.966301  6216 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:03.895955  6127 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.987298  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.956856  6141 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:03.982709  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:03.991746  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:04.003264  1590 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 13:03:03.992163  1742 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:03.934628  6067 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.008038  6127 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.009348  6230 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:04.009730  6067 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:04.009855  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89852.9us
I20260709 13:03:04.010289  6230 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:04.010766  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:04.010937  6055 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.011400  6230 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.012082  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:04.012733  5898 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:03.931820  6152 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 13:03:04.014235  1589 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 13:03:03.976076  6236 consensus_queue.cc:1243] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:03.950788  6072 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 13:03:03.932734  6054 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.014513  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2655) took 1179 ms. Trace:
I20260709 13:03:04.034201  1590 rpcz_store.cc:276] 0709 13:03:02.834710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.979449 (+1144739us) service_pool.cc:224] Handling call
0709 13:03:04.003001 (+ 23552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.004825 (+  1824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.014449 (+  9624us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:03.966954  6057 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.015748  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.038290  1660 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:04.021167  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:03.933394  5792 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.013441  6162 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:03.987845  6062 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:03.990981  6195 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:04.001045  6197 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:03.945549  6191 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.958168  6066 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:03.985703  6102 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 13:03:04.048254  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.048401  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.058218  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:04.014467  6033 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.020952  6094 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:04.018390  6231 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:04.018374  5898 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:04.034323  6072 consensus_queue.cc:1243] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 9.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 13:03:04.058077  1659 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:04.061676  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 240938us
I20260709 13:03:04.033293  6141 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:04.044780  6195 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:04.155316  1740 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:04.063000  5881 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.064509  6226 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.064555  6069 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:04.197284  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.050061  6062 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.199519  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:04.258777  1737 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.197387  6229 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.231302  6213 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:04.066355  6184 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:04.091691  6188 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.277064  6071 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:04.278635  6239 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260709 13:03:04.063231  6127 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.157575  6195 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:04.196928  6136 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.159005  6145 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:04.262710  6054 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.277565  6235 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:04.266320  6242 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.107s	user 0.007s	sys 0.000s
I20260709 13:03:04.266974  6191 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.156628  6094 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.104s
I20260709 13:03:04.267536  5898 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.066318  6231 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.273874  6213 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:04.275432  6184 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.236667  5881 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.249603  6181 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.156941  6225 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.116s
I20260709 13:03:04.249362  6186 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.273716  6069 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.303174  1741 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:04.257643  6226 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.248677  6057 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.295220  6128 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.314412  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:04.305835  6213 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:04.292511  6127 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.279729  6229 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.297660  6145 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.323165  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.309316  5981 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.328807  6135 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:04.328967  6062 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.329306  6135 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:04.331768  1667 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:04.332314  6231 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.302264  6141 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.333556  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2643) took 1728 ms. Trace:
I20260709 13:03:04.333680  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.334889  5881 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.335132  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.335846  1666 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:04.336272  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 435) took 1383 ms. Trace:
I20260709 13:03:04.336526  1659 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:04.336679  2311 rpcz_store.cc:276] 0709 13:03:02.953170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.953401 (+   231us) service_pool.cc:224] Handling call
0709 13:03:02.953887 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:03.417681 (+463794us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.336035 (+918354us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:04.336211 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1064616320}
I20260709 13:03:04.337438  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.342223  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59800 (request call id 15) took 1921 ms. Trace:
I20260709 13:03:04.342635  2315 rpcz_store.cc:276] 0709 13:03:02.421031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.421211 (+   180us) service_pool.cc:224] Handling call
0709 13:03:04.342022 (+1920811us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b54006004d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfd7853 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.342126 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26314112}
I20260709 13:03:04.312449  6181 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.344795  2315 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:04.345156  6237 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.108s	user 0.001s	sys 0.001s
I20260709 13:03:04.345907  6181 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.346352  2312 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:04.346720  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.347321  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.348587  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.336215  1666 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.349888  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.350327  1666 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.350482  6069 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.351061  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.351631  1666 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:04.352245  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.352948  2315 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:04.353729  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.354616  6102 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.355108  6102 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.357162  6102 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.330416  6135 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.359195  6135 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:04.359405  6184 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.360621   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 13:03:04.362408  6238 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:03:04.363339   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:04.364540  6033 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:04.365065  6229 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.366398  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.366681  1665 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:04.334789  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.333957  1587 rpcz_store.cc:276] 0709 13:03:02.604888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.779358 (+1174470us) service_pool.cc:224] Handling call
0709 13:03:03.815131 (+ 35773us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:03.967727 (+152596us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:03:03.967759 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:03.982624 (+ 14865us) log.cc:844] Serialized 52 byte log entry
0709 13:03:04.232336 (+249712us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:03:04.232379 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.232459 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:04.333307 (+100848us) raft_consensus.cc:1685] finished
0709 13:03:04.333328 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:04.333474 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26402,"raft.queue_time_us":68320,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":5165,"thread_start_us":344737,"threads_started":2,"wal-append.queue_time_us":281784}
I20260709 13:03:04.324059  1741 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:04.373106  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.376682  2309 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:03:04.376703  2310 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:04.380465  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.380947  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 760) took 1218 ms. Trace:
I20260709 13:03:04.381104  2310 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:04.381441  2310 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:04.383808  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:04.384001  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.385594  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.385704  6232 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:04.386449  6232 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:04.387468  6232 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:04.388079  6232 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:04.389015  5881 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:04.389439  5881 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.390419  5881 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.391808  5881 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:04.393887  6151 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:04.395674  6151 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:04.396056  6151 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.381350  2309 rpcz_store.cc:276] 0709 13:03:03.162265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.162922 (+   657us) service_pool.cc:224] Handling call
0709 13:03:04.345813 (+1182891us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:03:04.370085 (+ 24272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.378776 (+  8691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.380888 (+  2112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4844288}
I20260709 13:03:04.401276  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:03:04.406862  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.407768  1587 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:04.326401  6225 consensus_queue.cc:1243] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:04.323678  5792 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:04.419221  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46021.3us
I20260709 13:03:04.320602  6018 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:04.376629  6104 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.422377  2310 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.425096  1666 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:03:04.441154  1666 pending_rounds.cc:85] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:03:04.442673  1666 pending_rounds.cc:107] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:03:04.443992  1666 raft_consensus.cc:2889] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:04.451588  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:04.453167  2310 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:04.454538  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.454382  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.467517  1734 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:04.456451  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.397002  6151 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.473120  2311 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:03:04.474490  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.487946  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 679) took 1804 ms. Trace:
I20260709 13:03:04.495215  6127 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:04.495966  6248 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:03:04.498442  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.499235  6172 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:04.500334  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:04.422703  6207 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.000s
I20260709 13:03:04.508769  1734 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [10.7-11.8]
W20260709 13:03:04.426505  6182 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.528s	user 0.012s	sys 0.000s
I20260709 13:03:04.518174  2310 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:04.505788  6207 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.455854  6018 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:04.522578  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 137524us
I20260709 13:03:04.470337  6094 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 13:03:04.526892  1586 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:03:04.527033  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 70707.4us
I20260709 13:03:04.471648  6184 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.485687  6145 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.501945  6138 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:04.531428  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.536484  1734 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 13:03:04.539644  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:04.526436  6187 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.045s	sys 0.042s
I20260709 13:03:04.543787  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.545219  1739 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:04.526723  6066 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:04.529011  6068 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:04.530453  6151 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:04.529731  6184 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.529831  6102 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.554356  6022 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.556149  6207 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.469208  6225 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.073s
I20260709 13:03:04.558749  6226 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.565951  6018 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.526167  6182 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.502424  6104 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.573254  1734 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:04.545485  6187 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.045s	sys 0.042s
I20260709 13:03:04.568758  6154 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:04.547909  6145 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.546607  6069 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:04.545295  1587 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:04.548414  6094 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:04.595858  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:04.595852  1734 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:04.596357  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.582386  6162 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:03:04.603688  1734 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 13:03:04.591885  6068 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 13:03:04.610163  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.611012  1587 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:04.612469  1734 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 13:03:04.613718  1741 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:04.584448  6102 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.619371  1734 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:03:04.621829  6139 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:04.622228  6139 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:04.623178  6139 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.627069  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 684) took 1381 ms. Trace:
I20260709 13:03:04.627655  1740 rpcz_store.cc:276] 0709 13:03:03.245218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.245816 (+   598us) service_pool.cc:224] Handling call
0709 13:03:04.626781 (+1380965us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b5400284e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.626988 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1892072320}
I20260709 13:03:04.630900  6182 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.631767  6213 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.632938  2309 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:03:04.638492  6152 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:04.601442  6143 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:04.611375  6184 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:04.642896  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.642843  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.644991  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.645855  1740 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:04.648854  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.649981  1740 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:04.644398  1662 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 3
I20260709 13:03:04.651561  1662 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 9.4
I20260709 13:03:04.649889  1587 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:04.651830  1662 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:04.624167  6139 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:04.662778  1740 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:04.662494  6068 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:04.596755  6154 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.666657  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.666332  6195 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.667433  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.670130  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.671416  6204 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 13:03:04.674268  6186 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:04.674722  6213 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:03:04.674767  6104 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:04.676610  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1941) took 1588 ms (client timeout 1662 ms). Trace:
W20260709 13:03:04.719264  1742 rpcz_store.cc:269] 0709 13:03:03.088084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.095739 (+  7655us) service_pool.cc:224] Handling call
0709 13:03:04.676416 (+1580677us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b5400284954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.676544 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2058931072}
I20260709 13:03:04.677754  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.720966  1663 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:04.678678  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.678921  1739 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [13.10-14.11]
I20260709 13:03:04.679134  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2708) took 1581 ms. Trace:
I20260709 13:03:04.723587  1585 rpcz_store.cc:276] 0709 13:03:03.098689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.247128 (+148439us) service_pool.cc:224] Handling call
0709 13:03:04.678782 (+1431654us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54002184d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.678944 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1592540928}
I20260709 13:03:04.680114  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2712) took 1198 ms. Trace:
I20260709 13:03:04.724938  1593 rpcz_store.cc:276] 0709 13:03:03.481330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.512622 (+ 31292us) service_pool.cc:224] Handling call
0709 13:03:04.679917 (+1167295us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b540033aed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.680054 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118026624}
I20260709 13:03:04.726256  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.727254  1593 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:04.681352  1733 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:04.682197  1591 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 13:03:04.488595  1738 rpcz_store.cc:276] 0709 13:03:02.683315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.683457 (+   142us) service_pool.cc:224] Handling call
0709 13:03:02.683822 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:02.684084 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:02.730157 (+ 46073us) log.cc:844] Serialized 14 byte log entry
0709 13:03:03.483862 (+753705us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b180012c8e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:03:04.370046 (+886184us) spinlock_profiling.cc:243] Waited 1.49 ms on lock 0x7b180012c8e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:03:04.485751 (+115705us) log.cc:844] Serialized 14 byte log entry
0709 13:03:04.487433 (+  1682us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:04.487516 (+    83us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:04.487585 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.487688 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:04.487870 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":462945,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":4952,"raft.queue_time_us":25837,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":8682,"spinlock_wait_cycles":12581504,"thread_start_us":164265,"threads_started":2,"wal-append.queue_time_us":951351}
I20260709 13:03:04.685846  1590 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:03:04.690922  1660 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:03:04.692431  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.694988  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:04.701823  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:04.620194  6161 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:04.602686  6255 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 13:03:04.599015  6145 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:04.690145  6068 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:04.708520  6227 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.752s	user 0.011s	sys 0.001s
W20260709 13:03:04.708810  6241 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.507s	user 0.004s	sys 0.004s
I20260709 13:03:04.710942  6102 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.663744  6143 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:04.733927  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:04.781814  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48025us
I20260709 13:03:04.736130  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:04.783079  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:04.734642  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:04.784693  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.786324  2313 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:04.788995  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.789897  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.790525  1663 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:04.791671  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:04.792652  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 216855us
I20260709 13:03:04.797107  1666 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:04.647262  6094 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.091s
I20260709 13:03:04.801651  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.801903  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:04.802570  1660 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:03:04.813995  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 231618us
I20260709 13:03:04.815331  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:04.818774  1668 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:04.735255  6161 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:04.722455  6063 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:04.823629  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.836303  1587 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:04.738019  5898 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:04.836650  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:04.870419  1593 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 7 FOLLOWER]: Advancing to term 8
W20260709 13:03:04.861299  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46094.6us
W20260709 13:03:04.884068  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:04.718124  6186 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:04.795017  6172 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:04.891073  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.909837  5792 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:04.909479  6022 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:04.972178  1739 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:03:04.918848  6080 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 13:03:04.976011  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.992100  1740 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:04.993570  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.994810  1740 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:04.923879  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:04.924044  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2709) took 1801 ms (client timeout 1913 ms). Trace:
W20260709 13:03:04.996378  1592 rpcz_store.cc:269] 0709 13:03:03.122926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.169540 (+ 46614us) service_pool.cc:224] Handling call
0709 13:03:04.923737 (+1754197us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b540031f454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.923949 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1579100928}
I20260709 13:03:04.997489  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.924671  6143 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:04.925407  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 898) took 1941 ms (client timeout 2051 ms). Trace:
I20260709 13:03:04.928416  6109 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:04.934917  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2653) took 2102 ms. Trace:
I20260709 13:03:05.005121  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:04.933482  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:04.937803  6235 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 13:03:04.940140  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2625) took 1324 ms. Trace:
I20260709 13:03:04.909680  5898 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:04.945892  6162 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 13:03:04.946888  6104 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:04.950809  6186 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.953380  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:05.048353  1733 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:04.955480  1585 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:03:04.963824  1738 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:04.975289  1593 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 8.
I20260709 13:03:05.051106  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:05.051848  1593 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:04.975667  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.976533  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:04.909657  6258 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 13:03:04.966547  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:04.988943  1706 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 2.051s (SENT) [suppressed 13 similar messages]
W20260709 13:03:05.059618  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 70759.5us
I20260709 13:03:04.922827  6145 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:05.003387  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.005057  1592 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:05.005878  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1917) took 1422 ms. Trace:
I20260709 13:03:05.006255  1589 rpcz_store.cc:276] 0709 13:03:02.832055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.966382 (+1134327us) service_pool.cc:224] Handling call
0709 13:03:03.966695 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.018761 (+ 52066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.934526 (+915765us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:04.934831 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108765440}
I20260709 13:03:05.063951  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:05.063105  1661 rpcz_store.cc:276] 0709 13:03:03.583395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.608532 (+ 25137us) service_pool.cc:224] Handling call
0709 13:03:03.624978 (+ 16446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.770899 (+145921us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.770940 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:03.782258 (+ 11318us) log.cc:844] Serialized 26 byte log entry
0709 13:03:03.894737 (+112479us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:03.894797 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:03.894892 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:04.708017 (+813125us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:04.963476 (+255459us) raft_consensus.cc:1685] finished
0709 13:03:04.963498 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.005790 (+ 42292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1153,"raft.queue_time_us":81512,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":22270,"spinlock_wait_cycles":5270272,"thread_start_us":203389,"threads_started":2,"wal-append.queue_time_us":244435}
W20260709 13:03:04.935606  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48376.2us
I20260709 13:03:04.788352  6227 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:05.005699  2318 rpcz_store.cc:269] 0709 13:03:02.983908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.992889 (+  8981us) service_pool.cc:224] Handling call
0709 13:03:04.925088 (+1932199us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54000c49d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:04.925276 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301086464}
I20260709 13:03:05.024423  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.016801  1741 rpcz_store.cc:276] 0709 13:03:03.615729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.405582 (+789853us) service_pool.cc:224] Handling call
0709 13:03:04.423112 (+ 17530us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:04.614224 (+191112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.939850 (+325626us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:04.940067 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437927808}
I20260709 13:03:05.067152  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:04.890141  6225 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:05.012164  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:05.041519  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:04.999467  6172 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:05.053081  1734 raft_consensus.cc:2376] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12. Re-sending same reply.
I20260709 13:03:05.077984  1740 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:05.060292  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:05.062911  1742 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:05.068015  1589 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 13:03:05.067940  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43613.6us
I20260709 13:03:05.073401  1737 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:03:04.953869  6094 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:05.015437  6109 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:05.077031  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:05.083169  1585 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:05.077986  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.084445  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.084911  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.085745  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:05.088001  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.089042  1740 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:03:05.091401  1736 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:05.091744  1741 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:05.040187  6162 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.047820  6206 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:05.094596  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2700) took 2010 ms (client timeout 2166 ms). Trace:
W20260709 13:03:05.107887  1588 rpcz_store.cc:269] 0709 13:03:03.083966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.174061 (+ 90095us) service_pool.cc:224] Handling call
0709 13:03:03.890559 (+716498us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:03:05.094377 (+1203818us) spinlock_profiling.cc:243] Waited 28.9 ms on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.094519 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108953856}
I20260709 13:03:05.109340  1588 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:05.110066  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2705) took 2024 ms. Trace:
I20260709 13:03:05.110450  1588 rpcz_store.cc:276] 0709 13:03:03.085316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.108842 (+2023526us) service_pool.cc:224] Handling call
0709 13:03:05.109172 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.109779 (+   607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.109997 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.112754  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:05.113616  1741 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.095742  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.105460  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:05.119067  1742 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:05.087396  6135 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.728s	user 0.014s	sys 0.000s
I20260709 13:03:05.047482  6104 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.103523  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:05.160553  5881 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.768s	user 0.009s	sys 0.006s
I20260709 13:03:05.004894  6022 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:04.991017  5792 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.097168  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:05.199331  1740 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
W20260709 13:03:05.104475  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72704.2us
I20260709 13:03:05.139635  6206 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.179688  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2633) took 1459 ms. Trace:
I20260709 13:03:05.205958  2316 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:05.182524  6265 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:05.183810  5898 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.185256  1589 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:05.119632  6263 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:03:05.185644  6186 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:05.139469  6141 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:05.082459  6225 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.186928  6195 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:05.188328  6104 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.189800  1587 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 13. Ops: []
I20260709 13:03:05.144241  6207 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.218397  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2704) took 2133 ms. Trace:
I20260709 13:03:05.219089  1587 rpcz_store.cc:276] 0709 13:03:03.085068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.090950 (+2005882us) service_pool.cc:224] Handling call
0709 13:03:05.189595 (+ 98645us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.217742 (+ 28147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.218297 (+   555us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.190222  6227 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:05.223908  6139 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.000s
I20260709 13:03:05.224496  5792 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:05.224654  2311 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:03:05.077661  6204 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:05.190533  1588 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:05.226325  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2710) took 2087 ms. Trace:
I20260709 13:03:05.226729  1588 rpcz_store.cc:276] 0709 13:03:03.138978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.111313 (+1972335us) service_pool.cc:224] Handling call
0709 13:03:05.190282 (+ 78969us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.226014 (+ 35732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.226257 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.102200  6033 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 13:03:05.190747  6109 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.231815  6109 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:05.232117  1584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2635) took 2629 ms (client timeout 2168 ms). Trace:
I20260709 13:03:05.232957  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2656) took 2396 ms. Trace:
W20260709 13:03:05.232618  1584 rpcz_store.cc:269] 0709 13:03:02.602500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.747380 (+144880us) service_pool.cc:224] Handling call
0709 13:03:05.231826 (+2484446us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b5400442d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.232001 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1992697216}
I20260709 13:03:05.233465  1590 rpcz_store.cc:276] 0709 13:03:02.836141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.041974 (+1205833us) service_pool.cc:224] Handling call
0709 13:03:04.685593 (+643619us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.729581 (+ 43988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.232724 (+503143us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b54002184d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.232888 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452914944}
I20260709 13:03:05.235440  1590 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 10. (term mismatch)
I20260709 13:03:05.191066  6162 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.236656  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2641) took 2632 ms. Trace:
I20260709 13:03:05.191540  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.193670  6145 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:05.237139  1591 rpcz_store.cc:276] 0709 13:03:02.604192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:03.621902 (+1017710us) service_pool.cc:224] Handling call
0709 13:03:04.681913 (+1060011us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:04.729139 (+ 47226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.236424 (+507285us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540033aed0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.236589 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2318630784}
I20260709 13:03:05.237866  6145 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.239455  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 626) took 1177 ms. Trace:
I20260709 13:03:05.240595  6145 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:05.121878  6152 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:03:05.164767  6172 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.244813  6225 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:05.246158  6104 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:05.246758  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.247793  1668 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:05.220292  6141 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.249786  5881 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:05.250348  5881 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.251521  5881 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.248822  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.254241  1668 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.197758  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:05.258185  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60531.6us
I20260709 13:03:04.852362  6068 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.130090  1741 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:05.259390  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.203775  1666 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:03:05.260304  1666 pending_rounds.cc:85] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 6
I20260709 13:03:05.260555  1666 pending_rounds.cc:107] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:05.260808  1666 raft_consensus.cc:2889] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.178989  5981 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.207687  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2657) took 1786 ms. Trace:
I20260709 13:03:05.253193  5881 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:05.204809  1735 rpcz_store.cc:276] 0709 13:03:03.720737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.568188 (+847451us) service_pool.cc:224] Handling call
0709 13:03:04.568619 (+   431us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:04.569106 (+   487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:04.569139 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:04.569816 (+   677us) log.cc:844] Serialized 52 byte log entry
0709 13:03:04.819474 (+249658us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:04.819591 (+   117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:04.819718 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:05.179401 (+359683us) raft_consensus.cc:1685] finished
0709 13:03:05.179426 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.179623 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":249090,"threads_started":1,"wal-append.queue_time_us":492541}
I20260709 13:03:05.262876  1589 rpcz_store.cc:276] 0709 13:03:03.421704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.141551 (+1719847us) service_pool.cc:224] Handling call
0709 13:03:05.141871 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.207343 (+ 65472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.207622 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.270234  1664 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:05.213809  6022 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:05.214125  2315 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 13:03:05.178989  6151 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.597s	user 0.007s	sys 0.003s
I20260709 13:03:05.262789  6213 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:05.282079  6274 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 13:03:05.282272  6213 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.282712  6213 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.222484  1587 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:05.276626  6139 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.255204  6264 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:05.197052  6194 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:03:05.208590  6232 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.285171  6264 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.284988  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2658) took 1814 ms. Trace:
I20260709 13:03:05.285518  6232 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.286276  1587 rpcz_store.cc:276] 0709 13:03:03.470746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.220091 (+1749345us) service_pool.cc:224] Handling call
0709 13:03:05.222267 (+  2176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.284678 (+ 62411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.284926 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.287528  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.287876  6232 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:05.288059  6057 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:05.288463  6057 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.288833  1587 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:05.291337  1587 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:05.292990  1587 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 7. Current term is 8. Ops: []
I20260709 13:03:05.272877  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.276209  1742 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20260709 13:03:05.277959  1590 pending_rounds.cc:85] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 9
I20260709 13:03:05.239895  1591 rpcz_store.cc:276] 0709 13:03:04.064087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.238406 (+1174319us) service_pool.cc:224] Handling call
0709 13:03:05.238771 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.239025 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.239062 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:05.239124 (+    62us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:05.239170 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.239232 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.239388 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.293843  1661 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:03:05.294889  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.271812  6087 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 13:03:05.284965  6139 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.275100  6195 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 13:03:05.288760  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.213356  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 899) took 2224 ms. Trace:
W20260709 13:03:05.293056  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.216917  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:05.305001  1591 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.9-14.9]
W20260709 13:03:05.305326  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 88506.3us
W20260709 13:03:05.205067  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.132526  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.274290  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.309693  1664 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:05.304137  2317 rpcz_store.cc:276] 0709 13:03:02.989367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.989553 (+   186us) service_pool.cc:224] Handling call
0709 13:03:05.213090 (+2223537us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.213242 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":642363136}
I20260709 13:03:05.299089  1587 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260709 13:03:05.301290  6264 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.293883  1590 pending_rounds.cc:107] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 10.10
I20260709 13:03:05.318409  1590 raft_consensus.cc:2889] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.319208  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 625) took 1257 ms. Trace:
I20260709 13:03:05.319541  1590 rpcz_store.cc:276] 0709 13:03:04.061995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.234617 (+1172622us) service_pool.cc:224] Handling call
0709 13:03:05.234976 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.318941 (+ 83965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.319152 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:05.312047  1741 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:05.329030  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.330070  1666 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:05.372287  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 578693us
I20260709 13:03:05.376686  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 902) took 2345 ms. Trace:
I20260709 13:03:05.264719  6207 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.380070  1590 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:05.398815  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:05.325773  6087 consensus_queue.cc:1243] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:05.317922  6096 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:05.449882  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51358.8us
I20260709 13:03:05.450955  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:05.293686  5981 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.434605  1666 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:05.377204  2314 rpcz_store.cc:276] 0709 13:03:02.997319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.997469 (+   150us) service_pool.cc:224] Handling call
0709 13:03:03.019722 (+ 22253us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:03.099208 (+ 79486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:03.099230 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:03.117948 (+ 18718us) log.cc:844] Serialized 26 byte log entry
0709 13:03:04.930544 (+1812596us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b18003c5440. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:05.126935 (+196391us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:05.126985 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.127111 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:05.342502 (+215391us) raft_consensus.cc:1685] finished
0709 13:03:05.342529 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.342837 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54838,"raft.queue_time_us":96101,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":19474,"spinlock_wait_cycles":11227392,"thread_start_us":275184,"threads_started":2,"wal-append.queue_time_us":2222583}
I20260709 13:03:05.466493  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.472841  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:05.479851  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.007s	sys 0.019s
W20260709 13:03:05.480499  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.007s	sys 0.019s
I20260709 13:03:05.491005  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:05.289319  6057 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.496002  1666 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260709 13:03:05.448912  6141 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:05.491986  1590 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:05.455689  6096 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.497879  6224 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 13:03:05.511654  6224 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:05.512712  6224 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:05.513039  6224 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.498025  6194 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:05.513957  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:05.514183  6194 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:05.514746  2317 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:05.515848  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.515722  6194 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.469468  6069 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:05.515944  6154 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:05.498945  6138 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.956s	user 0.013s	sys 0.000s
I20260709 13:03:05.517976  6096 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:05.502362  6151 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 13:03:05.506086  6263 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 13:03:05.533118  6224 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.549543  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.556453  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.557154  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:05.559701  1586 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:05.535235  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:05.485457  6094 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 13:03:05.563082  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:05.517611  6149 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:03:05.567147  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.535969  6057 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.506975  6225 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 13:03:05.555121  6227 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.561297  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.562784  6137 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.603731  6022 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:05.605098  6022 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:05.605557  6022 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:05.606715  6022 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.608605  6022 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:05.611395  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:05.603788  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:05.612629  6154 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:05.613325  6154 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:05.613883  6154 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.614207  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:05.618681  2316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 768) took 1272 ms (client timeout 1648 ms). Trace:
W20260709 13:03:05.619215  2316 rpcz_store.cc:269] 0709 13:03:04.346626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.349649 (+  3023us) service_pool.cc:224] Handling call
0709 13:03:05.618467 (+1268818us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.618612 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1907872512}
I20260709 13:03:05.601054  2317 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:05.571812  6224 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:05.622974  1742 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:03:05.624194  1742 pending_rounds.cc:85] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 8
I20260709 13:03:05.624467  1742 pending_rounds.cc:107] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:03:05.624711  1742 raft_consensus.cc:2889] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:05.622867  1590 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 5. (index mismatch)
I20260709 13:03:05.648763  1590 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 13:03:05.657461  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2664) took 1300 ms. Trace:
I20260709 13:03:05.657980  1586 rpcz_store.cc:276] 0709 13:03:04.356978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.539249 (+182271us) service_pool.cc:224] Handling call
0709 13:03:05.657256 (+1118007us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400443754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.657390 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343854336}
I20260709 13:03:05.654560  6063 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:05.662158  6104 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:05.663663  6104 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:05.663156  1666 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:05.665366  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.667497  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:05.647706  6161 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.743s	user 0.006s	sys 0.007s
I20260709 13:03:05.677414  5792 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:03:05.679369  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.016s	sys 0.027s
W20260709 13:03:05.679828  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.016s	sys 0.027s
I20260709 13:03:05.687285  1590 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:05.691929  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.692741  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:05.676429  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.702438  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:05.641942  6227 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:05.710363  1593 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.017s	sys 0.017s
W20260709 13:03:05.726630  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.735603  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.736354  1593 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.017s	sys 0.017s
I20260709 13:03:05.736081  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.736980  2311 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:03:05.739128  1659 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.005s	sys 0.022s
W20260709 13:03:05.739859  1659 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.005s	sys 0.022s
I20260709 13:03:05.740826  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.672812  6097 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:05.676543  6063 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:05.744019  1665 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.008s	sys 0.020s
W20260709 13:03:05.782490  1665 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.008s	sys 0.020s
W20260709 13:03:05.783833  1668 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.526s	user 0.008s	sys 0.004s
I20260709 13:03:05.784873  1668 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 13:03:05.785966  1660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.006s	sys 0.017s
W20260709 13:03:05.786352  1660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.006s	sys 0.018s
W20260709 13:03:05.748385  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 66744us
I20260709 13:03:05.749668  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1934) took 1105 ms. Trace:
I20260709 13:03:05.766392  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.767264  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:05.789279  2317 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:05.664986  6104 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.772429  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:05.772433  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.015s	sys 0.015s
I20260709 13:03:05.643628  6137 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:05.794306  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.026s	sys 0.012s
W20260709 13:03:05.805663  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.026s	sys 0.012s
I20260709 13:03:05.711133  6033 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:05.793886  1662 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.011s	sys 0.030s
I20260709 13:03:05.805560  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:05.802977  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.823124  1662 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.011s	sys 0.030s
W20260709 13:03:05.794505  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.015s	sys 0.016s
I20260709 13:03:05.803367  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:05.824645  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 683769us
W20260709 13:03:05.825394  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:05.825618  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.788559  1659 rpcz_store.cc:276] 0709 13:03:04.644269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.644682 (+   413us) service_pool.cc:224] Handling call
0709 13:03:04.675426 (+ 30744us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:04.675875 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:04.675909 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:04.676755 (+   846us) log.cc:844] Serialized 78 byte log entry
0709 13:03:04.976009 (+299254us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:05.748659 (+772650us) log.cc:844] Serialized 14 byte log entry
0709 13:03:05.749058 (+   399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.749243 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:05.749304 (+    61us) raft_consensus.cc:1685] finished
0709 13:03:05.749325 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:05.749586 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3699,"prepare.queue_time_us":581933,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":27538,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":104,"thread_start_us":991181,"threads_started":2,"wal-append.queue_time_us":509779}
I20260709 13:03:05.835033  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.835820  1668 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
W20260709 13:03:05.835525  1736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.021s	sys 0.014s
W20260709 13:03:05.837044  1736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.021s	sys 0.014s
I20260709 13:03:05.680027  6161 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:05.837968  1733 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.012s	sys 0.011s
I20260709 13:03:05.767351  6235 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:05.843042  1733 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.012s	sys 0.011s
W20260709 13:03:05.844635  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:05.834369  6213 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.551s	user 0.014s	sys 0.000s
I20260709 13:03:05.794987  6187 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:05.826081  6097 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:05.834062  6033 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:05.824558  6063 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:05.842113  6230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.016s	sys 0.008s
W20260709 13:03:05.845412  6032 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.008s	sys 0.017s
W20260709 13:03:05.854656  1667 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.014s	sys 0.015s
W20260709 13:03:05.864001  1667 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.014s	sys 0.015s
I20260709 13:03:05.835501  6207 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:05.861423  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.003s	sys 0.021s
W20260709 13:03:05.864658  1589 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.009s	sys 0.016s
W20260709 13:03:05.882870  1589 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.009s	sys 0.016s
W20260709 13:03:05.866016  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.003s	sys 0.021s
I20260709 13:03:05.849342  6264 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.845764  5981 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:05.894618  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.895424  1659 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:03:05.897557  1584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.004s	sys 0.042s
W20260709 13:03:05.898126  1584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.005s	sys 0.042s
I20260709 13:03:05.911092  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 773) took 1563 ms. Trace:
I20260709 13:03:05.792982  6281 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 13:03:05.911860  2315 rpcz_store.cc:276] 0709 13:03:04.348127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.354479 (+  6352us) service_pool.cc:224] Handling call
0709 13:03:05.213792 (+859313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.275520 (+ 61728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.770566 (+495046us) spinlock_profiling.cc:243] Waited 51 ms on lock 0x7b54006a2320. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0a02 00007fcdedfdcdd6 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:03:05.893442 (+122876us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b54006a1b50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:05.910990 (+ 17548us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2062238720}
W20260709 13:03:05.882148  6032 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.009s	sys 0.017s
W20260709 13:03:05.881946  6230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.016s	sys 0.008s
I20260709 13:03:05.865542  6104 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:05.865087  6033 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:05.880072  5792 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:05.952143  1585 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:05.864517  6279 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:05.932741  6278 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:05.926054  6274 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:03:05.859570  6213 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.970678  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.974759  1659 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:05.976416  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:05.973860  6230 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:05.983206  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:05.973143  6154 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:05.987572  6154 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:05.987937  6096 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:05.996678  6096 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:05.928690  6152 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 13:03:05.964329  6279 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:06.016019  1664 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.014s	sys 0.047s
W20260709 13:03:06.017858  1664 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.015s	sys 0.047s
I20260709 13:03:05.967028  6278 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:06.021703  2314 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:05.950997  6018 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
W20260709 13:03:06.029125  1735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.023s	sys 0.012s
W20260709 13:03:06.029738  1735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.023s	sys 0.012s
W20260709 13:03:06.016521  1559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.015s	sys 0.020s
W20260709 13:03:06.033735  1559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.015s	sys 0.020s
W20260709 13:03:06.017812  6186 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.801s	user 0.014s	sys 0.000s
I20260709 13:03:06.012511  6230 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.076356  1661 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:06.078537  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:06.017524  6279 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.094693  1587 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:06.081995  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.020s	sys 0.028s
W20260709 13:03:06.095224  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.020s	sys 0.028s
I20260709 13:03:06.094280  1590 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:06.096887  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.097905  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.099485  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:06.073081  6151 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.010s	sys 0.024s
I20260709 13:03:06.024132  6278 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:06.102762  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.110841  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.111719  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:06.112583  1590 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:06.079932  6292 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.002s
W20260709 13:03:06.082377  5898 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.003s	sys 0.037s
I20260709 13:03:06.101985  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.115787  1591 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:06.117798  1587 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:06.118317  1587 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:06.077853  6143 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:06.123458  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:06.125061  1741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.015s	sys 0.026s
I20260709 13:03:06.119241  1587 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.180344  1587 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:06.079841  6096 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.079814  6109 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.093878  6227 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:06.113987  6151 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.011s	sys 0.024s
I20260709 13:03:06.167968  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:06.101624  6263 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.008s	sys 0.036s
I20260709 13:03:06.016783  6063 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:06.121879  5898 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.004s	sys 0.038s
W20260709 13:03:06.229080  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.247455  1590 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:06.106468  6138 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:06.179345  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.016s	sys 0.056s
W20260709 13:03:06.181488  1741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.015s	sys 0.031s
W20260709 13:03:06.270565  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.016s	sys 0.056s
I20260709 13:03:06.205566  1591 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:06.124961  6186 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:06.174983  6170 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.005s	sys 0.016s
W20260709 13:03:06.212375  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89015.3us
W20260709 13:03:06.177078  6094 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.011s
I20260709 13:03:06.256654  1587 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 13:03:06.283125  1587 pending_rounds.cc:85] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:06.283463  1587 pending_rounds.cc:107] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 13:03:06.283491  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1933) took 1640 ms. Trace:
I20260709 13:03:06.283823  1587 raft_consensus.cc:2889] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:06.210340  6172 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.012s	sys 0.041s
W20260709 13:03:06.169782  6225 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.007s	sys 0.012s
I20260709 13:03:06.126048  6032 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.172364  6161 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:06.283049  6094 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.009s	sys 0.012s
W20260709 13:03:06.273759  1706 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.022s	sys 0.041s
I20260709 13:03:06.202620  6096 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.273139  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2697) took 3189 ms. Trace:
W20260709 13:03:06.215728  6263 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.009s	sys 0.036s
W20260709 13:03:06.262518  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.284000  1662 rpcz_store.cc:276] 0709 13:03:04.643190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.643841 (+   651us) service_pool.cc:224] Handling call
0709 13:03:04.644151 (+   310us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:03:04.712155 (+ 68004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:04.712187 (+    32us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 13:03:04.712983 (+   796us) log.cc:844] Serialized 104 byte log entry
0709 13:03:05.131918 (+418935us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:05.840927 (+709009us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.272731 (+431804us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.274813 (+  2082us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.283086 (+  8273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.283173 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.283223 (+    50us) raft_consensus.cc:1685] finished
0709 13:03:06.283234 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.283428 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26427,"prepare.queue_time_us":427339,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":81,"raft.queue_time_us":2363,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":18873,"spinlock_wait_cycles":271232,"thread_start_us":1468281,"threads_started":3,"wal-append.queue_time_us":499181}
W20260709 13:03:06.294564  6225 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.008s	sys 0.014s
W20260709 13:03:06.333748  1660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.020s	sys 0.040s
W20260709 13:03:06.334146  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.020s	sys 0.040s
I20260709 13:03:06.334932  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 697) took 1640 ms. Trace:
I20260709 13:03:06.217923  6227 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:06.278071  6170 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.006s	sys 0.018s
W20260709 13:03:06.336748  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.019s	sys 0.036s
W20260709 13:03:06.204945  6182 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.023s	sys 0.057s
I20260709 13:03:06.273959  6186 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:06.288846  6172 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.014s	sys 0.044s
W20260709 13:03:06.300736  1706 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.022s	sys 0.041s
W20260709 13:03:06.350898  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 580042us
I20260709 13:03:06.352845  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:06.175179  6069 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.002s	sys 0.020s
I20260709 13:03:06.304006  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 630) took 2030 ms. Trace:
I20260709 13:03:06.204474  6143 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.207733  6109 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.318245  6161 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:03:06.319998  6285 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
W20260709 13:03:06.320883  6290 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
W20260709 13:03:06.321684  6284 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
W20260709 13:03:06.322471  6282 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
W20260709 13:03:06.326740  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.021s	sys 0.051s
W20260709 13:03:06.424041  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.021s	sys 0.052s
W20260709 13:03:06.424432  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 722129us
W20260709 13:03:06.327308  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.328290  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:06.327026  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2711) took 2850 ms. Trace:
W20260709 13:03:06.337097  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.019s	sys 0.036s
W20260709 13:03:06.428612  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 603099us
I20260709 13:03:06.343341  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2534) took 1552 ms. Trace:
I20260709 13:03:06.324326  1592 rpcz_store.cc:276] 0709 13:03:03.083238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.081022 (+1997784us) service_pool.cc:224] Handling call
0709 13:03:05.081415 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.081879 (+   464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.081933 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.120369 (+ 38436us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.242480 (+122111us) spinlock_profiling.cc:243] Waited 7.52 ms on lock 0x7b180043ae60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:05.834352 (+591872us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:05.834407 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.834490 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.243032 (+408542us) raft_consensus.cc:1685] finished
0709 13:03:06.243067 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.273032 (+ 29965us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27082,"raft.queue_time_us":11064,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":6274,"spinlock_wait_cycles":17307392,"thread_start_us":563778,"threads_started":1,"wal-append.queue_time_us":859204}
I20260709 13:03:06.354835  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:06.355423  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.012s	sys 0.002s
W20260709 13:03:06.517871  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.012s	sys 0.002s
W20260709 13:03:06.518421  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 983327us
I20260709 13:03:06.335314  1660 rpcz_store.cc:276] 0709 13:03:04.694472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.803202 (+108730us) service_pool.cc:224] Handling call
0709 13:03:04.888411 (+ 85209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:04.888772 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:05.786886 (+898114us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.334543 (+547657us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.334604 (+    61us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:06.334654 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.334725 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.334871 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":668876,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"thread_start_us":1354071,"threads_started":2}
I20260709 13:03:06.371629  1589 rpcz_store.cc:276] 0709 13:03:04.273914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.269998 (+996084us) service_pool.cc:224] Handling call
0709 13:03:05.270431 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.270813 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.270847 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.303529 (+ 32682us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.883452 (+579923us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:05.883502 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.883627 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.303737 (+420110us) raft_consensus.cc:1685] finished
0709 13:03:06.303763 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.303933 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1974,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":13002,"thread_start_us":565235,"threads_started":1,"wal-append.queue_time_us":846920}
W20260709 13:03:06.341387  6182 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.023s	sys 0.058s
W20260709 13:03:06.342788  6280 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.378s	user 0.001s	sys 0.001s
W20260709 13:03:06.371001  6069 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.811s	user 0.002s	sys 0.021s
I20260709 13:03:06.329480  6213 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:06.533792  6138 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:06.558211  6298 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:03:06.378667  5792 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 13:03:06.558943  5792 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:06.558645  6138 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.559455  5792 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:06.560369  1660 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:06.561084  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1939) took 1246 ms. Trace:
I20260709 13:03:06.429854  1667 rpcz_store.cc:276] 0709 13:03:04.790535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.015764 (+225229us) service_pool.cc:224] Handling call
0709 13:03:05.016096 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.016334 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:05.274798 (+258464us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.342401 (+1067603us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.342940 (+   539us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:06.343003 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.343097 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.343259 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31522,"prepare.queue_time_us":541857,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":489795,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":15258,"thread_start_us":1268860,"threads_started":3,"wal-append.queue_time_us":942868}
I20260709 13:03:06.561403  1660 rpcz_store.cc:276] 0709 13:03:05.314601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.546110 (+1231509us) service_pool.cc:224] Handling call
0709 13:03:06.560170 (+ 14060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.560776 (+   606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.561021 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:06.562300  6096 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:06.562664  1667 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:06.562922  6096 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.426934  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:06.471202  1586 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:06.300377  6032 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.563283  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 701) took 1015 ms. Trace:
I20260709 13:03:06.564713  1667 rpcz_store.cc:276] 0709 13:03:05.547695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.561993 (+1014298us) service_pool.cc:224] Handling call
0709 13:03:06.562343 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.563028 (+   685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.563229 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:06.483206  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2545) took 1299 ms. Trace:
I20260709 13:03:06.413091  6230 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:06.414652  6048 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.006s	sys 0.019s
I20260709 13:03:06.388579  6302 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:06.476169  6127 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.012s	sys 0.012s
W20260709 13:03:06.569815  6048 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.006s	sys 0.020s
I20260709 13:03:06.569931  6230 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:06.570071  6127 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.012s	sys 0.012s
W20260709 13:03:06.482223  6274 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:03:06.570314  6302 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:06.427793  6094 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:06.571349  6302 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:06.496739  5881 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.222s	user 0.019s	sys 0.004s
I20260709 13:03:06.509058  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 624) took 2447 ms. Trace:
I20260709 13:03:06.428395  1588 rpcz_store.cc:276] 0709 13:03:03.477035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.227682 (+1750647us) service_pool.cc:224] Handling call
0709 13:03:05.228188 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.228704 (+   516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.228754 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.229451 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.886791 (+657340us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:05.886856 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.886954 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.326768 (+439814us) raft_consensus.cc:1685] finished
0709 13:03:06.326795 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.326963 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7870,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":203,"thread_start_us":586260,"threads_started":1,"wal-append.queue_time_us":908987}
W20260709 13:03:06.523772  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53803.4us
I20260709 13:03:06.524617  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2530) took 1837 ms. Trace:
I20260709 13:03:06.486380  6263 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:06.531980  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 177283us
W20260709 13:03:06.532871  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 254154us
W20260709 13:03:06.533458  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.697s	user 0.008s	sys 0.008s
W20260709 13:03:06.539579  6294 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.267s	user 0.000s	sys 0.001s
I20260709 13:03:06.471732  5981 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
W20260709 13:03:06.515286  6293 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.234s	user 0.001s	sys 0.001s
I20260709 13:03:06.557829  6161 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:06.352065  6186 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:06.570480  6230 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:06.465938  6310 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.572116  1584 rpcz_store.cc:276] 0709 13:03:04.061704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.234154 (+1172450us) service_pool.cc:224] Handling call
0709 13:03:05.234541 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.234783 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:05.898701 (+663918us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.508236 (+609535us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.508473 (+   237us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:06.508556 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.508701 (+   145us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.508951 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36639,"prepare.queue_time_us":787609,"prepare.run_cpu_time_us":347,"prepare.run_wall_time_us":1780,"thread_start_us":1235748,"threads_started":3}
I20260709 13:03:06.573685  1663 rpcz_store.cc:276] 0709 13:03:04.687117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.803015 (+115898us) service_pool.cc:224] Handling call
0709 13:03:04.866228 (+ 63213us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:04.866750 (+   522us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:04.866796 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:04.867576 (+   780us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.680482 (+812906us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:05.680546 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.680637 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.524366 (+843729us) raft_consensus.cc:1685] finished
0709 13:03:06.524398 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.524556 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":55201,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":150,"thread_start_us":812272,"threads_started":1,"wal-append.queue_time_us":1109008}
I20260709 13:03:06.587415  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:06.558876  6069 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:06.563326  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 702) took 1015 ms. Trace:
I20260709 13:03:06.571285  6094 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.615801  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 1952) took 1627 ms. Trace:
I20260709 13:03:06.563414  6096 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.616422  1592 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:06.616302  1660 rpcz_store.cc:276] 0709 13:03:05.548136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.562282 (+1014146us) service_pool.cc:224] Handling call
0709 13:03:06.562652 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.562897 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.562935 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.562997 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:06.563048 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.563118 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.563257 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:06.620383  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.620786  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.621812  1592 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:06.621644  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.622439  1587 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:06.623219  1592 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 13:03:06.595132  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.008s	sys 0.008s
W20260709 13:03:06.624924  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 799446us
I20260709 13:03:06.625793  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.550419  6303 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:06.609983  1589 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 13:03:06.635638  1593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.020s	sys 0.015s
I20260709 13:03:06.604874  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:06.613159  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:06.636348  1593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.800s	user 0.020s	sys 0.015s
I20260709 13:03:06.639469  1663 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:06.640129  1663 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:06.640621  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:06.642110  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54862.8us
I20260709 13:03:06.682801  1663 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.614694  5898 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:06.584913  6302 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.695402  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 706) took 1143 ms. Trace:
I20260709 13:03:06.696696  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:06.697393  1662 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:06.702952  1663 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:06.633634  6187 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:06.645255  6161 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:03:06.738415  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 140836us
I20260709 13:03:06.872052  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:06.603360  6263 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.744503  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2698) took 3661 ms. Trace:
W20260709 13:03:06.652135  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 225269us
I20260709 13:03:06.703127  6109 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:06.707037  6057 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.072s	user 0.016s	sys 0.002s
I20260709 13:03:06.707679  6206 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:06.832363  6325 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
I20260709 13:03:06.704355  1660 rpcz_store.cc:276] 0709 13:03:05.551985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.635335 (+1083350us) service_pool.cc:224] Handling call
0709 13:03:06.693550 (+ 58215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.694981 (+  1431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.695016 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.695071 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:06.695107 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.695171 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.695328 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:06.846633  6151 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:06.830147  6069 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.705375  6063 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:06.703388  6225 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.596393  6309 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.098s
I20260709 13:03:06.707544  6182 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.124s
I20260709 13:03:06.838527  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:06.704846  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:06.644878  6315 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:06.841873  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 184271us
W20260709 13:03:06.842612  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95623.7us
I20260709 13:03:06.616396  1734 rpcz_store.cc:276] 0709 13:03:04.988566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.079643 (+ 91077us) service_pool.cc:224] Handling call
0709 13:03:05.080011 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.085137 (+  5126us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.085170 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.100714 (+ 15544us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.577736 (+477022us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:05.577812 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:05.577935 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:06.615527 (+1037592us) raft_consensus.cc:1685] finished
0709 13:03:06.615550 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.615727 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4694,"raft.queue_time_us":4862,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":13640,"thread_start_us":471985,"threads_started":1,"wal-append.queue_time_us":743263}
I20260709 13:03:06.705343  1588 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:06.663509  6138 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:06.634624  6310 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:06.875401  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.877192  5898 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:06.879158  1585 rpcz_store.cc:276] 0709 13:03:03.083474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.083777 (+2000303us) service_pool.cc:224] Handling call
0709 13:03:05.137121 (+ 53344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.955526 (+818405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.744058 (+788532us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b540031ef54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:06.744399 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1855293824}
I20260709 13:03:06.881773  1663 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:03:06.881898  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:06.950037  1740 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:06.950337  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 245625us
I20260709 13:03:06.950062  1663 pending_rounds.cc:85] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:03:06.882126  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:06.951622  1663 pending_rounds.cc:107] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:03:06.882452  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:06.952052  1663 raft_consensus.cc:2889] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:06.884055  6330 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 13:03:06.806514  6143 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:06.886399  6302 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:06.919312  6323 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.889811  6327 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:06.695292  6321 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:06.902864  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.902370  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:06.847416  6227 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:06.857824  6187 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:06.880326  6324 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 13:03:06.918056  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:06.959962  6161 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:06.945995  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:06.948386  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:06.837071  5881 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:06.952391  1736 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:06.912675  6313 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:06.909967  6151 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:06.839529  6230 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:06.958319  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 120397us
I20260709 13:03:06.870155  6265 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.029392  6151 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.029856  6265 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.031010  6151 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:07.031308  6186 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.032135  6265 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:06.960886  1660 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260709 13:03:07.036307  1660 pending_rounds.cc:85] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 3
I20260709 13:03:07.036621  1660 pending_rounds.cc:107] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 13:03:07.010365  6187 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:06.932871  6033 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.961s	user 0.012s	sys 0.003s
I20260709 13:03:07.039184  1736 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 13:03:06.968650  6309 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:06.964457  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89176.4us
I20260709 13:03:07.043195  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:06.965240  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:06.920819  6314 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:06.938786  6138 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:06.938529  6063 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.991475  1740 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:06.985715  5792 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.006209  1585 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:06.905972  6303 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
W20260709 13:03:07.010094  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.009838  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:06.929901  6225 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.016019  1734 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:07.090143  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:06.974082  6302 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.949831  6048 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 13:03:06.944095  6315 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.017243  6096 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.005610  6227 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:07.085625  6264 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.159s	user 0.016s	sys 0.000s
I20260709 13:03:07.010773  6341 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:06.975576  6323 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:06.959471  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:06.963587  6057 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 13:03:07.036906  1660 raft_consensus.cc:2889] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:06.933873  6235 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.086s	user 0.012s	sys 0.004s
I20260709 13:03:07.040693  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:06.993155  6032 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:06.960968  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 704) took 1412 ms. Trace:
I20260709 13:03:07.159015  1663 rpcz_store.cc:276] 0709 13:03:05.548786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.633416 (+1084630us) service_pool.cc:224] Handling call
0709 13:03:06.633783 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.960124 (+326341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.960689 (+   565us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:07.011381  6161 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:07.013811  6094 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:07.069970  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 129133us
I20260709 13:03:07.076972  6320 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:07.039248  6104 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.085s	user 0.014s	sys 0.003s
I20260709 13:03:07.033527  6255 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:07.033993  6316 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:06.954583  5898 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.079430  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:07.096217  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:07.208279  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 102902us
I20260709 13:03:07.210784  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:07.085368  1742 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:07.212354  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.021075  5881 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.215701  5881 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.041713  6326 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:07.027654  6307 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:07.218319  6307 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:07.219609  6307 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:07.104210  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 61133.2us
I20260709 13:03:07.222221  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:07.109243  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:07.076781  6265 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:07.224296  6265 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.086285  6338 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.102545  6302 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.142601  6235 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:07.061475  6033 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:07.144659  6301 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:03:07.100076  6314 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 13:03:07.107890  6341 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.149955  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:07.151492  6332 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
W20260709 13:03:07.152243  6308 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260709 13:03:07.152990  6138 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:07.154805  6311 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
W20260709 13:03:07.156006  6337 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.140s	user 0.000s	sys 0.001s
I20260709 13:03:07.024541  6313 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.111764  6343 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:07.071090  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47573.6us
I20260709 13:03:07.170428  1663 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:07.287954  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2559) took 1594 ms. Trace:
I20260709 13:03:07.288337  1663 rpcz_store.cc:276] 0709 13:03:05.693170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.165749 (+1472579us) service_pool.cc:224] Handling call
0709 13:03:07.166229 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.287661 (+121432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.287894 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:07.165886  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2553) took 1474 ms. Trace:
I20260709 13:03:07.289451  1660 rpcz_store.cc:276] 0709 13:03:05.691389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.925633 (+1234244us) service_pool.cc:224] Handling call
0709 13:03:06.955869 (+ 30236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.165532 (+209663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.165813 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:07.173619  6310 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.173837  6340 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:03:07.173678  6339 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.279s	user 0.000s	sys 0.002s
I20260709 13:03:07.174981  6186 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.085342  6063 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.298954  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.097925  1734 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:07.299540  6353 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.166790  6161 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.271154  6264 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:03:07.300101  6353 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.300953  6063 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.221447  6307 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:07.229234  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:07.230569  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:07.066890  6309 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.302407  1592 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:07.233691  6306 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 13:03:07.170970  6032 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.303399  6348 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:07.303905  6032 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.305187  6348 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:07.305814  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.306139  6032 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.306586  2311 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:07.237761  1667 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.000s	sys 0.010s
I20260709 13:03:07.240862  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:07.176819  6320 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.227380  5898 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:07.292034  6057 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.309103  6057 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.309250  1740 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:07.310365  6320 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.293325  2313 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [11.11-11.11]
I20260709 13:03:07.312016  6264 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.312516  6264 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.313680  6310 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.314242  6310 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.293010  2315 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:07.294737  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.294935  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.317306  1660 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:07.295801  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.318483  6137 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.319068  1663 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 8: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:07.319972  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.274788  6094 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:07.308426  1667 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.001s	sys 0.010s
I20260709 13:03:07.302415  1591 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.320840  6094 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.318524  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.326260  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.327598  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.242697  6326 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:07.337720  1740 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:07.342325  1591 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:07.346061  1660 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:07.347078  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.274313  6341 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:07.277367  6334 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:07.359269  1660 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:07.360071  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 707) took 1665 ms. Trace:
I20260709 13:03:07.360437  1660 rpcz_store.cc:276] 0709 13:03:05.694968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.358110 (+1663142us) service_pool.cc:224] Handling call
0709 13:03:07.358841 (+   731us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.359756 (+   915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.360005 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:07.377738  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85917.9us
I20260709 13:03:07.410638  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.281750  6263 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.296715  6138 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:06.834163  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 212752us
I20260709 13:03:07.333205  6069 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.427618  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.343210  6314 consensus_queue.cc:1243] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:03:07.413739  6332 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 13:03:07.236071  6338 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:07.437121  6331 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 13:03:07.437528  6359 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:07.438712  6326 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.227016  6316 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.439860  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 702) took 2178 ms. Trace:
W20260709 13:03:07.426172  6356 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 13:03:07.448287  1735 rpcz_store.cc:276] 0709 13:03:05.261486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.276581 (+ 15095us) service_pool.cc:224] Handling call
0709 13:03:05.276920 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.277304 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.277330 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.320227 (+ 42897us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.633161 (+312934us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:03:06.030469 (+397308us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:06.030536 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.030643 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.433678 (+1403035us) raft_consensus.cc:1685] finished
0709 13:03:07.433702 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.439757 (+  6055us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18908,"raft.queue_time_us":101,"raft.run_cpu_time_us":3092,"raft.run_wall_time_us":349126,"spinlock_wait_cycles":11638528,"thread_start_us":559312,"threads_started":1,"wal-append.queue_time_us":1586140}
I20260709 13:03:07.427724  6315 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.430563  6225 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.440367  5792 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.424726  6347 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.328284  6343 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:07.495645  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85086.3us
I20260709 13:03:07.435948  6069 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.466094  6315 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.455567  6355 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:07.522047  6355 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.473963  6314 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:07.538707  1660 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: [11.5-11.5]
W20260709 13:03:07.544628  6151 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.004s
I20260709 13:03:07.558851  6370 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 13:03:07.560689  6370 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:07.567621  6338 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.569954  6302 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.570111  6033 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:07.571828  6357 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.132s	user 0.001s	sys 0.001s
I20260709 13:03:07.579602  1663 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
W20260709 13:03:07.572068  6279 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.463s	user 0.015s	sys 0.000s
W20260709 13:03:07.509549  6278 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.386s	user 0.014s	sys 0.003s
I20260709 13:03:07.580296  6323 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.580626  6278 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.583302  6278 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.583889  1736 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:07.532660  6315 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.447917  6359 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.592259  6255 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:07.580263  6279 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.597218  6316 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.597774  6343 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.599928  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2702) took 4515 ms. Trace:
I20260709 13:03:07.601389  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 703) took 2053 ms. Trace:
I20260709 13:03:07.602578  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2551) took 2417 ms. Trace:
I20260709 13:03:07.561167  6370 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:07.625600  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.625698  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.624325  1667 rpcz_store.cc:276] 0709 13:03:05.548481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.565581 (+1017100us) service_pool.cc:224] Handling call
0709 13:03:06.565944 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.566331 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.566367 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.566867 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 13:03:07.320952 (+754085us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:07.320999 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.321068 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.601154 (+280086us) raft_consensus.cc:1685] finished
0709 13:03:07.601182 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.601322 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1094,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":749034,"threads_started":1,"wal-append.queue_time_us":926650}
I20260709 13:03:07.627094  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.624469  1666 rpcz_store.cc:276] 0709 13:03:05.185187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.497341 (+312154us) service_pool.cc:224] Handling call
0709 13:03:05.565003 (+ 67662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.665791 (+100788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.602141 (+1936350us) spinlock_profiling.cc:243] Waited 97.6 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:07.602447 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224554752}
I20260709 13:03:07.603101  5898 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.618182  6313 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.617934  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.630326  2310 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 8.
I20260709 13:03:07.629756  1667 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:07.605428  6162 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 7 FOLLOWER]: Advancing to term 8
I20260709 13:03:07.633137  6224 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:07.633553  6224 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:07.628474  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.634588  6094 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.634469  6224 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.635625  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.636230  1666 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:03:07.641198  6227 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.003s
I20260709 13:03:07.646255  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.647241  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.647050  2315 raft_consensus.cc:2376] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11. Re-sending same reply.
I20260709 13:03:07.623291  1593 rpcz_store.cc:276] 0709 13:03:03.084494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.087053 (+2002559us) service_pool.cc:224] Handling call
0709 13:03:05.138746 (+ 51693us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:03:05.194988 (+ 56242us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 13:03:05.737899 (+542911us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.902975 (+1165076us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 13:03:06.912211 (+  9236us) log.cc:844] Serialized 104 byte log entry
0709 13:03:06.912391 (+   180us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:06.912653 (+   262us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.912892 (+   239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.912985 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.599655 (+686670us) raft_consensus.cc:1685] finished
0709 13:03:07.599687 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.599865 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97965,"prepare.queue_time_us":772688,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":5092,"raft.queue_time_us":426601,"raft.run_cpu_time_us":774,"raft.run_wall_time_us":61722,"thread_start_us":1608442,"threads_started":3,"wal-append.queue_time_us":1144041}
W20260709 13:03:07.652559  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2554) took 1950 ms (client timeout 2179 ms). Trace:
W20260709 13:03:07.653184  1662 rpcz_store.cc:269] 0709 13:03:05.691726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.327950 (+636224us) service_pool.cc:224] Handling call
0709 13:03:07.641976 (+1314026us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b540044e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:07.642088 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511954816}
I20260709 13:03:07.654582  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.655323  2315 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:07.656903  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.656944  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:07.629753  6357 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.050s	user 0.000s	sys 0.002s
I20260709 13:03:07.607744  6315 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.615233  6314 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.609678  6359 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.665266  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.472074  6347 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.605476  6161 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:07.673378  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.624051  6278 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 13:03:07.672611  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:07.674657  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.681257  6161 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.681562  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.682080  6161 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:07.664857  6314 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.684191  6347 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.670310  6366 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:07.685092  6366 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:07.660804  6365 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:07.685536  6366 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.685750  6365 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.688221  2311 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:03:07.697038  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.701555  2312 raft_consensus.cc:2368] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:07.701723  6278 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:07.655851  6316 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 13:03:07.702304  6278 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:07.703184  6278 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.5, Last appended by leader: 5, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.688675  6161 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:07.704496  6278 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.701893  6319 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.705226  6161 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.706161  6341 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.709146  6341 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.713939  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:07.709692  6375 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:07.717034  6375 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.711165  6068 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:07.711731  6365 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:07.721699  6033 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.722280  6033 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.706177  6319 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.714865  2312 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:07.715667  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:07.716408  1632 leader_election.cc:400] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:07.724354  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.653002  6227 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.727599  1632 leader_election.cc:403] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:03:07.729199  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.723382  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.688880  6313 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:07.723167  6375 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:07.726310  6359 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:07.735841  1666 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 11. (term mismatch)
I20260709 13:03:07.730535  6316 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:07.731911  6313 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.763451  1666 pending_rounds.cc:85] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:03:07.762671  2318 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 8.
I20260709 13:03:07.673979  6370 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.759393  6323 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:07.785938  1666 pending_rounds.cc:107] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 11.11
I20260709 13:03:07.775499  6104 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.776911  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 636) took 1697 ms. Trace:
I20260709 13:03:07.777794  6361 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 13:03:07.758636  6376 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:07.779301  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.781041  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 701) took 2519 ms. Trace:
I20260709 13:03:07.789227  1586 rpcz_store.cc:276] 0709 13:03:06.078997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.079118 (+   121us) service_pool.cc:224] Handling call
0709 13:03:06.109190 (+ 30072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.564036 (+454846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.776576 (+1212540us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b540025c454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:07.776836 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822179712}
I20260709 13:03:07.788892  1666 raft_consensus.cc:2889] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:07.731195  6364 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 13:03:07.732043  6172 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 13:03:07.785053  6187 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.717s	user 0.014s	sys 0.000s
W20260709 13:03:07.805576  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.801528  1733 rpcz_store.cc:276] 0709 13:03:05.261201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.261641 (+   440us) service_pool.cc:224] Handling call
0709 13:03:05.261990 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.283259 (+ 21269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.283399 (+   140us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.283932 (+   533us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.834921 (+550989us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:03:07.780472 (+1945551us) spinlock_profiling.cc:243] Waited 1.94 ms on lock 0x7b180005d160. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1
0709 13:03:07.780672 (+   200us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:07.780711 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.780788 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.780837 (+    49us) raft_consensus.cc:1685] finished
0709 13:03:07.780847 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.780981 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8983,"raft.queue_time_us":109,"raft.run_cpu_time_us":3394,"raft.run_wall_time_us":562843,"spinlock_wait_cycles":13651584,"thread_start_us":559942,"threads_started":1,"wal-append.queue_time_us":913713}
I20260709 13:03:07.810537  6033 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:07.807173  6376 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:07.801030  6367 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:03:07.787813  6313 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.788064  6162 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:07.811729  6375 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.835729  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.836846  1734 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260709 13:03:07.787220  5898 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.815153  6366 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:07.807694  6104 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.816726  6316 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:07.819203  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83553.2us
I20260709 13:03:07.820364  1733 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:07.821439  1736 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:07.844310  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.821979  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.823666  1737 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:07.846532  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:07.827927  6281 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.649s	user 0.010s	sys 0.005s
I20260709 13:03:07.830571  1740 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:07.832855  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:07.895921  1742 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.847074  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:07.897073  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50123.1us
I20260709 13:03:07.852571  6375 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:07.832366  6232 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:07.855852  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:07.856966  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260709 13:03:07.904666  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.861043  6319 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:07.904996  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48398.7us
I20260709 13:03:07.828537  6187 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.863647  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.864181  6162 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.867883  1668 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:07.869266  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.910175  2318 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:07.870415  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.871313  6374 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:07.872218  6279 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.876775  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.876873  6314 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.882665  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 703) took 2580 ms. Trace:
I20260709 13:03:07.882216  6316 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:07.830231  6376 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:07.885075  6366 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:07.886830  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2730) took 1278 ms. Trace:
I20260709 13:03:07.905985  5898 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader election won for term 8
I20260709 13:03:07.892103  1737 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:06.569139  1664 rpcz_store.cc:276] 0709 13:03:05.183653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.320439 (+136786us) service_pool.cc:224] Handling call
0709 13:03:05.320787 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:05.321015 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:06.021399 (+700384us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.482655 (+461256us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.482822 (+   167us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:06.482870 (+    48us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:06.482920 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.482995 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:06.483137 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8227,"prepare.queue_time_us":238383,"prepare.run_cpu_time_us":162,"prepare.run_wall_time_us":5824,"raft.queue_time_us":109442,"raft.run_cpu_time_us":453,"raft.run_wall_time_us":33255,"spinlock_wait_cycles":33460352,"thread_start_us":1101804,"threads_started":2}
I20260709 13:03:07.935643  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.936820  1664 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:07.890376  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89380.7us
I20260709 13:03:07.861877  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:07.905133  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:07.938663  2317 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:07.907564  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:07.939708  1734 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:07.902707  6281 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.730s	user 0.011s	sys 0.005s
I20260709 13:03:07.922958  1735 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:07.926743  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:07.925436  1738 rpcz_store.cc:276] 0709 13:03:05.302531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.302698 (+   167us) service_pool.cc:224] Handling call
0709 13:03:05.303032 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.492695 (+189663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.492727 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.493323 (+   596us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.271428 (+778105us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:06.271492 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.882186 (+1610694us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:07.882245 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:07.882303 (+    58us) raft_consensus.cc:1685] finished
0709 13:03:07.882321 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:07.882598 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9704,"raft.queue_time_us":886,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":9179,"spinlock_wait_cycles":435147648,"thread_start_us":722080,"threads_started":1,"wal-append.queue_time_us":1184772}
I20260709 13:03:07.946877  1662 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:07.891201  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:07.948143  1736 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 12 because replica has last-logged OpId of term: 11 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 4.
I20260709 13:03:07.923985  6374 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:07.932881  1588 rpcz_store.cc:276] 0709 13:03:06.608644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.608822 (+   178us) service_pool.cc:224] Handling call
0709 13:03:06.644880 (+ 36058us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.934754 (+289874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:07.886455 (+951701us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b5400002654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:07.886727 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100708608}
I20260709 13:03:07.904141  6232 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:07.936411  2318 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:07.945412  1734 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
W20260709 13:03:07.927428  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.967501  1664 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:07.969309  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:07.943873  6305 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:07.968954  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.975721  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:07.982594  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:07.985865  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:07.988585  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:07.989990  6161 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:07.963891  5898 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:08.004441  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.005755  1662 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:08.000967  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:07.977043  6338 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:08.005254  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:08.011199  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.016611  1660 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 11
W20260709 13:03:08.015970  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 88331.3us
I20260709 13:03:08.003306  6161 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:07.972934  6305 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.010571  6235 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:08.035032  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.039254  1736 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:08.015599  6338 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:08.022806  6161 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:08.054075  2317 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:08.040572  6319 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:08.061094  1736 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [9.6-11.7]
I20260709 13:03:08.053493  6139 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.066951  6365 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:08.070156  6386 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 13:03:08.071072  6385 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 13:03:08.062462  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:08.010449  6232 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.059725  6279 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.051713  6187 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.049593  6235 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:08.052421  6161 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:08.072654  6319 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:08.092271  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:08.093484  1733 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:08.091665  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.032027  6374 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.100997  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.102658  1736 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:08.106859  1660 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:08.108451  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.115816  1734 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 13:03:08.113170  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.117247  1742 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:08.093945  6279 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.146097  6279 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:08.120774  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:08.147151  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:08.147367  1735 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:08.148157  6305 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.149686  5881 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:08.121098  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.072249  5898 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.123206  6323 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:08.123332  6376 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.102455  6338 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:08.124270  6390 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:03:08.107925  6395 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 13:03:08.126549  6162 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.127170  6235 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.129649  6374 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:08.133965  6319 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.135265  6333 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:08.140717  6161 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.174543  6319 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:08.175644  6161 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.142176  6384 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:08.098800  6314 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.177889  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.178733  1742 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:08.180081  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.181169  1742 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:03:08.182397  6316 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.183971  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2556) took 2491 ms. Trace:
I20260709 13:03:08.184379  1668 rpcz_store.cc:276] 0709 13:03:05.692340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.022278 (+1329938us) service_pool.cc:224] Handling call
0709 13:03:07.061052 (+ 38774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.909847 (+848795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.183691 (+273844us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:08.183905 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1854824192}
I20260709 13:03:08.122695  6365 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:08.150141  5881 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:08.165485  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 698) took 3467 ms. Trace:
I20260709 13:03:08.140806  6353 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.183830  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:08.188563  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87908.2us
I20260709 13:03:08.154733  6204 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 13:03:08.153838  1736 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:08.186336  1665 rpcz_store.cc:276] 0709 13:03:04.697799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.938779 (+240980us) service_pool.cc:224] Handling call
0709 13:03:04.939239 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:05.106250 (+167011us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.106287 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:05.137327 (+ 31040us) log.cc:844] Serialized 26 byte log entry
0709 13:03:05.783424 (+646097us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:05.783492 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.156658 (+2373166us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:08.156837 (+   179us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:08.156888 (+    51us) raft_consensus.cc:1685] finished
0709 13:03:08.156902 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.165408 (+  8506us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12660,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":50786,"spinlock_wait_cycles":383181696,"thread_start_us":603445,"threads_started":1,"wal-append.queue_time_us":861487,"wal-append.run_cpu_time_us":4483,"wal-append.run_wall_time_us":1532989}
I20260709 13:03:08.153838  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:08.189548  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:08.173606  6333 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:08.203248  6313 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.204236  6333 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.201887  1665 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:08.218847  1663 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:08.220899  1735 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
W20260709 13:03:08.225792  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.216915  6333 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:08.253764  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.256567  6309 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.250830  6313 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:08.283453  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 94066.8us
I20260709 13:03:08.284832  6376 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:08.296073  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.381052  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.400316  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.76865e+06us
W20260709 13:03:08.401856  2314 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 442) took 2838 ms (client timeout 1656 ms). Trace:
I20260709 13:03:08.285041  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.397423  6313 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:08.281527  6405 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:08.416813  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 131885us
W20260709 13:03:08.419183  2314 rpcz_store.cc:269] 0709 13:03:05.563741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.563925 (+   184us) service_pool.cc:224] Handling call
0709 13:03:08.299586 (+2735661us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:08.401754 (+102168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":531837696}
I20260709 13:03:08.419605  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.421093  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.403378  6364 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:08.418594  6308 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.133s	user 0.000s	sys 0.009s
I20260709 13:03:08.425470  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.412453  6404 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:08.422176  6301 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.136s	user 0.004s	sys 0.006s
W20260709 13:03:08.425267  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 164179us
I20260709 13:03:08.432905  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2546) took 3249 ms. Trace:
I20260709 13:03:08.424767  6279 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:08.435935  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:08.434255  1661 rpcz_store.cc:276] 0709 13:03:05.183898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.321181 (+137283us) service_pool.cc:224] Handling call
0709 13:03:05.321505 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.076820 (+755315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.432616 (+2355796us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:08.432832 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1723512704}
I20260709 13:03:08.433375  6404 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:08.438560  6404 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:08.436777  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.426437  6364 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:08.444010  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 210826us
I20260709 13:03:08.422032  6313 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.443429  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.454923  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 706) took 2935 ms. Trace:
I20260709 13:03:08.441645  6316 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:08.450404  6384 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 13:03:08.455319  1741 rpcz_store.cc:276] 0709 13:03:05.519163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.519302 (+   139us) service_pool.cc:224] Handling call
0709 13:03:05.519698 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:05.520079 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:05.520108 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:05.530076 (+  9968us) log.cc:844] Serialized 52 byte log entry
0709 13:03:06.270778 (+740702us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:06.480189 (+209411us) log.cc:844] Serialized 14 byte log entry
0709 13:03:06.500650 (+ 20461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:06.500770 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:08.307176 (+1806406us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b240020ff18. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2efb0 00007fcdedba5488 00007fcdedba5be8 00007fcdedb7d398 00007fcdedb7876a 00007fcdedb77bb6
0709 13:03:08.451480 (+144304us) raft_consensus.cc:1685] finished
0709 13:03:08.454717 (+  3237us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.454837 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42443,"prepare.queue_time_us":806444,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":57,"raft.queue_time_us":100968,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":10294,"spinlock_wait_cycles":106662784,"thread_start_us":935812,"threads_started":2,"wal-append.queue_time_us":1478985}
I20260709 13:03:08.460448  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:08.443878  6405 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:08.465879  6279 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:08.467336  6279 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:08.467731  6279 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:08.468756  6279 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.466274  6405 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:08.434020  6410 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:08.444684  6338 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:08.463195  6364 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.512522  6384 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:08.512810  6364 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:08.514266  6405 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:08.516846  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.483608  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.501583  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.460628  5898 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:03:08.484345  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48514.3us
W20260709 13:03:08.527233  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50276.1us
I20260709 13:03:08.483323  6279 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:08.536211  6172 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.562307  6172 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:08.540696  6411 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 13:03:08.486037  6271 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:03:08.555828  6413 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 13:03:08.485029  6401 raft_consensus.cc:2764] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:08.567061  6271 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:08.570521  6313 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.571550  2314 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:08.572768  6172 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:08.539506  6278 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 13:03:08.556937  6415 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:08.584707  6172 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:08.580734  6271 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.585158  6172 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:08.585799  6271 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:08.586230  6172 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.587987  6172 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:08.598016  6404 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.600606  6404 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:08.603323  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64457.6us
W20260709 13:03:08.607587  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 119490us
W20260709 13:03:08.694114  1708 connection.cc:256] RPC call timeout handler was delayed by -1.5473582900000338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260709 13:03:08.695434  1708 connection.cc:256] RPC call timeout handler was delayed by -1.1234582900000305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
W20260709 13:03:08.739446  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 131355us
W20260709 13:03:08.739519  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 117376us
W20260709 13:03:08.750308  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 705) took 3186 ms (client timeout 2247 ms). Trace:
W20260709 13:03:08.751497  1659 rpcz_store.cc:269] 0709 13:03:05.551159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.982865 (+431706us) service_pool.cc:224] Handling call
0709 13:03:08.737725 (+2754860us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:08.737978 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2978341248}
I20260709 13:03:08.765165  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:08.765996  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:08.766561  1738 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 12
W20260709 13:03:08.750084  6419 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:03:08.765380  1734 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 13:03:08.749960  6278 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.752568  6384 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:08.772706  6224 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:08.778386  6384 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.758163  6271 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:03:08.782696  6264 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.787045  6057 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.787379  1660 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 13:03:08.787889  1660 pending_rounds.cc:85] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:03:08.788188  1660 pending_rounds.cc:107] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 13:03:08.788455  1660 raft_consensus.cc:2889] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:08.791544  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2572) took 1466 ms. Trace:
I20260709 13:03:08.791941  1660 rpcz_store.cc:276] 0709 13:03:07.324695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.117947 (+793252us) service_pool.cc:224] Handling call
0709 13:03:08.118290 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:08.788970 (+670680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.791219 (+  2249us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:08.791460 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1537888512}
W20260709 13:03:08.815647  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.852365  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81738.7us
I20260709 13:03:08.879022  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 718) took 1012 ms. Trace:
I20260709 13:03:08.879743  1661 rpcz_store.cc:276] 0709 13:03:07.866824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.437657 (+570833us) service_pool.cc:224] Handling call
0709 13:03:08.438114 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:08.438798 (+   684us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:08.453397 (+ 14599us) log.cc:844] Serialized 14 byte log entry
0709 13:03:08.483757 (+ 30360us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:08.484288 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 13:03:08.484449 (+   161us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:08.484485 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.484573 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:08.878666 (+394093us) raft_consensus.cc:1685] finished
0709 13:03:08.878700 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.878910 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10901,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":4840,"raft.queue_time_us":257,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":35954,"thread_start_us":18211,"threads_started":1,"wal-append.queue_time_us":12663}
I20260709 13:03:08.861321  6400 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
W20260709 13:03:08.892446  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:08.901746  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 1942) took 3927 ms (client timeout 1533 ms). Trace:
I20260709 13:03:08.902408  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2655) took 1077 ms. Trace:
W20260709 13:03:08.902274  1739 rpcz_store.cc:269] 0709 13:03:04.974545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:05.004778 (+ 30233us) service_pool.cc:224] Handling call
0709 13:03:08.901434 (+3896656us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003f9dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:08.901627 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2927556352}
I20260709 13:03:08.902925  1737 rpcz_store.cc:276] 0709 13:03:07.824733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.846279 (+ 21546us) service_pool.cc:224] Handling call
0709 13:03:08.902196 (+1055917us) spinlock_profiling.cc:243] Waited 41.8 ms on lock 0x7b5400284454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:08.902333 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96248192}
I20260709 13:03:08.884212  6069 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.909294  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 918) took 1279 ms. Trace:
I20260709 13:03:08.847981  6278 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.913949  6338 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.914306  6355 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.914670  6338 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.915946  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2576) took 1036 ms. Trace:
I20260709 13:03:08.916396  1668 rpcz_store.cc:276] 0709 13:03:07.879897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.570431 (+690534us) service_pool.cc:224] Handling call
0709 13:03:08.570862 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:08.571250 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:08.571283 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:08.571813 (+   530us) log.cc:844] Serialized 26 byte log entry
0709 13:03:08.750630 (+178817us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:08.750673 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.750747 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:08.915720 (+164973us) raft_consensus.cc:1685] finished
0709 13:03:08.915742 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.915885 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1446,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":388,"thread_start_us":178362,"threads_started":1,"wal-append.queue_time_us":340915}
I20260709 13:03:08.919548  1738 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:08.911108  2313 rpcz_store.cc:276] 0709 13:03:07.630330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.630472 (+   142us) service_pool.cc:224] Handling call
0709 13:03:07.631133 (+   661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:07.631388 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:07.755847 (+124459us) log.cc:844] Serialized 14 byte log entry
0709 13:03:08.908578 (+1152731us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800404800. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:03:08.908754 (+   176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:08.908834 (+    80us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:08.908920 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.909029 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.909209 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":154574,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":9307,"raft.queue_time_us":115446,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":16786,"spinlock_wait_cycles":26225920,"thread_start_us":269346,"threads_started":3,"wal-append.queue_time_us":167164}
I20260709 13:03:08.921335  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:08.933218  6310 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:08.934810  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2672) took 2020 ms. Trace:
I20260709 13:03:08.935220  1584 rpcz_store.cc:276] 0709 13:03:06.914117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.914244 (+   127us) service_pool.cc:224] Handling call
0709 13:03:06.930602 (+ 16358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:06.930933 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:07.046197 (+115264us) log.cc:844] Serialized 14 byte log entry
0709 13:03:07.270348 (+224151us) log.cc:844] Serialized 14 byte log entry
0709 13:03:07.270798 (+   450us) log.cc:844] Serialized 14 byte log entry
0709 13:03:07.521091 (+250293us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b180043fde0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:08.847771 (+1326680us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x7b180043fde0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:08.934266 (+ 86495us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b180043fde0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:03:08.934436 (+   170us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:08.934500 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:08.934552 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:08.934623 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:08.934744 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49673,"prepare.queue_time_us":47682,"prepare.run_cpu_time_us":176,"prepare.run_wall_time_us":15228,"raft.queue_time_us":50937,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":159,"spinlock_wait_cycles":148765056,"thread_start_us":269161,"threads_started":2,"wal-append.queue_time_us":297007,"wal-append.run_cpu_time_us":4851,"wal-append.run_wall_time_us":1314726}
W20260709 13:03:08.938177  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 1.69743e+06us
I20260709 13:03:08.940157  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 715) took 1213 ms. Trace:
W20260709 13:03:08.941076  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.942014  6137 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:08.944628  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.51711e+06us
I20260709 13:03:08.944815  6329 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:08.945670  6329 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:08.946831  6329 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.948125  1586 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:08.948851  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.950661  1668 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:08.951004  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:08.951208  6340 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:08.951588  6340 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:08.952576  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:08.952350  6340 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.952875  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.953171  1587 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:08.953538  1591 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:08.954146  6334 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:08.955313  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:08.955128  6334 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.956336  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.950708  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.957082  1592 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:08.950570  6127 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260709 13:03:08.959295  6127 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:08.959641  6127 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:08.961220  6422 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.961748  6422 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:08.962014  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:08.962272  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:08.962953  1659 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 13:03:08.963658  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:08.964046  6422 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.967694  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:08.968796  1660 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:08.964327  5898 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:08.969764  6283 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:08.970250  6283 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:08.970098  5898 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.963719  6395 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.008s
I20260709 13:03:08.971256  6283 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.972354  6283 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:08.964234  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.964526  6187 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:08.974308  6187 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.975348  5881 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:08.976567  5881 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:08.976725  6334 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:08.940932  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:08.977706  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:08.978317  2313 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:08.978900  6410 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:08.981222  1666 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:08.982460  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:08.982573  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:08.983320  1591 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:08.983479  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:08.940714  1663 rpcz_store.cc:276] 0709 13:03:07.726640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.778618 (+ 51978us) service_pool.cc:224] Handling call
0709 13:03:08.939889 (+1161271us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54003c9454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:08.940042 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":846790528}
I20260709 13:03:08.984779  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:08.995141  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:08.989872  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:08.994249  1659 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:08.985438  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:08.995561  1557 connection.cc:256] RPC call timeout handler was delayed by -3.53172140800001s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 13:03:09.000110  1663 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:03:09.001348  1557 connection.cc:256] RPC call timeout handler was delayed by -3.5137214080000376s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
I20260709 13:03:08.997582  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:08.960455  6127 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:09.003829  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.010541  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:09.011476  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:09.012238  6135 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:08.973495  5898 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:09.010479  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.002653  6422 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:08.925472  6387 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:09.019130  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.004961  1557 connection.cc:256] RPC call timeout handler was delayed by -3.50292140800002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
I20260709 13:03:09.013279  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:09.010632  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.027232  2314 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:09.010637  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:08.981164  6340 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:09.018244  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.023232  1591 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:09.004267  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:09.038794  1589 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:09.017607  5898 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:08.996706  6318 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 13:03:08.977289  6187 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:09.028155  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.021541  6395 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:09.038944  1663 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:09.064007  6395 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:09.065060  6395 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.028535  6127 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:09.067631  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.068228  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:09.069106  1661 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:09.069927  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:09.023985  1587 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:09.072935  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:09.068195  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.073493  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.074235  1587 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:09.075759  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.041532  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.037885  6326 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.077556  1587 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
W20260709 13:03:09.074257  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.079936  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.055097  6265 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.037979  6387 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:09.058625  6338 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:09.060357  6278 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:09.061491  6370 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:09.029388  1557 connection.cc:256] RPC call timeout handler was delayed by -3.4210214080000014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20260709 13:03:09.105288  1557 connection.cc:256] RPC call timeout handler was delayed by -3.4039214080000306s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
I20260709 13:03:09.075896  1584 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:09.107540  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.079922  1663 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:09.088967  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.090616  1589 raft_consensus.cc:2368] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:09.110227  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.093276  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:09.077240  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 379894us
I20260709 13:03:09.095064  1666 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:09.111953  1660 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:09.094026  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.052380  6227 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:09.105973  1587 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:09.112766  1591 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:09.113713  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.113802  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.114755  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.114794  1587 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:03:09.105849  1557 connection.cc:256] RPC call timeout handler was delayed by -3.3913214080000103s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 13:03:09.118917  1557 connection.cc:256] RPC call timeout handler was delayed by -3.3904214080000088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.169s for a response
W20260709 13:03:09.119726  1557 connection.cc:256] RPC call timeout handler was delayed by -3.0718214080000052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20440000000000003s for a response
W20260709 13:03:09.120343  1557 connection.cc:256] RPC call timeout handler was delayed by -3.0430214080000155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2076s for a response
W20260709 13:03:09.120915  1557 connection.cc:256] RPC call timeout handler was delayed by -2.6704214080000384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24900000000000003s for a response
I20260709 13:03:09.121951  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:09.123028  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.127499  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:09.121428  1557 connection.cc:256] RPC call timeout handler was delayed by -2.3473214080000275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28490000000000004s for a response
W20260709 13:03:09.128202  1557 connection.cc:256] RPC call timeout handler was delayed by -2.2447214080000322s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2963s for a response
W20260709 13:03:09.128561  1557 connection.cc:256] RPC call timeout handler was delayed by -1.7646007050000208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20260709 13:03:09.078040  6319 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.080302  5898 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:09.147015  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.150211  1634 leader_election.cc:400] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:09.151712  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:09.106716  6137 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:09.184376  6232 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:09.184196  6137 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.186204  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 641) took 3062 ms. Trace:
I20260709 13:03:09.186467  6137 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.140483  6265 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:09.186653  1590 rpcz_store.cc:276] 0709 13:03:06.124245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:06.249196 (+124951us) service_pool.cc:224] Handling call
0709 13:03:06.249752 (+   556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:06.604118 (+354366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:06.604168 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:06.619234 (+ 15066us) log.cc:844] Serialized 26 byte log entry
0709 13:03:06.840964 (+221730us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:06.841014 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.185802 (+2344788us) spinlock_profiling.cc:243] Waited 87.1 ms on lock 0x7b540031db54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:09.185855 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:09.185901 (+    46us) raft_consensus.cc:1685] finished
0709 13:03:09.185910 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:09.186138 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28251,"raft.queue_time_us":310752,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":200430464,"thread_start_us":369868,"threads_started":2,"wal-append.queue_time_us":965629,"wal-append.run_cpu_time_us":1702,"wal-append.run_wall_time_us":1076361}
I20260709 13:03:09.189138  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.190361  1590 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:09.145740  6310 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:09.177588  6387 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.140964  6109 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.140507  6410 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:09.178866  6187 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:09.081660  6326 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.141182  6230 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.175352  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:09.193523  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:09.193794  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:09.234270  2318 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:09.191744  6343 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:09.236855  2315 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
W20260709 13:03:09.195871  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.199481  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:09.238517  2312 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:09.200656  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:09.205848  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.239368  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:09.213780  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.241945  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:09.158058  6355 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:09.179509  5898 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.221380  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:09.187330  6265 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:09.143586  6278 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
W20260709 13:03:09.224902  6411 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:03:09.223604  6137 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.228933  1591 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:03:09.231560  6326 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:09.220609  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:09.244325  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69075us
I20260709 13:03:09.278313  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.279424  1590 raft_consensus.cc:2368] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:09.279551  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.280586  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.208845  6310 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:09.259318  1632 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:09.207355  6315 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:09.159499  6319 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.256412  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.291540  1589 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:09.282408  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.207995  6366 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 13:03:09.267100  6057 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:09.222618  6109 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.287570  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.253662  6355 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:09.251395  6135 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:09.257284  6340 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:09.319546  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.322850  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.323547  1736 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:09.327804  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.328953  1735 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:09.275463  6422 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:09.290203  6326 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:09.267385  6265 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:09.286496  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:09.262691  6069 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.266163  6264 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.270642  6334 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:09.289891  6310 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:09.236179  6230 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.298115  6172 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:09.296239  6315 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.316919  6355 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:09.285473  6137 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.319198  6187 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.358143  6187 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:09.358661  6427 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:09.350555  6264 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.362352  6359 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:09.362756  6359 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:09.363710  6359 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.364341  6187 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:09.366053  6109 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:09.352221  6172 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:09.362777  6264 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:09.406428  6430 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:03:09.433532  6319 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:09.448539  6431 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 13:03:09.453358  6423 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:09.456879  6423 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:09.490438  6423 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.493754  6423 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:09.503378  6422 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.288236  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:09.560940  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 272647us
I20260709 13:03:09.563066  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:05.622371   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:09.566069  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.569973  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:09.571182  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.573015  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.572899  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:09.574566  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.006s	sys 0.015s
I20260709 13:03:09.574239  1585 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:03:09.575213  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.007s	sys 0.015s
I20260709 13:03:09.577442  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.579495  1585 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:09.574237  1586 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:09.586323  1590 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:09.598677  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.599658  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.600206  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.600689  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.287508  1632 leader_election.cc:403] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:09.602984  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 343763us
I20260709 13:03:09.305068  1659 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:09.606706  1665 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.008s	sys 0.019s
W20260709 13:03:09.607323  1665 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.009s	sys 0.019s
W20260709 13:03:09.607354  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.002s	sys 0.009s
W20260709 13:03:09.607847  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.002s	sys 0.009s
W20260709 13:03:09.608413  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 625039us
I20260709 13:03:09.614933  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:09.617865  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.002s	sys 0.009s
W20260709 13:03:09.618585  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.002s	sys 0.009s
W20260709 13:03:09.619216  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 543400us
W20260709 13:03:09.621569  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.622884  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.624748  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:09.626332  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.003s	sys 0.021s
W20260709 13:03:09.626902  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.004s	sys 0.022s
W20260709 13:03:09.627295  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 508724us
I20260709 13:03:09.630882  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:09.636412  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.309459  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:09.638720  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.022s	sys 0.026s
W20260709 13:03:09.639302  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.022s	sys 0.026s
I20260709 13:03:09.644721  1735 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:09.646692  1736 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 5. Preceding OpId from leader: term: 12 index: 5. (term mismatch)
I20260709 13:03:09.647333  1736 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:03:09.647663  1736 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 11.5
I20260709 13:03:09.648033  1736 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:09.646858  1486 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:09.648804  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.650023  1666 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:09.650324  1659 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:09.312922  1739 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:09.307907  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.653252  1740 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:09.314062  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:09.316890  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:09.673763  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.677789  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.679446  1588 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:03:09.683588  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69182us
I20260709 13:03:09.658511  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:09.416914  6172 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.695765  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.706657  6365 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:09.707945  6365 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 13:03:09.711097  6232 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.526s	user 0.011s	sys 0.001s
I20260709 13:03:09.349210  6069 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.712592  6438 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:09.713768  6094 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:09.716059  6230 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.716826  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:09.718007  6279 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:09.718639  6307 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:09.720070  6137 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.714021  6438 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:09.710590  6316 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:09.721239  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.723361  1590 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:09.701968  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70637.5us
I20260709 13:03:09.705245  1742 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
W20260709 13:03:09.729625  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.723150  6438 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:09.734218  6438 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:09.738252  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2660) took 1911 ms (client timeout 2024 ms). Trace:
W20260709 13:03:09.738862  1733 rpcz_store.cc:269] 0709 13:03:07.826139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.863375 (+ 37236us) service_pool.cc:224] Handling call
0709 13:03:09.737491 (+1874116us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54003520d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:09.737966 (+   475us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":420927360}
I20260709 13:03:09.739486  1590 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 13:03:09.739789  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.721804  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.747334  1590 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:09.748561  6279 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:09.464943  6315 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.757956  6057 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:09.765587  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.766790  1591 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:09.767058  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.768488  1591 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
W20260709 13:03:09.769804  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92065.4us
I20260709 13:03:09.770273  6172 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:09.770659  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.772122  1632 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:09.774001  6315 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:09.773900  6329 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:09.775111  6319 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.774571  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.775812  6319 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.776648  1593 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 13:03:09.781850  6353 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:09.782806  1593 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 12 index: 5. (index mismatch)
I20260709 13:03:09.782945  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:09.784319  6069 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:09.783174  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.788863  1586 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 13:03:09.790567  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.785928  1584 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:09.793103  1590 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 13:03:09.803588  1592 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:09.791218  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.785352  1587 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:09.760622  6057 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:09.828769  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 122600us
W20260709 13:03:09.829794  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.830390  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.830750  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:09.832096  6370 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.006s
I20260709 13:03:09.789865  6316 consensus_queue.cc:1243] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 13:03:09.834630  1589 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.542s	user 0.014s	sys 0.007s
I20260709 13:03:09.835353  1589 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:09.839359  6104 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:09.878978  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61822.6us
I20260709 13:03:09.847301  6307 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:09.887635  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115801us
I20260709 13:03:09.904407  6278 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 13:03:09.905666  6279 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:09.906625  6278 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:09.907435  6278 consensus_queue.cc:1243] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:09.908907  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.908795  6338 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:09.910085  1665 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:09.910437  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.911826  6033 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:09.913838  6316 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:09.923534  6376 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:09.924467  6376 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:09.926118  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:09.927255  1660 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:09.928280  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.930284  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.931056  1661 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 13:03:09.931411  1661 pending_rounds.cc:85] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:09.931775  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:09.932648  6319 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:09.933311  6326 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:09.933737  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:09.911985  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:09.932782  1663 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 11 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:09.929378  1660 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:03:09.931820  1661 pending_rounds.cc:107] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 13:03:09.936861  1661 raft_consensus.cc:2889] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:09.845221  6355 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.786648  6442 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.107s
I20260709 13:03:09.951514  6343 raft_consensus.cc:2764] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 13:03:09.954016  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 260024us
I20260709 13:03:09.954885  6109 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:09.955938  6109 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.951596  1667 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 9: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:09.957360  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:09.988308  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:09.999497  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52301.1us
I20260709 13:03:10.005759  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:10.006949  1664 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
W20260709 13:03:09.997794  6265 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.006s
I20260709 13:03:10.022948  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:10.002679  1634 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.119s (SENT) [suppressed 9 similar messages]
W20260709 13:03:10.024367  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 108022us
W20260709 13:03:10.047463  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.051365  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.047926  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:10.059460  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56837us
W20260709 13:03:10.069583  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.030835  6370 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.030113  6326 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:10.058439  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.074905  6334 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:10.075448  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93366.3us
W20260709 13:03:10.076417  2312 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.826s	user 0.012s	sys 0.008s
I20260709 13:03:10.076261  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.072167  1660 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:03:10.076785  6283 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.104s	user 0.021s	sys 0.005s
W20260709 13:03:10.080482  6127 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.013s	user 0.003s	sys 0.015s
I20260709 13:03:09.958673  6307 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:09.987653  6319 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:10.081662  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:10.082072  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49592.7us
W20260709 13:03:10.086169  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129053us
W20260709 13:03:10.088361  6423 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.594s	user 0.007s	sys 0.009s
I20260709 13:03:10.089133  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.089887  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:10.090348  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 147193us
I20260709 13:03:10.091224  6376 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.091727  6449 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:10.133546  1663 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:03:10.134284  6449 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:10.134992  6109 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.136384  6449 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.095690  1590 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:10.054462  6265 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.142201  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:10.088449  6359 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:10.138227  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.144179  6265 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.130661  1589 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:10.104168  6370 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.146440  1584 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:10.147445  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:10.148064  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.153280  6319 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.154521  6334 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.155483  1634 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
W20260709 13:03:10.155925  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52122.4us
I20260709 13:03:10.156266  6319 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:10.156863  6319 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.158169  6448 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.158782  6448 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.130815  2312 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:10.132347  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:10.132752  1665 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:03:10.132905  1592 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:03:10.138161  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 716) took 2290 ms (client timeout 2098 ms). Trace:
I20260709 13:03:10.163260  1665 pending_rounds.cc:85] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:03:10.173197  1665 pending_rounds.cc:107] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:03:10.173583  1665 raft_consensus.cc:2889] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:10.164608  1664 rpcz_store.cc:269] 0709 13:03:07.847998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:07.969032 (+121034us) service_pool.cc:224] Handling call
0709 13:03:10.137672 (+2168640us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:10.137917 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4316723072}
I20260709 13:03:10.157238  6334 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:10.188246  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78597.6us
W20260709 13:03:10.188176  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 92388.1us
I20260709 13:03:10.197232  1661 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:10.197789  6170 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260709 13:03:10.198876  6310 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.841s	user 0.015s	sys 0.005s
I20260709 13:03:10.199417  6310 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.200628  6109 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:10.137223  6449 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.204090  6449 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.204876  6449 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:10.212996  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.173429  6283 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:10.197808  1661 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:10.212026  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 792) took 1014 ms. Trace:
I20260709 13:03:10.229178  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:10.246711  6459 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 13:03:10.246918  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.248121  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.249549  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.250960  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.251432  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 652) took 1293 ms. Trace:
I20260709 13:03:10.251549  6355 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:10.252271  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 271302us
I20260709 13:03:10.252470  1661 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.253322  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:10.252000  1588 rpcz_store.cc:276] 0709 13:03:08.958287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.729005 (+770718us) service_pool.cc:224] Handling call
0709 13:03:09.793557 (+ 64552us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:09.794399 (+   842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:09.794449 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:09.795473 (+  1024us) log.cc:844] Serialized 52 byte log entry
0709 13:03:09.991012 (+195539us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:10.180735 (+189723us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.194037 (+ 13302us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.219511 (+ 25474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.219611 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.251194 (+ 31583us) raft_consensus.cc:1685] finished
0709 13:03:10.251214 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.251352 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55053,"prepare.queue_time_us":316441,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":45951,"raft.run_cpu_time_us":690,"raft.run_wall_time_us":65681,"thread_start_us":317554,"threads_started":2,"wal-append.queue_time_us":451818}
I20260709 13:03:10.254959  6353 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:10.255560  6353 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.260083  6456 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:10.260489  6422 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:10.229493  2312 rpcz_store.cc:276] 0709 13:03:09.198057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.199138 (+  1081us) service_pool.cc:224] Handling call
0709 13:03:10.187261 (+988123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:10.246407  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.260639  6456 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:10.251758  1660 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 13:03:10.261252  6315 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:10.271548  6315 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.272184  6315 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.272929  6310 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.267515  6283 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:10.273801  6283 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.274241  6283 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.247006  6370 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:10.217314  6319 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.278174  6319 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.269191  1664 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:03:10.279703  1665 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:10.280504  1664 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.281733  1664 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:10.283001  1664 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:10.284343  1664 pending_rounds.cc:85] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 5
I20260709 13:03:10.284636  1664 pending_rounds.cc:107] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:10.284873  1664 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:10.241632  6438 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.287214  6449 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:10.287828  6438 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:10.288230  6449 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.288733  6449 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.290515  1660 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:10.269937  1661 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:10.302999  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108617us
I20260709 13:03:10.304658  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:10.271585  5881 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.294s	user 0.011s	sys 0.010s
I20260709 13:03:10.306348  1659 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 13:03:10.306682  1659 pending_rounds.cc:85] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 8
I20260709 13:03:10.307029  1659 pending_rounds.cc:107] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 13:03:10.307340  1659 raft_consensus.cc:2889] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:10.310425  2310 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:10.310814  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.311933  6109 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:10.312364  1589 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:10.312788  6109 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:10.314039  1590 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:10.311884  1588 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:10.264866  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 654) took 1238 ms. Trace:
I20260709 13:03:10.269292  6456 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:10.318110  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:10.315203  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 183039us
I20260709 13:03:10.315830  1591 rpcz_store.cc:276] 0709 13:03:09.026002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.806911 (+780909us) service_pool.cc:224] Handling call
0709 13:03:09.841249 (+ 34338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:09.842069 (+   820us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:09.842118 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:09.843232 (+  1114us) log.cc:844] Serialized 26 byte log entry
0709 13:03:09.991405 (+148173us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:09.991448 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.991535 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.261467 (+269932us) raft_consensus.cc:1685] finished
0709 13:03:10.261517 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.264775 (+  3258us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":147604,"threads_started":1,"wal-append.queue_time_us":414403}
I20260709 13:03:10.260514  6448 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.333609  1660 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:03:10.331442  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:10.332829  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 190908us
I20260709 13:03:10.328619  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.335247  1590 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:10.314286  6109 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.339479  6388 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.357352  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.366942  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:10.371171  6387 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 13:03:10.373435  6395 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.374109  6395 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.376448  6395 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.377616  6397 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:10.378230  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.378479  6397 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.379233  1734 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:10.348789  6388 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.364789  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.383422  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:10.381151  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 118856us
I20260709 13:03:10.329787  6404 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 13:03:10.387938  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.352061  6376 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.386636  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:10.388836  1587 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:10.389999  1584 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:10.393846  1591 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.402043  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.406262  1661 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 13:03:10.406795  1661 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 9
I20260709 13:03:10.382112  5881 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:10.407526  1592 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:10.381428  6397 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.391676  6395 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:10.413543  1661 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 13:03:10.414970  1661 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:10.414964  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.418749  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.399333  6387 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:10.409164  5881 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.411190  6033 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:10.425803  6033 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.426434  6033 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.412082  6395 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:10.416288  6305 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.412114  6388 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
W20260709 13:03:10.435534  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 100388us
I20260709 13:03:10.432200  5881 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:10.421577  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.417569  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:10.425709  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56224.8us
I20260709 13:03:10.423188  6387 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:10.413606  6376 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.455675  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.454416  6319 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:10.438961  6395 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.454069  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.460826  1708 leader_election.cc:400] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:10.458600  6438 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.461014  1587 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:10.467785  1708 leader_election.cc:403] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:03:10.469854  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2581) took 1473 ms. Trace:
I20260709 13:03:10.469781  6438 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.470425  1668 rpcz_store.cc:276] 0709 13:03:08.960042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:08.998314 (+ 38272us) service_pool.cc:224] Handling call
0709 13:03:09.011559 (+ 13245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:09.011815 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:09.012122 (+   307us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.339161 (+1327039us) log.cc:844] Serialized 14 byte log entry
0709 13:03:10.407766 (+ 68605us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:10.413253 (+  5487us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:10.420787 (+  7534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.424349 (+  3562us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.433298 (+  8949us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29058,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":97,"raft.queue_time_us":21962,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":24046,"thread_start_us":1161979,"threads_started":3,"wal-append.queue_time_us":416974}
I20260709 13:03:10.472605  6438 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:10.473743  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.474462  1587 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:10.478956  6435 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:03:10.444581  6463 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
W20260709 13:03:10.466233  6433 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 13:03:10.483661  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48309.2us
I20260709 13:03:10.492803  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:10.496631  6230 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:10.457340  6448 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:10.502168  1742 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:10.499487  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.497225  6230 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:10.506809  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59402.2us
I20260709 13:03:10.509311  1592 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.483219  6438 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.462496  6319 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.465469  6279 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.442734  6305 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.523406  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.506156  6448 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:10.540704  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 157486us
I20260709 13:03:10.540578  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:10.514215  6278 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:10.514042  6310 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:10.548266  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 157778us
I20260709 13:03:10.547366  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:10.551334  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58739.1us
I20260709 13:03:10.537278  6319 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:10.564040  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.567250  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.550050  6448 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:10.552654  6370 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:10.573366  1591 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:10.572540  1735 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:10.564544  6265 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:10.580360  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.564631  6310 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:10.516083  6387 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.595258  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:10.558110  6462 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 13:03:10.598016  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:10.599830  1591 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:10.590214  6453 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 13:03:10.577745  6279 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.558200  6449 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.566774  6463 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 13:03:10.609115  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.607028  1590 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
W20260709 13:03:10.609207  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 144158us
I20260709 13:03:10.590906  6265 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:10.612749  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74613.9us
W20260709 13:03:10.618438  6466 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 13:03:10.618777  6422 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:10.619300  6422 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.619812  6422 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:10.621914  6387 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:10.624797  6476 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:03:10.627362  6310 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.631188  6370 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:10.631718  6370 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:10.535652  1486 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 11. New cstate: current_term: 11 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:10.634104  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.634308  6265 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:10.637400  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 159700us
I20260709 13:03:10.637892  6307 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:10.639137  6307 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:10.639530  6307 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:10.621397  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:10.642115  6469 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 13:03:10.642791  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.618468  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:10.607488  6355 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:10.648568  6278 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:10.651221  6279 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:10.652329  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 731) took 1030 ms. Trace:
I20260709 13:03:10.575204  6283 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.640599  6307 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.651455  1592 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:10.629305  1586 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:10.680892  1738 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 13:03:10.682142  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63862.1us
I20260709 13:03:10.679085  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.685016  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:10.685956  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.592235  6315 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.693055  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.727113  1734 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:10.647028  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:10.719625  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64655.5us
I20260709 13:03:10.690551  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 653) took 1669 ms. Trace:
I20260709 13:03:10.686208  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:10.681803  1736 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:10.695828  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.729650  1663 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 13:03:10.742201  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95327.7us
I20260709 13:03:10.655917  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:10.742738  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:10.721189  1662 rpcz_store.cc:276] 0709 13:03:09.622622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.624795 (+  2173us) service_pool.cc:224] Handling call
0709 13:03:09.625591 (+   796us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:09.626259 (+   668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:09.626302 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:09.627341 (+  1039us) log.cc:844] Serialized 26 byte log entry
0709 13:03:09.691234 (+ 63893us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:09.691291 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:09.691391 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.651928 (+960537us) raft_consensus.cc:1685] finished
0709 13:03:10.651958 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.652128 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":63049,"threads_started":1,"wal-append.queue_time_us":762057}
I20260709 13:03:10.680822  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:10.659662  6474 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 13:03:10.743348  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:10.680210  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.614867  6230 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.728300  1585 rpcz_store.cc:276] 0709 13:03:09.021105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.768207 (+747102us) service_pool.cc:224] Handling call
0709 13:03:09.768894 (+   687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:09.797936 (+ 29042us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:09.798114 (+   178us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:09.846036 (+ 47922us) log.cc:844] Serialized 26 byte log entry
0709 13:03:10.322125 (+476089us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:10.322166 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.322235 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:10.690259 (+368024us) raft_consensus.cc:1685] finished
0709 13:03:10.690287 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:10.690471 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5984,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":293,"thread_start_us":475574,"threads_started":1,"wal-append.queue_time_us":574061}
W20260709 13:03:10.911450  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 168211us
W20260709 13:03:10.658767  6475 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:03:10.683636  6442 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:10.913234  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 257496us
I20260709 13:03:10.840557  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:10.824007  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:10.739090  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84566.7us
W20260709 13:03:10.914769  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:10.834111  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 148026us
W20260709 13:03:10.796819  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67500.5us
I20260709 13:03:10.919708  1586 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.826802  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:10.897233  1741 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:10.889199  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:10.900517  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:10.781646  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:10.840723  1665 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [8.5-10.6]
W20260709 13:03:10.672041  6471 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.198s	user 0.004s	sys 0.000s
I20260709 13:03:10.682119  6104 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:10.670513  6376 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.663350  6305 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:10.744007  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 173914us
I20260709 13:03:10.684182  6355 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:10.683601  6449 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:10.935928  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 251099us
I20260709 13:03:10.700595  6319 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.729909  6278 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.799888  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:10.939606  6478 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:10.651235  6456 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.944896  6370 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.826066  1486 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 8 to 11, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:10.666536  6353 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.921386  6442 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.945600  6283 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.976621  1585 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:10.911934  6033 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.980180  6033 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.980520  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.807628  6069 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:10.839504  6137 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.836756  6279 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:10.981824  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 155243us
I20260709 13:03:10.981978  6069 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:10.822059  6326 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.125s
I20260709 13:03:10.983341  6069 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.772640  6397 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:10.984762  6397 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.985189  6397 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:10.827034  6109 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:10.986498  1491 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 227) took 64863 ms (client timeout 15000 ms). Trace:
I20260709 13:03:10.847888  5881 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.989161  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.990595  5881 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.990739  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.946702  6315 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.991418  1668 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:10.948568  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:10.991515  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.992307  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:10.992679  1659 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:03:10.992895  1668 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.993824  6032 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:10.994504  1667 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:10.994781  6032 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:10.934526  1739 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:10.952536  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:10.897126  5898 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:10.997642  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:10.998064  5898 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.998304  1736 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:10.998572  5898 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:10.957012  6057 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:11.000890  6057 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.001996  6057 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.003259  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:10.986904  1491 rpcz_store.cc:269] 0709 13:02:06.129416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:06.177960 (+ 48544us) service_pool.cc:224] Handling call
0709 13:03:10.986223 (+64808263us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:02.500441 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:02.788373 (+287932us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:02.794970 (+  6597us) write_op.cc:435] Acquired schema lock
0709 13:03:02.797806 (+  2836us) tablet.cc:662] Decoding operations
0709 13:03:02.841254 (+ 43448us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:02.841375 (+   121us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:02.841408 (+    33us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:03:02.842201 (+   793us) tablet.cc:701] Row locks acquired
0709 13:03:02.842250 (+    49us) write_op.cc:260] PREPARE: finished
0709 13:03:02.842457 (+   207us) write_op.cc:270] Start()
0709 13:03:02.842628 (+   171us) write_op.cc:276] Timestamp: P: 1783602182842367 usec, L: 0
0709 13:03:02.842667 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 13:03:02.863521 (+ 20854us) log.cc:844] Serialized 2478 byte log entry
0709 13:03:03.730387 (+866866us) op_driver.cc:464] REPLICATION: finished
0709 13:03:04.258904 (+528517us) write_op.cc:301] APPLY: starting
0709 13:03:04.280324 (+ 21420us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:04.368634 (+ 88310us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:04.368779 (+   145us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:04.710424 (+341645us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:04.710708 (+   284us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=12,mrs_lookups=0
0709 13:03:04.710775 (+    67us) write_op.cc:312] APPLY: finished
0709 13:03:04.711137 (+   362us) log.cc:844] Serialized 65 byte log entry
0709 13:03:05.004674 (+293537us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.015281 (+ 10607us) write_op.cc:454] Released schema lock
0709 13:03:05.039794 (+ 24513us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:05.071826 (+ 32032us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:05.071869 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3576143,"reported_tablets":36,"child_traces":[["op",{"apply.queue_time_us":429100,"apply.run_cpu_time_us":43493,"apply.run_wall_time_us":815978,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29219,"num_ops":6,"prepare.queue_time_us":367284,"prepare.run_cpu_time_us":60547,"prepare.run_wall_time_us":672240,"raft.queue_time_us":513829,"raft.run_cpu_time_us":38645,"raft.run_wall_time_us":309989,"replication_time_us":887512,"thread_start_us":1390915,"threads_started":5,"wal-append.queue_time_us":685299,"wal-append.run_cpu_time_us":35362,"wal-append.run_wall_time_us":2547654}]]}
I20260709 13:03:11.004171  6463 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:11.004601  6463 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.004980  6463 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:10.947903  6307 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:11.005666  6230 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:10.935879  6104 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:10.957666  6478 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:10.968116  6138 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:10.968271  1486 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 7, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:11.000385  6032 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:10.971277  6353 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:10.960537  6278 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:10.982012  6137 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.019670  1586 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:11.020610  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.024353  1663 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:11.025395  1740 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:11.038472  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:10.970118  1666 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:11.022030  6104 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:11.044122  6137 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.045331  2310 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:11.032406  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.053709  6109 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:10.957587  6305 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:03:10.955880  6354 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
W20260709 13:03:11.074265  6438 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.539s	user 0.006s	sys 0.006s
I20260709 13:03:11.027366  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:11.040372  1739 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:03:10.976673  6442 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:10.960161  6127 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:11.089934  6127 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.091259  1592 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: [9.4-9.4]
I20260709 13:03:11.092308  6127 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.093441  1585 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 13:03:11.093094  6442 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:11.093874  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.094599  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.053889  6355 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.053550  1486 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 10, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 10 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.055029  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:11.024716  6478 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:11.058063  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45510.6us
I20260709 13:03:11.058622  6319 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.060737  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:10.963553  6359 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 13:03:10.954993  6334 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.082s
I20260709 13:03:11.070996  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.076336  6423 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:11.079174  6370 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:11.080539  6395 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.002s
I20260709 13:03:11.082647  6456 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:11.075589  6438 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:11.067639  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.121660  6376 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:11.122754  1666 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:11.089138  1736 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:11.100554  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.050863  6397 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.100493  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.085975  6448 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.095511  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.110046  2312 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:11.105978  6353 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:11.100615  6032 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
W20260709 13:03:11.114318  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53714.4us
I20260709 13:03:11.093146  5898 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.071276  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.118711  1665 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:11.074342  6278 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:11.122674  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112193us
I20260709 13:03:11.120159  6326 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:11.134038  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:11.169909  6278 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.170959  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:11.170900  6278 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.181123  2310 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:11.142669  6422 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.522s	user 0.017s	sys 0.000s
W20260709 13:03:11.138057  6265 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.503s	user 0.014s	sys 0.003s
I20260709 13:03:11.191502  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:11.202519  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.206918  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:11.123472  6395 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:11.214375  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 287114us
W20260709 13:03:11.222723  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 274313us
W20260709 13:03:11.223737  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.012s	sys 0.000s
W20260709 13:03:11.224474  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.012s	sys 0.000s
I20260709 13:03:11.200964  6456 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:11.220329  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 233941us
I20260709 13:03:11.230445  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:11.223518  1585 pending_rounds.cc:85] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:03:11.232086  1585 pending_rounds.cc:107] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:11.232368  1585 raft_consensus.cc:2889] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:11.233834  1668 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:11.201094  6438 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:11.239617  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.242545  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.205122  6265 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.224146  6279 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:11.205730  6422 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.237633  6319 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:11.244664  6388 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:11.243300  6310 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.242106  6438 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:11.304872  2312 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:11.306010  1742 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:11.306985  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.273989  6127 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:11.313246  1588 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:11.317391  1590 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:03:11.287693  6283 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:11.325336  1662 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:11.278362  5898 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:11.293341  6449 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:11.322561  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83075.9us
I20260709 13:03:11.297403  6278 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:11.334510  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 128457us
I20260709 13:03:11.330497  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:11.336148  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 264962us
W20260709 13:03:11.347622  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:11.355597  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.321352  6388 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:11.319183  6376 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:11.367640  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:11.340572  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:11.369762  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.373808  1584 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:11.340584  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:11.357776  6283 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.353389  6359 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:11.413419  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.414913  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.272903  1486 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:11.420856  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 145688us
I20260709 13:03:11.377610  6486 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:11.429992  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.331238  6310 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:11.433709  6490 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 13:03:11.470615  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:11.433961  6279 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.471789  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.462419  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:11.434659  6495 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:11.434355  6388 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.434894  6307 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:11.429548  6438 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.491611  6307 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:11.493067  6497 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 13:03:11.494336  6307 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:11.436162  6422 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:11.436611  6359 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.497052  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.437772  6463 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.438306  6283 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.501938  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2717) took 1177 ms. Trace:
I20260709 13:03:11.502327  1587 rpcz_store.cc:276] 0709 13:03:10.324717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.475196 (+150479us) service_pool.cc:224] Handling call
0709 13:03:10.475581 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:10.634043 (+158462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:10.634087 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:10.634704 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 13:03:10.952791 (+318087us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:10.952833 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:10.952907 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:11.501688 (+548781us) raft_consensus.cc:1685] finished
0709 13:03:11.501711 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:11.501871 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":157952,"raft.queue_time_us":160916,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":317619,"threads_started":1,"wal-append.queue_time_us":505787}
I20260709 13:03:11.418630  6397 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
W20260709 13:03:11.439534  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 72600.7us
W20260709 13:03:11.441040  6489 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.196s	user 0.000s	sys 0.000s
I20260709 13:03:11.440481  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.441740  6502 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:11.441941  6135 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:11.517619  6135 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:11.442283  6498 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.518702  6135 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.519155  6498 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.518499  1486 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 14, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 14 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:11.520325  6135 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.442721  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:11.521670  6491 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 13:03:11.445698  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.522120  1589 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:11.522683  1664 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:11.429041  6448 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.449193  6265 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.450181  1736 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:11.451220  6395 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.455221  6486 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.372275  6462 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:03:11.456933  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:11.459048  6305 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.402318  6485 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.489272  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:11.490713  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 160329us
W20260709 13:03:11.499949  6499 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 13:03:11.500579  6359 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:11.504428  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:11.496819  6492 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.121s	user 0.001s	sys 0.001s
I20260709 13:03:11.520874  1486 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.533828  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:11.494908  6310 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.565307  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
W20260709 13:03:11.539078  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.566399  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.536024  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:11.566990  1587 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:11.507720  6230 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:11.507441  6334 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:11.548389  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.553658  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.557080  6495 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:11.547647  6486 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:11.542501  1589 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:11.568321  1486 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 9, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 9 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.570346  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 664) took 1019 ms. Trace:
I20260709 13:03:11.571462  1588 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:11.581825  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.560513  6498 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:11.583586  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.576924  1593 rpcz_store.cc:276] 0709 13:03:10.550589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.550734 (+   145us) service_pool.cc:224] Handling call
0709 13:03:10.551104 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:10.634150 (+ 83046us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:10.634187 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:10.634774 (+   587us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.025299 (+390525us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:11.025350 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.025437 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:11.570092 (+544655us) raft_consensus.cc:1685] finished
0709 13:03:11.570120 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:11.570271 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63734,"raft.queue_time_us":66789,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":390100,"threads_started":1,"wal-append.queue_time_us":559234}
I20260709 13:03:11.585579  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.586136  1593 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:11.586344  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.586943  1589 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:11.589488  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.591450  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.579700  6370 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:11.560325  6456 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
W20260709 13:03:11.598228  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 82140.9us
I20260709 13:03:11.596946  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.600013  1589 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:11.562350  6485 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.603312  6487 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:11.603775  6487 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:11.604691  6487 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.605613  6504 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 13:03:11.579133  6506 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:03:11.611047  6496 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.014s
I20260709 13:03:11.611755  6478 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.564841  6502 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260709 13:03:11.618011  6104 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.577s	user 0.010s	sys 0.008s
I20260709 13:03:11.618460  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.648000  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:11.648366  1586 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.595273  6283 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:11.621667  6493 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 13:03:11.576081  6305 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.595894  6310 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:11.624068  6359 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:11.648820  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.627372  6485 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:11.628468  2309 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:11.594067  1486 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 2 to 6, term changed from 5 to 9, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193), VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194), VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195), VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 9 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.635066  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:11.621764  6505 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.183s	user 0.006s	sys 0.000s
I20260709 13:03:11.660787  6265 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:11.661228  6265 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.661607  6265 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:11.663023  1589 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:11.664515  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.665266  1589 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:11.642521  6370 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:11.666723  6370 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.668107  6305 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.669185  6370 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:11.632059  6463 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.644061  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.650709  6104 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.671221  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.650630  6283 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.665663  6310 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.640641  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.671980  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.680677  1588 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:11.674006  1667 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:11.675133  1593 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:11.678153  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.685612  1590 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:11.689675  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:11.690868  6478 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.659658  6516 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 13:03:11.695257  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:11.697679  1586 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:11.660032  6487 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:11.694175  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.698663  6506 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:11.697724  6442 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:11.697535  6496 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:11.702992  6310 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.693866  6370 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:11.706074  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 667) took 1137 ms. Trace:
I20260709 13:03:11.678943  6283 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.694098  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:11.711422  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:11.712549  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63821.2us
W20260709 13:03:11.730017  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:11.733378  1661 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:11.714843  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:11.717118  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 168828us
I20260709 13:03:11.717252  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.737951  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:11.711313  6230 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:11.748752  6230 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.751187  1589 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:11.717253  6138 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:11.753669  6138 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:11.754276  6138 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.725970  6370 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.754678  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.758312  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.762164  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:11.762410  1666 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:11.763170  5898 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:11.763523  6442 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:11.764088  6442 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:11.765087  6442 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.766223  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.767967  1592 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:11.755184  6305 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:11.724442  1591 rpcz_store.cc:276] 0709 13:03:10.568239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:10.646111 (+ 77872us) service_pool.cc:224] Handling call
0709 13:03:10.646499 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:10.652058 (+  5559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:10.652092 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:10.652617 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.225014 (+572397us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:11.225058 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.225129 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:11.701883 (+476754us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b24001867b8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2efb0 00007fcdedba5488 00007fcdedba5be8 00007fcdedb7d398 00007fcdedb7876a 00007fcdedb77bb6
0709 13:03:11.705589 (+  3706us) raft_consensus.cc:1685] finished
0709 13:03:11.705615 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:11.705972 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80659,"raft.queue_time_us":6771,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":17947392,"thread_start_us":571951,"threads_started":1,"wal-append.queue_time_us":658577}
I20260709 13:03:11.769461  1592 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [8.5-10.6]
I20260709 13:03:11.738639  1742 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:11.742267  6456 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:11.743841  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.745285  1668 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:11.746035  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:11.747248  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.778013  1664 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:11.779803  1584 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 13:03:11.723172  1486 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 9, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 9 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.807590  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.811249  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:11.801352  1588 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:11.816459  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:11.816866  1586 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 13:03:11.817111  1665 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:11.807495  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.819765  6230 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:11.820314  6230 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.820704  6230 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:11.820775  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:11.821585  1736 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.826385  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.832387  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.833539  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:11.832993  1486 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 5 to 11, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 11 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:11.834136  1591 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:11.833990  6354 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.835454  6354 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:11.838195  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91055.8us
I20260709 13:03:11.839334  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:11.840620  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:11.841415  6488 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.841928  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:11.842131  6488 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.842736  6319 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:11.844509  6305 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.845346  2315 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:11.845630  2312 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:11.845739  6488 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.846364  6319 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:11.845749  6305 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.846843  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:11.847647  1592 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:11.847374  6319 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.849460  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:11.831490  1736 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:11.724999  6516 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:11.728578  6135 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:11.833035  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:11.840674  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 160332us
I20260709 13:03:11.850042  6355 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:11.862932  1742 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:11.864794  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:11.868436  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.850239  6354 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.816219  5881 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:11.875257  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.828868  6104 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:11.828698  6422 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.881464  1738 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:11.870200  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:11.877847  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.859896  6135 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:11.876019  1734 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
W20260709 13:03:11.890599  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51401.6us
I20260709 13:03:11.890008  1591 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:11.869192  6310 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:11.913844  5881 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:11.902776  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 208787us
I20260709 13:03:11.874110  6127 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:11.905680  5898 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:11.915397  5881 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.916666  5898 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:11.917104  5898 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:11.918072  5898 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.918998  1739 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:11.919480  5898 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:11.898761  6135 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:11.905691  6520 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 13:03:11.923597  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.924291  1662 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:11.925192  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.906445  1586 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [13.11-15.12]
I20260709 13:03:11.907748  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:11.872154  6516 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:11.909977  6370 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.926558  1740 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:11.928278  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:11.915390  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:11.910228  6456 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:11.932237  2314 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:11.934010  1587 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:11.937399  1664 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:03:11.933007  1665 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:11.938758  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.947849  1736 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:11.944991  1591 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:11.943482  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:11.926512  1662 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:11.951671  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.952497  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.955174  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:11.960433  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.963137  1741 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:11.962045  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:11.964003  1742 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:11.955394  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:11.981678  1740 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:11.955111  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:11.987563  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44227.7us
I20260709 13:03:11.991117  1662 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:11.956606  1588 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:11.998498  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:11.875664  6355 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:11.999540  1737 raft_consensus.cc:2376] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11. Re-sending same reply.
I20260709 13:03:11.958981  6135 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.015139  1736 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:12.015713  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:12.016968  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:12.017112  6135 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:11.950747  6279 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:11.951391  6456 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:11.947962  6278 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:11.984795  6395 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:11.953809  6104 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:11.898674  6422 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:11.960654  6127 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 13:03:11.962859  6498 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:12.027374  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.026103  6278 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.011718  6516 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:12.022774  6456 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:11.958851  6265 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.020500  6279 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.076516  1665 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:12.001132  6442 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:11.994490  6353 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:03:12.088027  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.119238  1736 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
W20260709 13:03:12.102037  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 113751us
I20260709 13:03:12.044148  6448 raft_consensus.cc:2764] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:12.075980  6326 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.044425  6069 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
W20260709 13:03:12.111254  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 156181us
I20260709 13:03:12.120714  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:12.123739  6138 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.130019  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.131697  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.135636  6355 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.139716  6265 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.141552  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.108558  1668 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:12.117143  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 162132us
I20260709 13:03:12.095279  6279 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:12.048416  6395 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.113842  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.077827  6278 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:12.094568  6442 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.066860  6498 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:12.145035  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 157150us
I20260709 13:03:12.114319  6307 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.150056  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:12.093577  6283 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.160486  6496 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:12.156038  6395 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:12.149304  6486 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.559s	user 0.013s	sys 0.000s
I20260709 13:03:12.158818  6138 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.159106  6422 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:12.161307  6283 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.162164  1666 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:12.154696  6498 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.174306  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.152312  6442 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:12.219457  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:12.188783  6486 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.191512  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:12.198122  6538 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:12.227648  2310 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:03:12.227962  6507 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.786s	user 0.002s	sys 0.005s
I20260709 13:03:12.229698  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1971) took 1142 ms. Trace:
W20260709 13:03:12.229858  6506 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.503s	user 0.004s	sys 0.008s
I20260709 13:03:12.230322  6506 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:12.231200  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.175479  6307 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.230113  1659 rpcz_store.cc:276] 0709 13:03:11.087106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.087248 (+   142us) service_pool.cc:224] Handling call
0709 13:03:11.089922 (+  2674us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.090293 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:11.090325 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:11.090832 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 13:03:11.236899 (+146067us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:11.236946 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:11.237031 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.227291 (+990260us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:12.229337 (+  2046us) raft_consensus.cc:1685] finished
0709 13:03:12.229358 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.229628 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1282,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":51554048,"thread_start_us":145641,"threads_started":1,"wal-append.queue_time_us":278259}
I20260709 13:03:12.233209  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:12.221793  6456 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.234130  1661 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:12.224880  6438 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:12.233376  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:12.235692  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45582.9us
I20260709 13:03:12.238955  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.239840  1659 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:12.240008  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.240103  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:12.237666  1735 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:12.234424  6127 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:12.133435  6326 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.232945  6230 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.253175  6307 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:12.254501  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107219us
I20260709 13:03:12.255769  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:12.203399  6278 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.257786  6353 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:12.258489  6283 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:12.259423  6283 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:12.257841  6305 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:12.261127  6305 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.262521  6278 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:12.263571  6305 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.225613  6536 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:03:12.264815  6536 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:12.265194  6536 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:12.258534  6353 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:12.269907  1661 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:12.240865  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 120262us
I20260709 13:03:12.194871  6496 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:12.272182  6353 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:12.274475  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:12.268082  6138 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:12.279467  1661 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:12.280248  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.281021  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:12.281963  1661 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:12.283061  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:12.283915  6307 raft_consensus.cc:2764] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:03:12.296504  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.298507  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.299527  1734 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:12.298667  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.246721  6537 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:12.300755  6537 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:12.304979  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.307116  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:12.307729  6533 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.227s	user 0.001s	sys 0.001s
I20260709 13:03:12.308306  1734 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:12.313606  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.318034  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.320845  6278 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:12.322841  6496 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:12.326401  6525 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.219s	user 0.000s	sys 0.002s
I20260709 13:03:12.327216  6543 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:03:12.330245  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:12.333786  6506 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:12.335774  6487 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.001s
I20260709 13:03:12.338276  6537 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.340366  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:12.343029  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:12.348979  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.352201  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.352142  1734 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:12.374930  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:12.375905  1593 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:12.384824  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.385741  1740 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:12.387853  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:12.387888  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:12.388960  2314 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:12.389693  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.316802  5898 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.350723  6138 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:12.394583  1733 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:12.394709  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.400231  1585 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:12.402655  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.403995  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:12.405036  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:12.405789  1589 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:12.408437  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:12.409973  1668 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:12.410986  2309 raft_consensus.cc:2368] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:12.419041  1737 raft_consensus.cc:2368] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:12.372195  6540 raft_consensus.cc:2764] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:12.425037  1584 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 13:03:12.341398  6442 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.392547  6278 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:12.350854  6353 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.445736  6138 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:12.447623  1663 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:12.353135  6543 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:03:12.371621  6307 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:03:12.371867  6127 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:03:12.351104  6541 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
W20260709 13:03:12.403244  1632 connection.cc:256] RPC call timeout handler was delayed by -1.3816344489999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260709 13:03:12.482489  1632 connection.cc:256] RPC call timeout handler was delayed by -1.2106344489999969s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21050000000000002s for a response
W20260709 13:03:12.483039  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0270344489999843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23090000000000002s for a response
I20260709 13:03:12.464708  2312 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:12.408392  6537 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:12.444044  6542 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:03:12.498631  1661 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:12.478369  6511 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 13:03:12.502354  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:12.501130  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:12.477919  6307 consensus_queue.cc:1243] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:12.480338  6278 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:12.519055  6541 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:12.519529  6541 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.454064  5898 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.479672  6343 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 13:03:12.519862  6541 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:12.489941  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 135185us
I20260709 13:03:12.531344  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.537187  1661 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:12.543432  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.543799  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.544379  1739 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:12.547257  2313 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:12.547297  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:12.555500  6528 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.413s	user 0.000s	sys 0.005s
I20260709 13:03:12.535329  6506 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:12.560658  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.568838  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.556202  6353 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.549116  6540 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:12.580015  1738 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:12.582064  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:12.585337  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:12.592682  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79860.7us
I20260709 13:03:12.594427  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.595211  1739 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:12.553387  6543 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 13:03:12.601593  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.597573  2280 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:12.606930  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:12.608664  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:12.579430  6540 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.550258  6279 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.628938  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.621728  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.634586  1737 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:12.638695  1739 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:12.639743  1660 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:12.642570  1659 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:12.644440  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:12.646405  6127 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:12.646643  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.647339  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.648423  6544 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:12.575446  6138 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:03:12.650125  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.604619  6353 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:12.651216  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.597213  6278 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.652256  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.653909  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.655150  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.655732  1741 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:12.656056  6395 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:12.656518  6395 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:12.656910  6395 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:12.629880  6343 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:12.661937  6343 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:12.662365  6343 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:12.662902  6543 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:12.663132  1735 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:12.663398  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:12.663496  6543 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.664142  1592 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:12.665267  1663 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:12.649370  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.647950  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:12.667367  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 89605.7us
I20260709 13:03:12.667713  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:12.668066  1742 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:12.668226  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:12.668588  1585 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:12.669319  1742 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:12.677979  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.679591  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:12.600630  6552 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.689282  1585 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:12.681943  1741 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:12.627288  6485 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:12.636765  6540 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:12.626811  6481 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:12.681607  6544 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.685463  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:12.688678  1736 raft_consensus.cc:2368] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:12.694346  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:12.684171  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:12.731050  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45733.3us
I20260709 13:03:12.698619  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2771) took 1417 ms. Trace:
I20260709 13:03:12.700860  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.725663  1662 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260709 13:03:12.743301  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 95427.9us
I20260709 13:03:12.744451  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.734908  1590 rpcz_store.cc:276] 0709 13:03:11.281544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.718019 (+436475us) service_pool.cc:224] Handling call
0709 13:03:11.724758 (+  6739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:11.829851 (+105093us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:11.930305 (+100454us) log.cc:844] Serialized 14 byte log entry
0709 13:03:12.154767 (+224462us) log.cc:844] Serialized 14 byte log entry
0709 13:03:12.379885 (+225118us) log.cc:844] Serialized 14 byte log entry
0709 13:03:12.697939 (+318054us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:12.698037 (+    98us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:12.698106 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.698249 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.698406 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40417,"prepare.queue_time_us":40273,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":119,"raft.queue_time_us":142227,"raft.run_cpu_time_us":692,"raft.run_wall_time_us":43358,"spinlock_wait_cycles":29568,"thread_start_us":726287,"threads_started":3,"wal-append.queue_time_us":323129}
I20260709 13:03:12.681588  6138 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:12.689184  6353 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 13:03:12.749050  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64973.3us
I20260709 13:03:12.756204  1664 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 13:03:12.742787  6069 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:12.690832  6559 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:12.692137  6283 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:12.692349  6127 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:12.753814  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.760742  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.761571  1741 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:12.756340  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 147790us
I20260709 13:03:12.698890  6504 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:12.759706  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:12.763032  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 120858us
I20260709 13:03:12.763870  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.767338  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:12.741170  6485 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:12.769058  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.779227  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.753551  6502 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:12.785597  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:12.780807  1662 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:12.746315  6307 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:12.795532  1741 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:12.798768  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.802094  1741 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [10.7-12.8]
I20260709 13:03:12.755702  6553 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:12.857270  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:12.777817  6506 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:12.778053  6442 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:12.797009  6502 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:12.860708  6552 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:12.747412  6544 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:12.903911  6438 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.656s	user 0.008s	sys 0.003s
I20260709 13:03:12.796937  6138 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.905182  6541 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.907130  6138 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:12.907958  6541 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:12.907820  6138 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:12.870368  6496 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:12.908774  6127 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:12.865304  6561 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:12.909121  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2618) took 1028 ms. Trace:
I20260709 13:03:12.909251  6127 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:12.870859  6502 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.909706  1660 rpcz_store.cc:276] 0709 13:03:11.880481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.643983 (+763502us) service_pool.cc:224] Handling call
0709 13:03:12.644356 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:12.645001 (+   645us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:12.645031 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:12.645978 (+   947us) log.cc:844] Serialized 26 byte log entry
0709 13:03:12.761419 (+115441us) raft_consensus.cc:1645] Marking committed up to 4
0709 13:03:12.761461 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.761555 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.908896 (+147341us) raft_consensus.cc:1685] finished
0709 13:03:12.908919 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.909059 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":229,"raft.queue_time_us":1849,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":115040,"threads_started":1,"wal-append.queue_time_us":250592}
I20260709 13:03:12.910446  6127 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.911213  6502 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:12.911898  6127 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:12.912122  1660 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:03:12.913437  1660 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:12.909080  6496 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.909761  6561 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.918844  6563 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:12.919574  6563 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:12.920293  6496 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:12.921182  6279 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:12.923230  1742 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:12.927055  6557 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:03:12.928599  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1979) took 1119 ms. Trace:
I20260709 13:03:12.928997  1668 rpcz_store.cc:276] 0709 13:03:11.808882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.453747 (+644865us) service_pool.cc:224] Handling call
0709 13:03:12.481997 (+ 28250us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:12.482487 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:12.482526 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:12.483203 (+   677us) log.cc:844] Serialized 52 byte log entry
0709 13:03:12.541739 (+ 58536us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:12.541796 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:12.541884 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:12.928343 (+386459us) raft_consensus.cc:1685] finished
0709 13:03:12.928370 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:12.928537 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":950,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":8253,"thread_start_us":24289,"threads_started":1,"wal-append.queue_time_us":255104}
I20260709 13:03:12.930613  6561 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:12.867743  6278 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:12.932825  6278 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.864089  6559 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.032s
I20260709 13:03:12.934422  6559 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:12.934818  6559 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:12.935772  6559 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.936925  6278 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:12.937932  1737 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:12.872268  5898 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:12.941278  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 178879us
I20260709 13:03:12.937117  6559 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:12.940984  6481 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:12.948139  6481 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.928205  6343 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:12.871922  6550 raft_consensus.cc:2764] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 13:03:12.956934  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:12.958626  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:12.940407  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:12.967155  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.870589  6552 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.977422  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.961706  6485 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.982348  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.983389  1738 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:12.984740  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:12.991880  6343 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:12.997531  6563 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:12.970908  6486 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.011039  6544 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.042977  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:12.968639  6495 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
W20260709 13:03:13.048429  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 281190us
W20260709 13:03:12.969300  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.049204  6544 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:13.062476  1737 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:13.045188  6495 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:13.045171  6541 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260709 13:03:13.098783  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:13.050617  6442 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:03:13.052656  6506 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:13.085556  6544 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:13.109243  1591 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:13.109936  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2742) took 1124 ms. Trace:
I20260709 13:03:13.110307  1591 rpcz_store.cc:276] 0709 13:03:11.985790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.105073 (+1119283us) service_pool.cc:224] Handling call
0709 13:03:13.105464 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:13.109659 (+  4195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.109877 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:13.117090  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:13.118185  1585 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:13.119831  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:13.120146  6567 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:03:13.120457  1737 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:13.121371  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:13.123425  1591 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:13.124806  1591 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:13.126284  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:13.127090  1587 raft_consensus.cc:2368] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10.
I20260709 13:03:13.128937  1737 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:13.129407  1585 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:13.130115  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:13.130962  1590 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:13.062247  6556 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:03:13.107236  6541 consensus_queue.cc:1243] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 13:03:13.143307  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:13.112221  6544 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:13.150436  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 199561us
W20260709 13:03:13.185420  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57321.9us
I20260709 13:03:13.184612   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:03:13.192698  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.151618  6353 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:13.197706  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1968) took 2190 ms. Trace:
W20260709 13:03:13.182183  6516 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.002s	sys 0.023s
I20260709 13:03:13.179845  6550 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:13.202180  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 120447us
I20260709 13:03:13.198896  1667 rpcz_store.cc:276] 0709 13:03:11.000189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.021782 (+ 21593us) service_pool.cc:224] Handling call
0709 13:03:11.035781 (+ 13999us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:11.681458 (+645677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.188781 (+1507323us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:13.190037 (+  1256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1462738688}
I20260709 13:03:13.203102  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 763) took 1101 ms. Trace:
I20260709 13:03:13.151181  6395 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:13.204834  6516 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.096s	user 0.003s	sys 0.023s
I20260709 13:03:13.206941  1659 rpcz_store.cc:276] 0709 13:03:12.102106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.683769 (+581663us) service_pool.cc:224] Handling call
0709 13:03:12.724352 (+ 40583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.724655 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:12.862670 (+138015us) log.cc:844] Serialized 14 byte log entry
0709 13:03:13.198569 (+335899us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:13.198625 (+    56us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:13.198687 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.202888 (+  4201us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:13.203041 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":250548,"prepare.run_cpu_time_us":5143,"prepare.run_wall_time_us":33218,"thread_start_us":446554,"threads_started":3,"wal-append.queue_time_us":317888}
I20260709 13:03:13.180662  6485 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 13:03:13.234397  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:13.247843  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.259956  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73651.3us
I20260709 13:03:13.260331  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2615) took 1388 ms. Trace:
W20260709 13:03:13.240548  6565 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.193s	user 0.000s	sys 0.001s
I20260709 13:03:13.233918  6506 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.227037  6516 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:13.261070  1665 rpcz_store.cc:276] 0709 13:03:11.861888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.955108 (+ 93220us) service_pool.cc:224] Handling call
0709 13:03:13.249988 (+1294880us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:13.250134 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211650304}
W20260709 13:03:13.243120  6537 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.742s	user 0.014s	sys 0.000s
I20260709 13:03:13.235878  6359 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.273933  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:13.276018  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:13.222862  6550 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:13.287130  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:13.301754  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:13.304529  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69471.6us
I20260709 13:03:13.306918  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:13.303785  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:13.308243  1667 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.314240  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:13.270996  6552 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:13.302587  6127 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.400981  6127 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:13.401726  6574 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 13:03:13.352094  6537 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 13:03:13.302121  6359 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:13.404773  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97950.9us
W20260709 13:03:13.400846  6585 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 13:03:13.450937  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:13.274016  6506 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:13.375515  6553 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:13.504063  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:13.511118  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.439143  6485 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:13.534343  6152 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:03:13.537015  6592 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 13:03:13.487123  6343 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.539535  6582 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:13.485704  6496 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:13.439723  6138 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:13.559211  6581 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260709 13:03:13.560900  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2612) took 1750 ms. Trace:
I20260709 13:03:13.510703  6572 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.564028  6504 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.564855  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 1984) took 1288 ms. Trace:
I20260709 13:03:13.566936  6307 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:13.567673  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2613) took 1757 ms (client timeout 1787 ms). Trace:
I20260709 13:03:13.704804  1661 rpcz_store.cc:276] 0709 13:03:12.276950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.282809 (+  5859us) service_pool.cc:224] Handling call
0709 13:03:13.564652 (+1281843us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540013e2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:13.564790 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":494341888}
I20260709 13:03:13.568266  6595 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:13.572063  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.573876  6359 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:13.376607  6353 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:13.751260  2310 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:13.578208  6069 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:13.586870  6456 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:13.588815  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2685) took 1291 ms. Trace:
I20260709 13:03:13.590566  6279 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.594022  6230 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.602527  1659 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:03:13.611142  6536 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:13.612067  6551 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:13.612512  6550 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.440186  6561 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.352429  6586 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.199s
I20260709 13:03:13.614369  6395 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.514290  6516 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:13.616376  6127 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:13.621537  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:13.975385  6230 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.452598  6552 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:13.538558  6579 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
I20260709 13:03:13.635588  6152 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:13.654875  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 420574us
I20260709 13:03:13.643471  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:13.694151  1663 rpcz_store.cc:276] 0709 13:03:11.810052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.497796 (+687744us) service_pool.cc:224] Handling call
0709 13:03:12.644781 (+146985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.674458 (+ 29677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.560538 (+886080us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540013e7d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:13.560796 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":300671360}
I20260709 13:03:13.523391  5898 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:13.708779  1666 rpcz_store.cc:269] 0709 13:03:11.810364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:11.869893 (+ 59529us) service_pool.cc:224] Handling call
0709 13:03:13.567464 (+1697571us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:13.567611 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641270656}
I20260709 13:03:13.717896  1661 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:13.719710  1667 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:03:13.721091  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.820955  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:13.715278  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:13.576355  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:13.826172  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:13.656260  6582 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:14.031786  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.001s	sys 0.039s
W20260709 13:03:13.999357  6559 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.048s	user 0.008s	sys 0.006s
I20260709 13:03:13.832512  1736 rpcz_store.cc:276] 0709 13:03:12.297918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.298386 (+   468us) service_pool.cc:224] Handling call
0709 13:03:13.588512 (+1290126us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:13.588727 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":892102912}
I20260709 13:03:13.836347  1659 pending_rounds.cc:85] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:13.531116  6553 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.045389  1742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.008s	sys 0.033s
W20260709 13:03:14.045931  1742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.009s	sys 0.033s
W20260709 13:03:13.957631  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:14.046727  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:13.826176  6502 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.826650  6359 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:13.710345  6343 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:13.917331  6556 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:13.926266  6551 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:13.985797  6540 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.235s	user 0.018s	sys 0.000s
I20260709 13:03:13.975409  6541 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:13.819274  6307 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:13.998383  1741 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:13.624297  6548 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 1.211s	user 0.007s	sys 0.003s
I20260709 13:03:13.821532  6572 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:13.752149  6543 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:13.995844  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.035s	sys 0.062s
W20260709 13:03:14.079412  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.036s	sys 0.063s
W20260709 13:03:14.079900  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 807220us
I20260709 13:03:13.702817  6496 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.013355  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:13.993294  6536 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:13.925586  6279 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.026713  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 383405us
I20260709 13:03:14.030098  2312 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:14.035336  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.001s	sys 0.039s
W20260709 13:03:14.103561  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 864813us
I20260709 13:03:13.971958  6561 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.029485  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:14.036882  1558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.009s	sys 0.031s
I20260709 13:03:14.039489  2309 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:14.112749  1558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.009s	sys 0.031s
I20260709 13:03:14.043838  1659 pending_rounds.cc:107] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:03:14.113890  1659 raft_consensus.cc:2889] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:14.053292  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.018s	sys 0.036s
I20260709 13:03:14.029316  6152 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:14.053727  1662 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.015s	sys 0.039s
W20260709 13:03:14.117444  1662 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.916s	user 0.015s	sys 0.039s
W20260709 13:03:14.057031  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.011s	sys 0.042s
I20260709 13:03:13.717597  6504 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:13.756006  6138 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.123653  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94309.5us
I20260709 13:03:14.057345  6502 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.125653  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:14.195233  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:14.052460  6599 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:14.036464  6582 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:14.016775  6230 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.083859  1736 pending_rounds.cc:85] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:03:14.204794  1734 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:14.205011  1736 pending_rounds.cc:107] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:14.058241  6359 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:14.057881  6278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.011s	sys 0.041s
W20260709 13:03:14.100917  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 87651.5us
I20260709 13:03:14.101881  1634 leader_election.cc:400] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10.
I20260709 13:03:14.257145  1634 leader_election.cc:403] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 13:03:14.104704  2281 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.257583  1632 leader_election.cc:400] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:14.258057  1632 leader_election.cc:403] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:14.258741  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 154590us
I20260709 13:03:14.037106  5898 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.260135  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:14.114884  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.019s	sys 0.036s
I20260709 13:03:14.112164  6481 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:14.118188  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.012s	sys 0.042s
I20260709 13:03:14.118717  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 1981) took 1970 ms. Trace:
I20260709 13:03:14.281028  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.116124  5881 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:13.649727  6485 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:14.098031  6506 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:14.094308  1741 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 13:03:14.251916  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52459.7us
I20260709 13:03:14.291208  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:14.251848  1736 raft_consensus.cc:2889] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:14.131544  6343 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.029284  6552 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.256703  6278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.992s	user 0.011s	sys 0.041s
W20260709 13:03:14.267176  6603 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 13:03:14.268023  5898 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.268541  6487 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:14.268672  6575 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:14.268955  6279 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.343156  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:14.270088  6152 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.270881  6561 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.081553  6556 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.272140  6230 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:14.166707  6597 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
W20260709 13:03:14.146232  6591 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.611s	user 0.001s	sys 0.001s
I20260709 13:03:14.074179  6541 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:14.166035  6590 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 13:03:14.162237  6563 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.128989  6127 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.365645  6127 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:14.367502  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57083.8us
I20260709 13:03:14.368505  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2746) took 1262 ms. Trace:
I20260709 13:03:14.068620  6540 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.299400  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:14.151633  6586 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:14.151312  6595 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:14.115310  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2622) took 1886 ms. Trace:
I20260709 13:03:14.275161  1662 rpcz_store.cc:276] 0709 13:03:12.148638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.790386 (+641748us) service_pool.cc:224] Handling call
0709 13:03:12.855884 (+ 65498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:12.858029 (+  2145us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:13.088296 (+230267us) log.cc:844] Serialized 14 byte log entry
0709 13:03:14.117802 (+1029506us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:14.118418 (+   616us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:14.118502 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.118585 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.118674 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":186973,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":291347,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":7644,"thread_start_us":1238779,"threads_started":3,"wal-append.queue_time_us":161050}
I20260709 13:03:14.345860  6498 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:14.326619  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.129163  6550 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:14.340689  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.340839  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.207653  6559 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:14.346765  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 248947us
I20260709 13:03:14.347018  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.205010  6538 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.123s
I20260709 13:03:14.355418  6438 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:14.384923  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:14.385736  1741 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:03:14.359807  6561 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.335464  6456 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:03:14.290745  6481 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.128778  6138 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.290907  6600 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:14.299299  6543 raft_consensus.cc:2764] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:03:14.288733  6599 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 13:03:14.363521  6606 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:14.336191  6343 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.364198  6584 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:14.364780  6598 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:14.364549  6563 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.398469  6438 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.440306  6563 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:14.371888  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.441200  6438 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:14.441461  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69704.1us
I20260709 13:03:14.416939  6485 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 10.11, Last appended by leader: 11, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.442358  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.443547  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.444396  6601 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:14.445186  6601 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:14.445297  6609 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.445658  6596 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:14.445910  6609 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.368907  1592 rpcz_store.cc:276] 0709 13:03:13.106897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.107044 (+   147us) service_pool.cc:224] Handling call
0709 13:03:13.134614 (+ 27570us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:13.135062 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.135103 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:13.151597 (+ 16494us) log.cc:844] Serialized 52 byte log entry
0709 13:03:13.251348 (+ 99751us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:03:13.251393 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.251482 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.368246 (+1116764us) raft_consensus.cc:1685] finished
0709 13:03:14.368276 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.368436 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":669,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":7385,"thread_start_us":68821,"threads_started":1,"wal-append.queue_time_us":528412}
W20260709 13:03:14.375084  1589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.013s	sys 0.019s
W20260709 13:03:14.447379  1589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.013s	sys 0.019s
I20260709 13:03:14.375221  2312 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:14.446511  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.136739  6496 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:14.344457  5898 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:14.349112  6334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s	user 0.002s	sys 0.029s
I20260709 13:03:14.376689  1664 rpcz_store.cc:276] 0709 13:03:12.229192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.889791 (+660599us) service_pool.cc:224] Handling call
0709 13:03:12.941965 (+ 52174us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:12.942357 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:12.942386 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:12.946748 (+  4362us) log.cc:844] Serialized 52 byte log entry
0709 13:03:13.144451 (+197703us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:13.184933 (+ 40482us) log.cc:844] Serialized 14 byte log entry
0709 13:03:13.205925 (+ 20992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.206013 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.114929 (+908916us) raft_consensus.cc:1685] finished
0709 13:03:14.114960 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.115223 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53746,"prepare.queue_time_us":44690,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":129672,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":11304,"thread_start_us":176014,"threads_started":2,"wal-append.queue_time_us":205758}
I20260709 13:03:14.111430  6495 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.397218  6559 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:14.398082  1660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.056s	sys 0.138s
I20260709 13:03:14.374506  6230 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.403056  1587 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:14.268553  6502 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.377170  6326 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.421550  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:14.428218  1588 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:03:14.418565  6602 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10.
I20260709 13:03:14.431890  6572 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.432935  6359 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:14.393379  6504 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:14.433213  2318 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 13:03:14.380841  6575 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:03:14.381248  5881 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 13:03:14.398195  6561 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.397403  6550 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.420797  6553 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.381886  6586 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.434998  6481 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:14.395249  6595 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.436592  6606 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.437562  6343 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.451563  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:14.273228  6551 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.448575  6609 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:14.394134  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:14.429152  6544 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.250s	user 0.014s	sys 0.000s
I20260709 13:03:14.415592  6600 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:14.419142  6138 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:14.472064  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.056s	sys 0.138s
I20260709 13:03:14.427189  6279 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:14.467674  6334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.813s	user 0.002s	sys 0.030s
I20260709 13:03:14.522960  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2013) took 1697 ms. Trace:
I20260709 13:03:14.523387  1739 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:14.528661  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.530992  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.531649  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.475291  6536 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.485858  6495 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:14.475621  6611 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:14.496907  6305 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 13:03:14.475530  6230 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.486995  5898 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.498652  6584 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
W20260709 13:03:14.658874  1740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.010s	sys 0.007s
W20260709 13:03:14.659824  1740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.011s	sys 0.007s
W20260709 13:03:14.665897  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:14.667320  1665 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.036s	user 0.005s	sys 0.012s
W20260709 13:03:14.667834  1665 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.005s	sys 0.012s
W20260709 13:03:14.644176  6610 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 13:03:14.551760  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:14.571614  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 120150us
I20260709 13:03:14.589303  1590 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:14.590807  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2014) took 1791 ms. Trace:
I20260709 13:03:14.566299  6600 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:14.615252  6559 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.624727  6536 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.642483  6552 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.569481  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:14.695870  1741 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:14.608642  2316 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:14.588428  6607 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:14.623178  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:14.573614  1737 rpcz_store.cc:276] 0709 13:03:12.799150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.129703 (+330553us) service_pool.cc:224] Handling call
0709 13:03:13.130011 (+   308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:13.146225 (+ 16214us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.146263 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:13.151386 (+  5123us) log.cc:844] Serialized 26 byte log entry
0709 13:03:13.228988 (+ 77602us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:13.229033 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:13.229113 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.145537 (+916424us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:14.489881 (+344344us) raft_consensus.cc:1685] finished
0709 13:03:14.489911 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.496948 (+  7037us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8871,"raft.queue_time_us":8655,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2629,"spinlock_wait_cycles":230177664,"thread_start_us":50824,"threads_started":1,"wal-append.queue_time_us":330986}
I20260709 13:03:14.556888  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:14.699739  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.542142  6359 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.634003  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2634) took 1033 ms. Trace:
I20260709 13:03:14.673278  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:14.541787  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:14.684408  1662 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:14.686184  1587 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:14.684491  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.688702  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.627622  6504 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:14.674067  1742 rpcz_store.cc:276] 0709 13:03:12.799417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.261337 (+461920us) service_pool.cc:224] Handling call
0709 13:03:13.261782 (+   445us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:13.264431 (+  2649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.264476 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:13.272437 (+  7961us) log.cc:844] Serialized 78 byte log entry
0709 13:03:14.046515 (+774078us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:14.540474 (+493959us) log.cc:844] Serialized 14 byte log entry
0709 13:03:14.589571 (+ 49097us) log.cc:844] Serialized 14 byte log entry
0709 13:03:14.589914 (+   343us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.590021 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.590082 (+    61us) raft_consensus.cc:1685] finished
0709 13:03:14.590100 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.590305 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48360,"prepare.queue_time_us":617816,"raft.queue_time_us":49430,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":34690,"spinlock_wait_cycles":7938688,"thread_start_us":1129804,"threads_started":2,"wal-append.queue_time_us":1058771}
I20260709 13:03:14.573132  6138 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:14.669451  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:14.702562  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:14.665868  6540 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:14.552052  6580 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.143s
W20260709 13:03:14.699970  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 143215us
I20260709 13:03:14.734380  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 692) took 1281 ms. Trace:
I20260709 13:03:14.573287  6279 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.572649  6544 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:14.696444  6566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.717s	user 0.018s	sys 0.020s
I20260709 13:03:14.704550  1660 rpcz_store.cc:276] 0709 13:03:13.601049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.604363 (+  3314us) service_pool.cc:224] Handling call
0709 13:03:13.604740 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:13.605189 (+   449us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.605233 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:13.653058 (+ 47825us) log.cc:844] Serialized 26 byte log entry
0709 13:03:14.571215 (+918157us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:14.571279 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.571380 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:14.633647 (+ 62267us) raft_consensus.cc:1685] finished
0709 13:03:14.633683 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.633916 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4783,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"thread_start_us":741544,"threads_started":1,"wal-append.queue_time_us":881627}
W20260709 13:03:14.799394  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.018s	sys 0.033s
W20260709 13:03:14.799893  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.018s	sys 0.033s
I20260709 13:03:14.800053  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.749655  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:14.752213  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:14.695514  6541 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:14.800971  1660 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:14.810758  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:14.752173  1589 rpcz_store.cc:276] 0709 13:03:13.453198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.641721 (+188523us) service_pool.cc:224] Handling call
0709 13:03:13.642278 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:13.642570 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:14.447756 (+805186us) log.cc:844] Serialized 14 byte log entry
0709 13:03:14.733920 (+286164us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:14.733979 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:14.734030 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.734107 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:14.734294 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":491804,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"thread_start_us":1040791,"threads_started":2}
I20260709 13:03:14.703814  6583 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:14.755838  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:14.647612  6611 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.792623  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:14.706367  6548 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.010s	sys 0.005s
I20260709 13:03:14.813834  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:14.849030  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 310594us
I20260709 13:03:14.857882  1589 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:14.858703  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 106595us
I20260709 13:03:14.708104  6496 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:14.859586  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.706763  6552 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:14.864280  1586 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:14.711889  6359 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.709954  6628 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:14.862102  1660 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:14.888540  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:14.799001  6610 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 13:03:14.799301  6566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.827s	user 0.019s	sys 0.020s
I20260709 13:03:14.709312  6506 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.883198  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:14.901638  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 414079us
W20260709 13:03:14.863124  6548 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.010s	sys 0.009s
I20260709 13:03:14.889380  1660 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:14.906160  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.900586  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.856443  6609 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:14.915227  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:14.930774  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.880419  6541 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:14.932426  1662 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:14.733405  6607 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:14.747119  6504 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:14.936250  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.954798  1660 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:14.877151  6540 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.907167  1659 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 8.
I20260709 13:03:14.956037  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.945032  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:14.934445  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:14.949565  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.885466  6583 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:14.911581  6438 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:14.995967  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:14.987707  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:15.027097  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.028304  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.911159  6442 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.033061  1659 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
W20260709 13:03:15.036257  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 91376.2us
W20260709 13:03:15.044543  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.052570  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.054994  1660 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:15.057804  1589 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:15.059962  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.060173  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:15.060828  1589 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:15.063045  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.063654  1589 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:15.064198  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:15.065948  1589 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 13:03:15.073738  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 779) took 1627 ms (client timeout 2025 ms). Trace:
I20260709 13:03:15.075294  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:14.933406  6550 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.078286  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:15.074249  1734 rpcz_store.cc:269] 0709 13:03:13.446810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.576039 (+129229us) service_pool.cc:224] Handling call
0709 13:03:15.062440 (+1486401us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:15.073659 (+ 11219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378945280}
W20260709 13:03:15.083362  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 281864us
I20260709 13:03:15.076216  1662 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:15.082219  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:14.915920  6506 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.076396  6569 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:15.051517  6343 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:14.894387  6605 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.182s
I20260709 13:03:15.090553  6612 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:15.092106  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.092981  1661 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:15.117245  1659 raft_consensus.cc:2410] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:03:15.093915  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2635) took 1492 ms. Trace:
I20260709 13:03:15.096199  1662 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 13:03:15.118567  1662 pending_rounds.cc:85] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:03:15.099789  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:14.979929  5881 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:15.100754  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:15.104159  1666 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 13:03:15.090071  6614 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.476s	user 0.001s	sys 0.001s
I20260709 13:03:15.125065  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:14.914734  6556 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.142304  6556 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:15.142807  6556 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:15.143230  6556 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:15.143873  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 209528us
I20260709 13:03:14.958724  6601 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.138s
I20260709 13:03:15.103626  6566 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:15.118901  1662 pending_rounds.cc:107] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:03:15.118435  1665 rpcz_store.cc:276] 0709 13:03:13.601341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.628971 (+ 27630us) service_pool.cc:224] Handling call
0709 13:03:13.629384 (+   413us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:13.629823 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.629854 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:13.630515 (+   661us) log.cc:844] Serialized 52 byte log entry
0709 13:03:14.668450 (+1037935us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:14.668512 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:14.668677 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.093684 (+425007us) raft_consensus.cc:1685] finished
0709 13:03:15.093708 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.093856 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5428,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4400,"thread_start_us":1037415,"threads_started":1,"wal-append.queue_time_us":1254353}
I20260709 13:03:15.146148  1662 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:14.901243  6628 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.136273  1661 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:03:15.146739  1662 pending_rounds.cc:107] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 13:03:15.147207  1662 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:15.119982  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2637) took 1479 ms. Trace:
I20260709 13:03:14.959523  6552 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.086237  6609 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:15.149942  6609 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.150348  6609 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:15.149207  1668 rpcz_store.cc:276] 0709 13:03:13.640545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.751130 (+110585us) service_pool.cc:224] Handling call
0709 13:03:13.836652 (+ 85522us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:13.836872 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:14.371093 (+534221us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.117694 (+746601us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:15.117903 (+   209us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:15.118110 (+   207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.119351 (+  1241us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.119778 (+   427us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31375,"prepare.queue_time_us":771458,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":47,"thread_start_us":1139564,"threads_started":3}
I20260709 13:03:15.146100  6566 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.151831  6566 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:15.056493  6540 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:15.127643  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 454465us
I20260709 13:03:15.190358  1632 leader_election.cc:400] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:15.190721  1632 leader_election.cc:403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:15.191669  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46200.5us
I20260709 13:03:15.055918  6438 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.070210  6442 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.192126  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91464.3us
I20260709 13:03:15.099673  6569 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.196818  1663 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 13:03:15.196239  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 300307us
I20260709 13:03:15.112025  6538 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:15.105154  6606 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.116353  6230 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.126674  6584 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:15.126256  6550 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.197896  6536 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:15.222378  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97433.9us
I20260709 13:03:15.218099  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:15.135725  6326 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:15.135841  6612 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:15.211310  6647 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:15.198630  6343 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.198547  5898 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.209993  6569 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:15.214246  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.698s (SENT) [suppressed 10 similar messages]
I20260709 13:03:15.261742  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:15.270179  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52187.2us
I20260709 13:03:15.132153  5881 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.272433  2309 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:15.210161  6538 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:15.287879  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73639.3us
I20260709 13:03:15.278182  6230 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:15.233716  6646 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election won for term 10
W20260709 13:03:15.310636  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.306768  6623 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.307040  6230 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:15.283427  6614 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:03:15.327795  6663 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:15.328406  6663 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:15.328518  6649 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:15.328866  6663 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:15.329252  6552 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.269472  6326 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.329864  6641 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 13:03:15.329907  6645 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:15.265128  6596 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.330677  6645 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.330809  6596 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:15.331663  6633 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.245s	user 0.001s	sys 0.000s
W20260709 13:03:15.281164  6582 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.008s	user 0.014s	sys 0.000s
I20260709 13:03:15.334348  5881 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.334136  6582 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.335214  5881 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11.
I20260709 13:03:15.335902  5881 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.336405  6582 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.333276  6596 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.314265  6230 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:15.303316  5898 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.338837  6438 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.338929  6651 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:15.339792  6552 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:15.340516  6651 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:15.340878  6646 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:15.269958  6138 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.341992  6646 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.336440  5881 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.342427  6552 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:15.344354  6612 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.330890  6607 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.270495  6544 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:15.284981  6630 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 13:03:15.272266  6536 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.304097  6605 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.271987  6647 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:15.333185  6645 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:15.337131  6625 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260709 13:03:15.308126  6538 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:15.265394  6550 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.254097  6584 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.320271  6611 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.351603  6279 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.283666  6343 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.320618  6586 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.345082  6552 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:15.413766  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110061us
I20260709 13:03:15.391672  6653 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:15.418684  1632 connection.cc:256] RPC call timeout handler was delayed by -1.3705499989999907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 13:03:15.410137  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 202007us
I20260709 13:03:15.402089  6649 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:15.407060  6359 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.429000  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:15.426532  1632 connection.cc:256] RPC call timeout handler was delayed by -1.1770499990000189s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
I20260709 13:03:15.314227  6623 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.424250  6620 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260709 13:03:15.417929  6596 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:15.432536  1632 connection.cc:256] RPC call timeout handler was delayed by -1.132049999000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
I20260709 13:03:15.436023  6662 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:15.436332  6506 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.401535  6536 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.406718  6556 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.407850  6605 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.427332  6586 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.472023  1632 leader_election.cc:400] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:15.434640  6504 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.447441  6279 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.496068  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:15.442428  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:15.518087  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 693) took 1696 ms. Trace:
I20260709 13:03:15.439179  6540 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.434839  6359 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:15.522053  1593 rpcz_store.cc:276] 0709 13:03:13.821265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.822823 (+  1558us) service_pool.cc:224] Handling call
0709 13:03:13.835966 (+ 13143us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.113512 (+277546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:14.113556 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:14.113622 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:14.113669 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.517744 (+1404075us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b5400431a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:15.517818 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.517998 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":637667200}
I20260709 13:03:15.429451  6653 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.539673  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 43730.7us
I20260709 13:03:15.474188  6659 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.507617  6586 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.546569  6561 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.547953  5881 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:15.549220  6648 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
W20260709 13:03:15.551903  6495 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.910s	user 0.007s	sys 0.004s
I20260709 13:03:15.558045  6551 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:15.536474  6359 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.609313  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.561468  6643 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:15.449249  6623 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.494807  6647 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.567004  6605 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.446951  6649 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.586407  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45547.3us
I20260709 13:03:15.632122  1742 raft_consensus.cc:2410] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 13:03:15.590161  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 147840us
W20260709 13:03:15.590680  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 121266us
I20260709 13:03:15.472256  6596 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.607028  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.524741  6442 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.644390  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:15.648345  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:15.650964  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2645) took 1311 ms. Trace:
I20260709 13:03:15.651381  1664 rpcz_store.cc:276] 0709 13:03:14.339123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.587270 (+1248147us) service_pool.cc:224] Handling call
0709 13:03:15.650897 (+ 63627us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:15.652113  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:15.657295  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:15.659950  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:15.588472  6127 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.595189  6638 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:15.532714  6279 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.677212  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:15.531808  6609 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.679687  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:15.683907  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:15.683842  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125365us
I20260709 13:03:15.697461  6673 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.698253  6673 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.700997  6673 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.701874  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:15.702387  6556 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.702639  1591 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:15.703297  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.703755  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:15.704525  1591 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:15.704878  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.705624  6127 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 13:03:15.705911  6540 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:15.706240  6584 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.706514  6540 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:15.707036  6540 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:15.707108  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2648) took 1234 ms. Trace:
I20260709 13:03:15.707808  6653 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.707580  1661 rpcz_store.cc:276] 0709 13:03:14.472518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.147403 (+674885us) service_pool.cc:224] Handling call
0709 13:03:15.147790 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.193313 (+ 45523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.193352 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.193952 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.392758 (+198806us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:15.392826 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.392913 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.706869 (+313956us) raft_consensus.cc:1685] finished
0709 13:03:15.706894 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.707041 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45016,"raft.queue_time_us":186787,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":89,"thread_start_us":139839,"threads_started":1,"wal-append.queue_time_us":492246}
I20260709 13:03:15.711057  6279 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.711552  6359 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.605705  6659 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.662802  6640 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:15.704525  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:15.714571  1592 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:15.663623  6643 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:15.715646  6643 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.702534  6586 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.680532  6634 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:15.713414  6609 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.717060  6634 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:15.717954  6634 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:15.720760  6612 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:15.722820  6612 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:15.724675  6612 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:15.726524  6550 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:15.727831  6498 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.339s	user 0.007s	sys 0.008s
I20260709 13:03:15.655512  6569 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.731655  6569 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.737313  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2017) took 2937 ms. Trace:
I20260709 13:03:15.737886  1741 rpcz_store.cc:276] 0709 13:03:12.800254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.021235 (+2220981us) service_pool.cc:224] Handling call
0709 13:03:15.077320 (+ 56085us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.086805 (+  9485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.086841 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.112159 (+ 25318us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.416369 (+304210us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:15.416426 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.416539 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.736969 (+320430us) raft_consensus.cc:1685] finished
0709 13:03:15.737002 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.737226 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8974,"raft.queue_time_us":14332,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":13675,"thread_start_us":216042,"threads_started":1,"wal-append.queue_time_us":471285}
I20260709 13:03:15.740559  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.742726  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2756) took 1342 ms. Trace:
I20260709 13:03:15.743167  1585 rpcz_store.cc:276] 0709 13:03:14.400755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.403612 (+  2857us) service_pool.cc:224] Handling call
0709 13:03:14.404005 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:14.802448 (+398443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:14.802497 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:14.851735 (+ 49238us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.136768 (+285033us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:15.136814 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.136887 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.742453 (+605566us) raft_consensus.cc:1685] finished
0709 13:03:15.742482 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.742658 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115972,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":8288,"thread_start_us":600389,"threads_started":2,"wal-append.queue_time_us":783179}
W20260709 13:03:15.751808  6674 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 13:03:15.763820  6611 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.764715  6611 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.741331  1741 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
W20260709 13:03:15.779621  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 133145us
I20260709 13:03:15.597568  6552 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.783963  1585 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:15.794576  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.792191  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2717) took 2969 ms. Trace:
W20260709 13:03:15.797941  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 77648.4us
I20260709 13:03:15.796638  1734 rpcz_store.cc:276] 0709 13:03:12.823042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.084919 (+2261877us) service_pool.cc:224] Handling call
0709 13:03:15.085351 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.101676 (+ 16325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.101709 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.104312 (+  2603us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.305556 (+201244us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:15.407689 (+102133us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.791623 (+383934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:15.791775 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:15.791849 (+    74us) raft_consensus.cc:1685] finished
0709 13:03:15.791872 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:15.792103 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8448,"prepare.queue_time_us":109298,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":113904,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":15368,"thread_start_us":480042,"threads_started":2,"wal-append.queue_time_us":514810}
I20260709 13:03:15.799167  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.800174  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.801189  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.802233  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.802788  1734 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:15.803602  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.804153  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:15.804940  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.706032  6127 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:15.807717  1588 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 13:03:15.812176  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.812855  1741 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:15.805429  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:15.818586  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.820247  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.819298  1588 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:15.823784  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:15.820948  1734 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:15.813156  6127 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:15.828006  1592 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:15.832614  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.832815  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.843429  6596 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.846172  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:15.846820  1734 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:15.852223  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:15.854027  6600 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.155s	user 0.014s	sys 0.000s
I20260709 13:03:15.854509  6600 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.856869  6600 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.838625  6230 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.863473  6663 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.001s
I20260709 13:03:15.864019  6230 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.863955  6663 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.865725  1589 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:15.875973  1733 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:15.878046  6609 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.879622  6609 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.880681  6659 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.880662  6634 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:15.882704  6566 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.730s	user 0.020s	sys 0.015s
I20260709 13:03:15.883117  6566 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.886747  6307 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.887986  6566 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.876729  6456 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.890039  6456 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:15.875608  5898 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.004s
I20260709 13:03:15.853576  6343 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.896586  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:15.898824  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48872us
I20260709 13:03:15.888223  6611 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:15.904716  6634 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:15.905701  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:15.897454  5898 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:15.928784  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.929622  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:15.936918  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:15.946722  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:15.947376  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.948122  1664 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:15.952031  1591 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:15.955009  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:15.957079  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.957674  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.959741  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:15.959654  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.960546  1592 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:15.961313  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.961436  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.962203  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:15.951607  6663 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:15.965476  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.964957  6676 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 13:03:15.967635  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.969002  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.969676  1659 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:15.973794  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.975337  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.975323  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.975986  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:15.976734  1659 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:15.977015  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.977757  6556 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:15.980105  6127 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:15.981354  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:15.981541  1588 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:15.982698  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.983697  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.984427  1659 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 13:03:15.985589  6438 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:15.986128  6438 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:15.985898  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.990298  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:15.992312  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:15.992827  1667 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:15.993029  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2025) took 1701 ms. Trace:
I20260709 13:03:15.993407  1742 rpcz_store.cc:276] 0709 13:03:14.292129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.749133 (+457004us) service_pool.cc:224] Handling call
0709 13:03:15.992810 (+1243677us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400352fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:15.992971 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909340032}
I20260709 13:03:15.995241  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:15.965371  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:15.995782  1659 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:15.997284  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.000306  1659 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:15.997735  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:15.999035  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.008054  1661 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:16.009342  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.011449  1667 raft_consensus.cc:2410] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 13:03:16.014813  1742 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:16.015012  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.018784  1664 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:03:15.958163  6649 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:15.959165  6617 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:03:15.998780  6609 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.037379  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.027410  6127 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.030234  6152 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.035368  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:16.041852  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2019) took 3102 ms (client timeout 2216 ms). Trace:
I20260709 13:03:16.044021  1592 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:16.007949  6551 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:16.013995  6279 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:16.053629  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:16.043689  1738 rpcz_store.cc:269] 0709 13:03:12.932166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.984427 (+ 52261us) service_pool.cc:224] Handling call
0709 13:03:16.034456 (+3050029us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:16.034581 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3957161728}
I20260709 13:03:16.064463  1742 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:16.070708  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.071211  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.071797  1742 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.071514  1738 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:16.036621  6617 consensus_queue.cc:1243] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:16.073344  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.077371  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.082891  1592 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:16.086692  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.042637  6638 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:16.083587  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:16.087491  1591 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:16.089854  1734 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:16.062206  6438 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:16.061870  6551 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:16.092175  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.093132  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:16.090454  1738 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:16.094977  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.096218  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.085830  6506 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.093632  1589 raft_consensus.cc:2376] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11. Re-sending same reply.
I20260709 13:03:16.098641  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.099272  1591 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:16.099315  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:16.100302  1742 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:16.101164  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.097251  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.103127  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.102522  1738 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:16.054514  6649 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.105280  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.053903  6540 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.111078  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.111898  1592 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:16.107297  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.107638  1738 raft_consensus.cc:2410] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 10.
I20260709 13:03:16.094141  6551 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.110991  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.114136  1591 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:16.106012  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.113405  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.116379  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.118490  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.117892  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.122225  1585 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:16.127354  1738 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:16.127473  1592 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:16.118220  1663 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:16.126303  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.127807  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.106742  6127 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:16.130964  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:16.122270  6538 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.703s	user 0.007s	sys 0.007s
I20260709 13:03:16.106894  6506 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.151927  6540 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.152546  6506 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.153692  6649 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.155382  6628 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.157464  6538 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.161948  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.161366  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.163561  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.136492  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.148015  6575 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:16.149765  6595 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.169576  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.170961  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.173153  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.173486  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:16.163101  6695 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:03:16.168565  6575 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:16.177433  6634 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:16.175540  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:16.173925  6540 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:16.178947  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.184725  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.185847  1742 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 13:03:16.187130  6307 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:16.188520  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.186091  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.178467  6638 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:16.187626  6307 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:16.190522  6575 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.181950  6634 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.175612  6651 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:16.183483  6540 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:16.198287  6538 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.199625  6570 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:16.204023  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.211570  1591 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 13:03:16.223488  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:16.224620  1591 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.158919  6127 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.213776  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.219573  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.187809  6343 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:16.215751  6307 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:16.225715  1742 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:16.226140  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.225768  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.212075  6566 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.215917  6623 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:16.212363  6638 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:16.221208  6634 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:16.222190  6651 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:16.232586  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.234294  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2718) took 3367 ms. Trace:
I20260709 13:03:16.246094  1742 rpcz_store.cc:276] 0709 13:03:12.878966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.225097 (+3346131us) service_pool.cc:224] Handling call
0709 13:03:16.225386 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.234026 (+  8640us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.234235 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.222733  6540 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:16.242069  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.247033  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:16.228991  6575 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.236413  6646 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:16.247288  2316 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:16.242978  6623 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:16.244944  6638 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:16.243623  6566 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.274456  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.391695  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.384927  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.248232  6651 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.252094  6628 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:16.360666  1738 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
W20260709 13:03:16.327996  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 547520us
I20260709 13:03:16.398793  2314 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:16.334770  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2755) took 1934 ms. Trace:
I20260709 13:03:16.329586  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:16.398041  1665 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:16.318517  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.402201  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 775) took 2958 ms. Trace:
I20260709 13:03:16.404903  2318 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 13:03:16.296072  1591 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:16.453918  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.455120  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:16.479743  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:16.485621  1742 rpcz_store.cc:276] 0709 13:03:13.443623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.249589 (+2805966us) service_pool.cc:224] Handling call
0709 13:03:16.249922 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.250145 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:16.251257 (+  1112us) log.cc:844] Serialized 14 byte log entry
0709 13:03:16.401826 (+150569us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:16.401884 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:16.401933 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.402004 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.402146 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":994,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":1774,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":51185,"thread_start_us":148374,"threads_started":1,"wal-append.queue_time_us":234367}
I20260709 13:03:16.476282  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:16.489776  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2752) took 3363 ms. Trace:
I20260709 13:03:16.469942  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:16.490211  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 777) took 3045 ms. Trace:
I20260709 13:03:16.470697  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:16.490206  1586 rpcz_store.cc:276] 0709 13:03:13.121317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.122389 (+  1072us) service_pool.cc:224] Handling call
0709 13:03:13.122676 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.867227 (+1744551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.484412 (+1617185us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400330c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:16.484639 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4004972288}
I20260709 13:03:16.492574  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.492375  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:16.417301  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.493037  1742 rpcz_store.cc:276] 0709 13:03:13.445248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.489340 (+3044092us) service_pool.cc:224] Handling call
0709 13:03:16.489696 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.489902 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.489929 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:16.489972 (+    43us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:16.490001 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.490055 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.490157 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.275717  6623 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:16.463598  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 776) took 3019 ms. Trace:
I20260709 13:03:16.503336  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.456167  1591 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:16.504709  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2706) took 3857 ms (client timeout 2222 ms). Trace:
I20260709 13:03:16.494782  6032 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:16.478960  6609 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 13:03:16.371002  6606 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:16.497797  1590 rpcz_store.cc:276] 0709 13:03:14.400425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.403513 (+  3088us) service_pool.cc:224] Handling call
0709 13:03:14.428720 (+ 25207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.673724 (+245004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.334407 (+1660683us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54003b79d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:16.334681 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":368005376}
I20260709 13:03:16.455881  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.404069  6305 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:16.456204  6343 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:16.503543  1738 rpcz_store.cc:276] 0709 13:03:13.444464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.461318 (+3016854us) service_pool.cc:224] Handling call
0709 13:03:16.462909 (+  1591us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.463148 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.463188 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:16.463248 (+    60us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:16.463292 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.463362 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.463510 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:16.505120  1739 rpcz_store.cc:269] 0709 13:03:12.647684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.652015 (+  4331us) service_pool.cc:224] Handling call
0709 13:03:16.504535 (+3852520us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:16.504641 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4111189888}
I20260709 13:03:16.372067  6326 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.273156  6646 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:16.514464  1739 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:16.498227  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 778) took 3052 ms. Trace:
I20260709 13:03:16.506990  1737 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:16.453018  6628 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:16.331820  6634 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.519666  1742 rpcz_store.cc:276] 0709 13:03:13.446029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.496864 (+3050835us) service_pool.cc:224] Handling call
0709 13:03:16.497626 (+   762us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.497905 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.497937 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:16.497982 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:16.498013 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.498068 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.498173 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.523193  1738 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:16.525924  1737 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:16.526583  6700 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 13:03:16.526675  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2022) took 2235 ms. Trace:
I20260709 13:03:16.527179  1737 rpcz_store.cc:276] 0709 13:03:14.291177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.522107 (+2230930us) service_pool.cc:224] Handling call
0709 13:03:16.525719 (+  3612us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.526375 (+   656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.526616 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:16.528195  6702 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 13:03:16.506639  6127 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.520720  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 783) took 2246 ms. Trace:
I20260709 13:03:16.508330  6032 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:16.533005  1739 rpcz_store.cc:276] 0709 13:03:14.274704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.513679 (+2238975us) service_pool.cc:224] Handling call
0709 13:03:16.514050 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.519565 (+  5515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.520659 (+  1094us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.517104  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.535390  6566 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.541198  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2643) took 2202 ms. Trace:
I20260709 13:03:16.518780  6710 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:16.541746  1660 rpcz_store.cc:276] 0709 13:03:14.338519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.955687 (+617168us) service_pool.cc:224] Handling call
0709 13:03:16.540992 (+1585305us) spinlock_profiling.cc:243] Waited 97.3 ms on lock 0x7b540013ced4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:16.541123 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":223875456}
I20260709 13:03:16.543066  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:16.543697  1660 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:16.528651  1737 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:16.518573  6536 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.523722  6628 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.555979  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2026) took 2263 ms. Trace:
I20260709 13:03:16.581938  1737 rpcz_store.cc:276] 0709 13:03:14.292410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.528111 (+2235701us) service_pool.cc:224] Handling call
0709 13:03:16.528460 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.550789 (+ 22329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.555886 (+  5097us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.558364  6540 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.560108  1742 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:16.531620  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 780) took 3084 ms. Trace:
I20260709 13:03:16.587280  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2023) took 2295 ms. Trace:
I20260709 13:03:16.587853  1742 rpcz_store.cc:276] 0709 13:03:14.291512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.524683 (+2233171us) service_pool.cc:224] Handling call
0709 13:03:16.554928 (+ 30245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.586886 (+ 31958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.587199 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.561059  1668 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:16.533964  1590 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:16.508497  6305 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.584939  1738 rpcz_store.cc:276] 0709 13:03:13.447560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.510542 (+3062982us) service_pool.cc:224] Handling call
0709 13:03:16.522989 (+ 12447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.531318 (+  8329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.531556 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.515718  6606 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.607596  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:16.615332  1738 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:16.619840  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:16.620265  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 149611us
I20260709 13:03:16.624132  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:16.638270  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 788) took 1537 ms. Trace:
I20260709 13:03:16.639200  1737 rpcz_store.cc:276] 0709 13:03:15.100882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.620530 (+1519648us) service_pool.cc:224] Handling call
0709 13:03:16.637171 (+ 16641us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.637765 (+   594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.637838 (+    73us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:16.637905 (+    67us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:16.637956 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.638035 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.638185 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.642266  1590 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:03:16.646615  1668 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:16.647997  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.650534  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 787) took 1747 ms. Trace:
I20260709 13:03:16.652163  1738 rpcz_store.cc:276] 0709 13:03:14.903133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.614636 (+1711503us) service_pool.cc:224] Handling call
0709 13:03:16.615067 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.621867 (+  6800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.650390 (+ 28523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.634101  6485 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 13:03:16.659787  6634 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.656068  6611 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 13:03:16.660804  6634 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:16.661259  6611 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:16.661737  6611 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:16.661885  6634 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:16.662300  6634 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:16.662679  6611 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.663056  6634 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.630059  6710 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:16.664290  6611 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:16.634792  6127 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:16.665518  6127 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:16.667733  6127 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:16.672116  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:16.673955  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:16.678838  6305 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.664294  6634 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:16.679351  6032 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.684823  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.685212  6127 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.688954  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.689810  2318 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:16.693732  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.689276  6634 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:16.664669  6710 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.697640  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.700019  1738 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 13:03:16.700484  1738 pending_rounds.cc:85] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:03:16.700730  1738 pending_rounds.cc:107] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:16.701000  1738 raft_consensus.cc:2889] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:16.700152  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:16.704452  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2015) took 3904 ms. Trace:
I20260709 13:03:16.705731  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.708796  1663 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:16.711905  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:16.706516  1740 rpcz_store.cc:276] 0709 13:03:12.799692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.485569 (+685877us) service_pool.cc:224] Handling call
0709 13:03:13.512534 (+ 26965us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:13.651182 (+138648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:13.651236 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:13.690548 (+ 39312us) log.cc:844] Serialized 26 byte log entry
0709 13:03:14.141275 (+450727us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b1800083c80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:14.660528 (+519253us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:14.660591 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.704072 (+2043481us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400403b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:16.704138 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:16.704188 (+    50us) raft_consensus.cc:1685] finished
0709 13:03:16.704202 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.704362 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106729,"raft.queue_time_us":8509,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":102269,"spinlock_wait_cycles":335920256,"thread_start_us":518963,"threads_started":1,"wal-append.queue_time_us":1228597,"wal-append.run_cpu_time_us":3444,"wal-append.run_wall_time_us":1446940}
W20260709 13:03:16.763753  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 212361us
I20260709 13:03:16.769757  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:16.769802  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.771049  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.773530  2309 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:16.760735  1737 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 13:03:16.775568  1737 pending_rounds.cc:85] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 5
I20260709 13:03:16.776064  1737 pending_rounds.cc:107] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 13:03:16.776484  1737 raft_consensus.cc:2889] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:16.777362  1660 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:16.781168  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.698922  6127 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.779681  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2721) took 1560 ms. Trace:
I20260709 13:03:16.783516  1738 rpcz_store.cc:276] 0709 13:03:15.219054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.699296 (+1480242us) service_pool.cc:224] Handling call
0709 13:03:16.699628 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.779374 (+ 79746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.779598 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.772887  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:16.784343  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.786237  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.677281  6307 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.787998  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.774336  6516 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:16.715986  6506 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 13:03:16.789875  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:16.777474  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2720) took 1558 ms. Trace:
I20260709 13:03:16.795457  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.799823  1737 rpcz_store.cc:276] 0709 13:03:15.218748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.699205 (+1480457us) service_pool.cc:224] Handling call
0709 13:03:16.712897 (+ 13692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.777185 (+ 64288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.777413 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.792637  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.802594  1738 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:16.658279  6646 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.778524  6715 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:16.812079  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.816159  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2760) took 2414 ms. Trace:
I20260709 13:03:16.818818  1587 rpcz_store.cc:276] 0709 13:03:14.402007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.474275 (+ 72268us) service_pool.cc:224] Handling call
0709 13:03:14.474584 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.708524 (+233940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.815905 (+2107381us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54003b1fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:16.816097 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":485733248}
I20260709 13:03:16.819384  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.820308  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:16.821017  1587 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:16.816478  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 694) took 2994 ms. Trace:
I20260709 13:03:16.823700  1740 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:16.821807  1584 rpcz_store.cc:276] 0709 13:03:13.821685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:13.832238 (+ 10553us) service_pool.cc:224] Handling call
0709 13:03:13.952862 (+120624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:14.253667 (+300805us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:14.288786 (+ 35119us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.060099 (+771313us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:15.060164 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:15.060218 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.816221 (+1756003us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400443754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:16.816273 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:16.816417 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46995,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":484172,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10946,"spinlock_wait_cycles":691085056,"thread_start_us":770800,"threads_started":2,"wal-append.queue_time_us":578459,"wal-append.run_cpu_time_us":4659,"wal-append.run_wall_time_us":1523716}
I20260709 13:03:16.829197  1663 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:16.811295  6712 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:16.816340  1737 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:16.841847  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2722) took 1622 ms. Trace:
I20260709 13:03:16.842518  1737 rpcz_store.cc:276] 0709 13:03:15.219306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.801855 (+1582549us) service_pool.cc:224] Handling call
0709 13:03:16.815975 (+ 14120us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.841416 (+ 25441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.841761 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:16.800770  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:16.816216  6506 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:16.854802  1584 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:16.862120  6506 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.864859  6540 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.866109  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:16.838583  6712 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:16.878468  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:16.871935  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.845554  6516 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:16.899266  1584 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:03:16.874207  6713 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:03:16.867148  6506 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:16.914060  1584 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:16.908385  6575 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.657s	user 0.013s	sys 0.000s
I20260709 13:03:16.876569  6710 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.925611  6606 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:16.926683  6551 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.809s	user 0.006s	sys 0.006s
I20260709 13:03:16.927109  6551 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.927999  6638 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:16.938476  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.942106  1741 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:16.944514  1666 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:16.933670  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2696) took 4389 ms (client timeout 1934 ms). Trace:
W20260709 13:03:16.948634  1733 rpcz_store.cc:269] 0709 13:03:12.543750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.545225 (+  1475us) service_pool.cc:224] Handling call
0709 13:03:16.933358 (+4388133us) spinlock_profiling.cc:243] Waited 3.16 s on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:16.933552 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7270908800}
I20260709 13:03:16.929423  6551 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:16.934638  6638 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.958109  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:16.937959  1735 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:16.878960  1740 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:16.968207  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.973708  2316 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:16.885088  1738 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:16.996321  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:16.997103  1733 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:16.912125  6628 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:16.937846  6651 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.014386  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:16.887050  6307 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:16.998142  6486 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:03:16.998615  6485 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:17.014714  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
W20260709 13:03:17.019874  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 247114us
I20260709 13:03:17.020458  1740 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:17.023288  1738 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:17.023900  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2723) took 1804 ms. Trace:
I20260709 13:03:17.024187  1738 rpcz_store.cc:276] 0709 13:03:15.219591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.022875 (+1803284us) service_pool.cc:224] Handling call
0709 13:03:17.023145 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.023697 (+   552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.023853 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.025478  1738 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 13:03:17.026486  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2726) took 1806 ms. Trace:
I20260709 13:03:17.027848  1738 rpcz_store.cc:276] 0709 13:03:15.220334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.024905 (+1804571us) service_pool.cc:224] Handling call
0709 13:03:17.025280 (+   375us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:17.026153 (+   873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.026418 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.031601  1738 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 12. (index mismatch)
I20260709 13:03:17.034049  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 790) took 1704 ms. Trace:
I20260709 13:03:17.035813  1738 rpcz_store.cc:276] 0709 13:03:15.330163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.030687 (+1700524us) service_pool.cc:224] Handling call
0709 13:03:17.031119 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.033676 (+  2557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.033967 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.021034  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:17.016534  6623 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.501s	user 0.006s	sys 0.004s
I20260709 13:03:17.050550  1740 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:17.051326  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2725) took 1831 ms. Trace:
I20260709 13:03:17.051694  1740 rpcz_store.cc:276] 0709 13:03:15.220100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.021237 (+1801137us) service_pool.cc:224] Handling call
0709 13:03:17.050308 (+ 29071us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.051089 (+   781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.051270 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.052529  1733 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:16.880085  6712 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.053222  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2724) took 1833 ms. Trace:
I20260709 13:03:17.049260  6485 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:17.058399  1737 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:17.058740  6485 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:17.013815  6032 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 13:03:17.070058  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2709) took 4421 ms. Trace:
I20260709 13:03:17.070559  1735 rpcz_store.cc:276] 0709 13:03:12.648764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:12.663876 (+ 15112us) service_pool.cc:224] Handling call
0709 13:03:16.937540 (+4273664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.966028 (+ 28488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.069788 (+103760us) spinlock_profiling.cc:243] Waited 4.27 s on lock 0x7b5400107cd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.069965 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9828567424}
I20260709 13:03:17.051450  1738 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:17.074370  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 791) took 1603 ms. Trace:
I20260709 13:03:17.074795  1738 rpcz_store.cc:276] 0709 13:03:15.471408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.038156 (+1566748us) service_pool.cc:224] Handling call
0709 13:03:17.051271 (+ 13115us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.074020 (+ 22749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.074307 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.076668  1738 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:17.053198  5898 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.056471  1733 rpcz_store.cc:276] 0709 13:03:15.219847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.019920 (+1800073us) service_pool.cc:224] Handling call
0709 13:03:17.021331 (+  1411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.052959 (+ 31628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.053166 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.089389  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:17.054157  6716 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:17.097051  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.098629  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:17.092245  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:17.093950  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.093858  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:17.094012  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 590084us
I20260709 13:03:17.123246  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:17.137336  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:17.140010  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:17.052886  6623 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.153438  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:17.158370  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69199.1us
I20260709 13:03:17.159437  6506 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:17.161407  6649 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:17.163457  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69733.9us
I20260709 13:03:17.166137  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.167227  1661 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:17.168385  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.164264  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.105244  6712 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:17.092175  6600 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.154049  6580 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:17.090869  6609 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.166850  6506 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:17.121935  6716 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:17.094326  6583 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:17.179960  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:17.176153  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2658) took 1020 ms. Trace:
I20260709 13:03:17.206979  1668 rpcz_store.cc:276] 0709 13:03:16.155596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.590635 (+435039us) service_pool.cc:224] Handling call
0709 13:03:17.175937 (+585302us) spinlock_profiling.cc:243] Waited 23.9 ms on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.176077 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54958464}
I20260709 13:03:17.156069  6422 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:17.105146  5898 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.209147  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:17.170389  6326 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:17.178231  6612 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:17.162514  6572 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:17.214383  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.161940  6649 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.196059  6307 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:17.220182  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.220654  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.223697  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.215868  6646 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:17.211380  6541 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:17.209720  6716 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:17.212744  6422 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:17.239074  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2805) took 1459 ms. Trace:
I20260709 13:03:17.237535  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.272493  1585 rpcz_store.cc:276] 0709 13:03:15.780984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.810688 (+ 29704us) service_pool.cc:224] Handling call
0709 13:03:15.810984 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.127437 (+316453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.238783 (+1111346us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54002184d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:17.239004 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715396352}
I20260709 13:03:17.242885  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:17.245626  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 282475us
I20260709 13:03:17.258613  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:17.262743  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2793) took 1577 ms. Trace:
I20260709 13:03:17.274871  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:17.202374  6305 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:03:17.197474  6600 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:17.284708  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:17.231806  6612 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:17.274017  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.214193  6583 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.224472  6334 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:17.265564  6638 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:17.275511  1593 rpcz_store.cc:276] 0709 13:03:15.685766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.685899 (+   133us) service_pool.cc:224] Handling call
0709 13:03:15.686231 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:15.734347 (+ 48116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:15.734399 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:15.735073 (+   674us) log.cc:844] Serialized 26 byte log entry
0709 13:03:15.828981 (+ 93908us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:15.829030 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.239024 (+1409994us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b5400036bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:17.239109 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.239166 (+    57us) raft_consensus.cc:1685] finished
0709 13:03:17.239181 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.262649 (+ 23468us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16672,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":114,"spinlock_wait_cycles":109535104,"thread_start_us":93432,"threads_started":1,"wal-append.queue_time_us":167117,"wal-append.run_cpu_time_us":1864,"wal-append.run_wall_time_us":1046906}
I20260709 13:03:17.282294  6714 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:17.224892  6720 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:17.235793  6572 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:17.252794  6307 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:17.208956  6609 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:17.261883  6611 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.240990  6481 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 13:03:17.279850  6628 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:17.322710  1742 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: []
W20260709 13:03:17.319100  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 789) took 2034 ms (client timeout 2025 ms). Trace:
W20260709 13:03:17.336170  1741 rpcz_store.cc:269] 0709 13:03:15.284471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.813485 (+529014us) service_pool.cc:224] Handling call
0709 13:03:17.318755 (+1505270us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.318997 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2569066368}
I20260709 13:03:17.280908  6506 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:17.327168  1661 raft_consensus.cc:2410] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 12 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 13:03:17.303115  6516 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:17.254077  6541 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:17.278158  6422 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:17.266348  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57288.7us
W20260709 13:03:17.309029  6721 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 13:03:17.311484  6719 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 13:03:17.327595  6638 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:17.342196  6572 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.342695  6600 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:17.337994  6714 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:17.344619  6456 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:17.337956  6611 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.318266  6651 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.349001  5898 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:17.283946  6326 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:03:17.350075  6152 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:17.308323  6612 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:17.306430  6543 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:03:17.354251  6583 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.355441  6623 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.359747  6649 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.283648  6724 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:17.312161  6711 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260709 13:03:17.310839  6647 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:17.367066  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:17.240849  6486 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 13:03:17.308298  6334 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:17.373723  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:17.379194  1586 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:17.379796  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2055) took 1111 ms. Trace:
I20260709 13:03:17.483368  1735 rpcz_store.cc:276] 0709 13:03:16.268905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.071598 (+802693us) service_pool.cc:224] Handling call
0709 13:03:17.071994 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.072226 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:17.120541 (+ 48315us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.379401 (+258860us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.379463 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:17.379518 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.379600 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.379731 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36822,"prepare.queue_time_us":93152,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":3568,"raft.queue_time_us":31206,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":157,"thread_start_us":258484,"threads_started":1}
I20260709 13:03:17.392544  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:17.338104  6665 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:17.485743  1733 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:17.338478  6506 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:17.486570  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:17.487389  1735 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:17.486550  6506 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:17.402266  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2809) took 1564 ms (client timeout 1782 ms). Trace:
W20260709 13:03:17.489689  1588 rpcz_store.cc:269] 0709 13:03:15.814294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.823347 (+  9053us) service_pool.cc:224] Handling call
0709 13:03:17.378662 (+1555315us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b540025ce54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.378857 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340357888}
I20260709 13:03:17.491017  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.491642  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.492147  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:17.361688  6720 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:17.493180  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.493979  6720 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.407074  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2016) took 4607 ms. Trace:
I20260709 13:03:17.494016  6714 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.495128  1736 rpcz_store.cc:276] 0709 13:03:12.799979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:14.069318 (+1269339us) service_pool.cc:224] Handling call
0709 13:03:14.083521 (+ 14203us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:14.312967 (+229446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:14.313014 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:14.313919 (+   905us) log.cc:844] Serialized 78 byte log entry
0709 13:03:14.686256 (+372337us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:15.008138 (+321882us) log.cc:844] Serialized 14 byte log entry
0709 13:03:15.417792 (+409654us) log.cc:844] Serialized 14 byte log entry
0709 13:03:16.335278 (+917486us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b18003567a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:03:16.954587 (+619309us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.386816 (+432229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.406455 (+ 19639us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.406576 (+   121us) raft_consensus.cc:1685] finished
0709 13:03:17.406618 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.406997 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":164964,"prepare.queue_time_us":528055,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":11278,"raft.queue_time_us":227472,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":41674,"spinlock_wait_cycles":4090624,"thread_start_us":1916615,"threads_started":6,"wal-append.queue_time_us":1588387,"wal-append.run_cpu_time_us":2730,"wal-append.run_wall_time_us":1007552}
I20260709 13:03:17.362126  6516 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:17.497721  6516 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:17.499663  6506 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.416947  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.501714  6536 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:17.502345  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.503013  1585 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:17.503796  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.504335  1585 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:17.505138  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.418059  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.505822  1589 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:17.507210  6506 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:17.420724  1591 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:17.423995  1584 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:17.426486  6634 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:17.429657  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:17.510344  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80884.8us
I20260709 13:03:17.512710  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:17.513702  6628 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:17.378341  6638 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:17.514276  6307 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.459678  1734 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:17.518955  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 785) took 3081 ms. Trace:
I20260709 13:03:17.493187  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:17.524374  1736 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:17.519549  1739 rpcz_store.cc:276] 0709 13:03:14.437142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.554011 (+2116869us) service_pool.cc:224] Handling call
0709 13:03:16.554891 (+   880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:16.555456 (+   565us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.555492 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:16.591716 (+ 36224us) log.cc:844] Serialized 26 byte log entry
0709 13:03:16.951637 (+359921us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:16.951755 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:16.951913 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.518653 (+566740us) raft_consensus.cc:1685] finished
0709 13:03:17.518686 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.518874 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":3968,"thread_start_us":310696,"threads_started":1,"wal-append.queue_time_us":442092}
I20260709 13:03:17.521005  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:17.541647  1588 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 9: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:17.502345  1735 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 13:03:17.537237  1733 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:17.527073  6571 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.018s
W20260709 13:03:17.544951  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.546716  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.547808  6628 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:17.548384  6628 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:17.550129  6746 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:17.549491  6628 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.553934  6571 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.554602  6571 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.556806  6571 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.559005  6611 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.509188  6634 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.483004  6543 consensus_queue.cc:1243] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 13.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:03:17.567519  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 795) took 1540 ms. Trace:
W20260709 13:03:17.574271  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53408.3us
I20260709 13:03:17.575409  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:17.576495  1734 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:17.578204  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 792) took 1597 ms. Trace:
I20260709 13:03:17.579911  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:17.578802  1664 rpcz_store.cc:276] 0709 13:03:15.981189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.031999 (+ 50810us) service_pool.cc:224] Handling call
0709 13:03:16.034318 (+  2319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.034550 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:16.036595 (+  2045us) log.cc:844] Serialized 14 byte log entry
0709 13:03:16.041647 (+  5052us) log.cc:844] Serialized 14 byte log entry
0709 13:03:16.537470 (+495823us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b18001bcee0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdedb93d70 00007fcdedb87293 00007fcdedb7ccea 00007fcdedb7d23a 00007fcdedfe7a2a 00007fcdedff70bf 00007fcdedff7038
0709 13:03:17.103939 (+566469us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.564196 (+460257us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.564259 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:17.564311 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.564540 (+   229us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.578099 (+ 13559us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":567299,"prepare.queue_time_us":332910,"prepare.run_cpu_time_us":163,"prepare.run_wall_time_us":3914,"raft.queue_time_us":669987,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":11472,"spinlock_wait_cycles":37257344,"thread_start_us":459914,"threads_started":1}
I20260709 13:03:17.581593  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:17.592185  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:17.602067  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2668) took 1435 ms. Trace:
I20260709 13:03:17.568137  1663 rpcz_store.cc:276] 0709 13:03:16.026862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.870660 (+843798us) service_pool.cc:224] Handling call
0709 13:03:16.870949 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.871157 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:03:17.234612 (+363455us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.567059 (+332447us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.567141 (+    82us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:17.567209 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.567299 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.567449 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":58672,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"thread_start_us":694366,"threads_started":2}
I20260709 13:03:17.619473  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2670) took 1451 ms. Trace:
I20260709 13:03:17.619714  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2654) took 1465 ms. Trace:
I20260709 13:03:17.619905  1668 rpcz_store.cc:276] 0709 13:03:16.168366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.215212 (+1046846us) service_pool.cc:224] Handling call
0709 13:03:17.215576 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.215807 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:17.216052 (+   245us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.619044 (+402992us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.619106 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:17.619174 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.619253 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.619411 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38678,"prepare.queue_time_us":134058,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":363539,"threads_started":1}
I20260709 13:03:17.621660  1665 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:17.620373  1660 rpcz_store.cc:276] 0709 13:03:16.154564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.888760 (+734196us) service_pool.cc:224] Handling call
0709 13:03:16.889118 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.889348 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:17.205100 (+315752us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.619250 (+414150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.619327 (+    77us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:17.619392 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.619488 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.619638 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26665,"prepare.queue_time_us":40709,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":28,"thread_start_us":674378,"threads_started":2}
I20260709 13:03:17.622862  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.622620  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.625092  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:17.626312  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 448806us
I20260709 13:03:17.444487  6541 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.628239  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.639241  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.640512  1733 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:17.644836  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.608770  1666 rpcz_store.cc:276] 0709 13:03:16.166260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.947813 (+781553us) service_pool.cc:224] Handling call
0709 13:03:16.948156 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:16.948387 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:16.963089 (+ 14702us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.601535 (+638446us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.601621 (+    86us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:17.601692 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.601799 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.601987 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26556,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":327949,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"thread_start_us":637804,"threads_started":2,"wal-append.queue_time_us":518520}
I20260709 13:03:17.647243  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.600168  6623 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:17.564062  6326 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260709 13:03:17.654196  1660 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:17.648185  1666 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:17.617738  6586 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:17.600695  6712 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:17.655148  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.591161  6611 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:17.619223  6651 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:17.665457  6033 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 13:03:17.668359  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.669879  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.674484  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.628356  6583 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.675361  6583 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.670120  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.678568  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "249474541a174b90a099d77784ed4801"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:03:17.679283  1664 raft_consensus.cc:606] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:17.658671  2281 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:17.682731  6583 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.665319  6566 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:03:17.687896  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 797) took 1488 ms. Trace:
I20260709 13:03:17.688390  6422 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.688503  1668 rpcz_store.cc:276] 0709 13:03:16.199160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.686791 (+1487631us) service_pool.cc:224] Handling call
0709 13:03:17.687182 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.687446 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.687486 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.687552 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:17.687602 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.687679 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.687801 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.691035  6326 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:17.691891  6586 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.700630  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:17.706290  1663 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:17.701912  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.644466  6736 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:17.708158  6736 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:17.697878  6127 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:17.708892  6127 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:17.709306  6127 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:17.709956  6573 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:17.709429  6736 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.711351  6736 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:17.711822  6573 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:17.713063  6680 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:17.713455  6680 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:17.714587  1738 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:17.715377  6741 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.155s	user 0.000s	sys 0.000s
I20260709 13:03:17.715039  6680 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:17.717005  6727 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 13:03:17.723500  6481 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:17.724129  6481 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:17.727277  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 800) took 1400 ms. Trace:
I20260709 13:03:17.727725  1666 rpcz_store.cc:276] 0709 13:03:16.326874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.726289 (+1399415us) service_pool.cc:224] Handling call
0709 13:03:17.726670 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.726878 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.726907 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.726956 (+    49us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:17.726993 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.727057 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.727195 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.729913  6609 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.730013  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 798) took 1509 ms. Trace:
I20260709 13:03:17.730563  1660 rpcz_store.cc:276] 0709 13:03:16.220481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.699005 (+1478524us) service_pool.cc:224] Handling call
0709 13:03:17.699518 (+   513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.700113 (+   595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.700142 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.729701 (+ 29559us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:17.729755 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.729823 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.729948 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.731742  6634 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:17.704201  6334 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:17.725404  6481 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:17.743945  6481 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:17.737464  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 799) took 1416 ms. Trace:
I20260709 13:03:17.750561  1668 rpcz_store.cc:276] 0709 13:03:16.320639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.736381 (+1415742us) service_pool.cc:224] Handling call
0709 13:03:17.736826 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.737054 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.737089 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.737146 (+    57us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:17.737187 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.737254 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.737389 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.752772  1733 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:03:17.732928  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2675) took 1054 ms. Trace:
I20260709 13:03:17.753952  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.754738  1735 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:17.754004  1660 rpcz_store.cc:276] 0709 13:03:16.678972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.732086 (+1053114us) service_pool.cc:224] Handling call
0709 13:03:17.732411 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.732610 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.732635 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.732675 (+    40us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:17.732707 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.732765 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.732866 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:17.759768  1660 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 13:03:17.765028  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.765489  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2674) took 1304 ms. Trace:
I20260709 13:03:17.766049  1666 rpcz_store.cc:276] 0709 13:03:16.461112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.728850 (+1267738us) service_pool.cc:224] Handling call
0709 13:03:17.729164 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.729382 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.729412 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:17.729459 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:17.729494 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.765278 (+ 35784us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:17.765427 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:17.770383  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.005s	sys 0.039s
W20260709 13:03:17.771236  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.006s	sys 0.040s
W20260709 13:03:17.771716  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 600445us
I20260709 13:03:17.749119  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 710) took 1533 ms. Trace:
I20260709 13:03:17.773025  1586 rpcz_store.cc:276] 0709 13:03:16.215737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.511739 (+296002us) service_pool.cc:224] Handling call
0709 13:03:17.378989 (+867250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.518381 (+139392us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.748749 (+230368us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b540025ce50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.749003 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1993933568}
I20260709 13:03:17.771883  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.794741  1660 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:17.797531  1666 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:17.806588  6506 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:17.802892  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:17.801865  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:17.816931  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.022s	sys 0.044s
W20260709 13:03:17.820501  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.022s	sys 0.044s
I20260709 13:03:17.697602  6651 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:17.822873  1632 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.021s	sys 0.034s
I20260709 13:03:17.772642  6326 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:17.842834  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.007s	sys 0.050s
W20260709 13:03:17.867396  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.007s	sys 0.050s
W20260709 13:03:17.867925  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 570550us
I20260709 13:03:17.846994  6634 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:17.831892  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.002s	sys 0.053s
W20260709 13:03:17.872324  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.003s	sys 0.053s
W20260709 13:03:17.872972  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 614489us
W20260709 13:03:17.825047  6580 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.619s	user 0.013s	sys 0.000s
I20260709 13:03:17.813433  6611 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:17.700979  6733 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.177s
W20260709 13:03:17.857475  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 570114us
W20260709 13:03:17.858213  1632 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.021s	sys 0.034s
W20260709 13:03:17.831274  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:17.949350  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 807760us
W20260709 13:03:17.864494  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2656) took 1709 ms (client timeout 1894 ms). Trace:
W20260709 13:03:17.950533  1665 rpcz_store.cc:269] 0709 13:03:16.155081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.503074 (+347993us) service_pool.cc:224] Handling call
0709 13:03:17.864220 (+1361146us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540030af54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:17.864411 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2570426112}
I20260709 13:03:17.953073  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:17.833455  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:17.955272  6623 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:17.955900  6623 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:17.956429  6623 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:17.955004  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 121782us
W20260709 13:03:17.958835  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.960153  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.960626  1735 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:17.870543  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:17.819109  1666 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:17.850030  6376 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 13:03:17.885596  2343 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:39071 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:39071 timed out after 15.000s (SENT)
W20260709 13:03:17.898694  1659 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.005s	sys 0.039s
I20260709 13:03:17.811949  6738 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 13:03:17.962973  1659 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.005s	sys 0.039s
I20260709 13:03:17.953856  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:17.968189  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:18.027173  1586 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 13:03:18.029711  1591 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:17.961861  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.869876  6651 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:18.062999  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:18.064191  1739 raft_consensus.cc:2376] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10. Re-sending same reply.
I20260709 13:03:17.887784  6634 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:18.065306  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46722.8us
W20260709 13:03:18.066077  1593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.012s	sys 0.034s
I20260709 13:03:18.066255  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.069329  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:18.077087  1593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.012s	sys 0.034s
I20260709 13:03:17.979797  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.072880  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:18.021667  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67939.3us
I20260709 13:03:18.072880  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.076858  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:18.073393  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 718) took 1526 ms. Trace:
I20260709 13:03:18.079092  1587 rpcz_store.cc:276] 0709 13:03:16.546755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.824432 (+277677us) service_pool.cc:224] Handling call
0709 13:03:16.824916 (+   484us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:16.831987 (+  7071us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.832035 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:16.856494 (+ 24459us) log.cc:844] Serialized 52 byte log entry
0709 13:03:17.215711 (+359217us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:17.268929 (+ 53218us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.271484 (+  2555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.362125 (+ 90641us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b54001c4154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:17.362204 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.025403 (+663199us) raft_consensus.cc:1685] finished
0709 13:03:18.073089 (+ 47686us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.073315 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7007,"prepare.queue_time_us":64675,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":6269,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":7818,"spinlock_wait_cycles":30581376,"thread_start_us":350952,"threads_started":1,"wal-append.queue_time_us":564156}
I20260709 13:03:18.090252  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:17.952131  6551 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:18.105821  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:17.968081  6376 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:18.116241  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:18.123193  1737 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.032s	sys 0.036s
W20260709 13:03:18.154031  1737 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.032s	sys 0.038s
I20260709 13:03:18.133881  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:18.109897  6516 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.612s	user 0.005s	sys 0.006s
I20260709 13:03:18.093161  6634 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:18.074173  6723 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.740s	user 0.005s	sys 0.000s
W20260709 13:03:18.152652  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75916.2us
I20260709 13:03:18.155287  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.162549  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2044) took 2447 ms. Trace:
I20260709 13:03:18.113204  6570 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:18.091995  6646 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:18.156977  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:18.181097  1737 rpcz_store.cc:276] 0709 13:03:15.714777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.059171 (+1344394us) service_pool.cc:224] Handling call
0709 13:03:17.059501 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.059698 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:17.457970 (+398272us) log.cc:844] Serialized 14 byte log entry
0709 13:03:18.161900 (+703930us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:18.161981 (+    81us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:18.162047 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.162145 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.162457 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2811,"prepare.queue_time_us":35465,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"thread_start_us":1098510,"threads_started":2}
I20260709 13:03:18.156219  6575 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:18.179184  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:18.185410  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.238179  1739 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:18.198562  6744 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.230s	user 0.000s	sys 0.000s
I20260709 13:03:18.251302  1739 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:18.202021  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 716) took 1728 ms. Trace:
I20260709 13:03:18.251307  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.252282  1590 rpcz_store.cc:276] 0709 13:03:16.473280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.644740 (+171460us) service_pool.cc:224] Handling call
0709 13:03:16.645317 (+   577us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:16.645877 (+   560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:16.645925 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:16.714871 (+ 68946us) log.cc:844] Serialized 26 byte log entry
0709 13:03:17.107612 (+392741us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:17.155511 (+ 47899us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.163435 (+  7924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:17.163519 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:17.790030 (+626511us) spinlock_profiling.cc:243] Waited 4.22 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:18.201168 (+411138us) raft_consensus.cc:1685] finished
0709 13:03:18.201425 (+   257us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.201908 (+   483us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10895,"prepare.queue_time_us":48346,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":4204,"raft.queue_time_us":22514,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":3662,"spinlock_wait_cycles":9717120,"thread_start_us":315741,"threads_started":1,"wal-append.queue_time_us":508485}
W20260709 13:03:18.206840  6740 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.319s	user 0.000s	sys 0.000s
I20260709 13:03:18.157842  6551 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:18.215641  6638 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.701s	user 0.011s	sys 0.001s
I20260709 13:03:18.221755  6422 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:17.967976  6580 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:18.227815  1667 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.046s	sys 0.072s
W20260709 13:03:18.284025  1667 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.799s	user 0.046s	sys 0.073s
W20260709 13:03:18.228163  6737 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 13:03:17.949319  6665 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:18.293606  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.003s	sys 0.037s
W20260709 13:03:18.279994  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 101171us
I20260709 13:03:18.226063  6570 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:18.267066  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.015s	sys 0.032s
I20260709 13:03:18.181993  6516 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.293641  1734 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:03:18.253334  6571 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.005s	sys 0.013s
W20260709 13:03:18.298136  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.003s	sys 0.037s
I20260709 13:03:18.229890  6712 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.059s
W20260709 13:03:18.292807  6566 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.020s	sys 0.020s
I20260709 13:03:18.290457  1740 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:18.288662  6628 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.026s	sys 0.043s
W20260709 13:03:18.322435  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.000s	sys 0.024s
I20260709 13:03:18.299216  6580 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.281009  6753 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:18.300946  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.015s	sys 0.033s
W20260709 13:03:18.347362  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 755409us
W20260709 13:03:18.322175  6628 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.026s	sys 0.044s
W20260709 13:03:18.318509  1558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.012s	sys 0.011s
W20260709 13:03:18.355005  1558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.012s	sys 0.011s
W20260709 13:03:18.355710  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.287725  6743 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:18.287824  6138 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:18.322854  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.000s	sys 0.024s
W20260709 13:03:18.309608  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 625921us
I20260709 13:03:18.338064  1741 raft_consensus.cc:2368] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14.
I20260709 13:03:18.370292  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.372924  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:18.309113  6571 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.006s	sys 0.014s
W20260709 13:03:18.315135  6647 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.004s	sys 0.018s
W20260709 13:03:18.376257  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 469834us
W20260709 13:03:18.368448  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 564934us
W20260709 13:03:18.378364  1663 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.671s	user 0.016s	sys 0.000s
W20260709 13:03:18.378870  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.379263  1663 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:18.302956  6665 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:18.318066  6566 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.020s	sys 0.020s
I20260709 13:03:18.349829  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:18.294276  6544 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.012s	sys 0.028s
I20260709 13:03:18.287009  6638 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.346725  6753 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:18.286711  6595 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:18.347362  6498 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:18.365285  1664 pending_rounds.cc:85] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:18.403909  1664 pending_rounds.cc:107] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 13:03:18.404244  1664 raft_consensus.cc:2889] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:18.404259  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.374125  1585 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:18.376943  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 802) took 1221 ms. Trace:
I20260709 13:03:18.407536  1661 rpcz_store.cc:276] 0709 13:03:17.155275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.168160 (+ 12885us) service_pool.cc:224] Handling call
0709 13:03:18.364246 (+1196086us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:18.376689 (+ 12443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362824064}
I20260709 13:03:18.380113  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2021) took 1077 ms. Trace:
I20260709 13:03:18.383739  1668 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 8. Current term is 9. Ops: []
I20260709 13:03:18.389051  1666 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:18.411504  1666 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:18.342128  6649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.015s	sys 0.008s
W20260709 13:03:18.413966  6649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.849s	user 0.016s	sys 0.009s
I20260709 13:03:18.390897  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:18.365574  6612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.000s	sys 0.032s
I20260709 13:03:18.361421  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:18.416548  6612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.869s	user 0.000s	sys 0.032s
I20260709 13:03:18.402593  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.410574  1663 rpcz_store.cc:276] 0709 13:03:17.302327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.652187 (+349860us) service_pool.cc:224] Handling call
0709 13:03:18.380040 (+727853us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:18.365425  6583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.005s	sys 0.011s
I20260709 13:03:18.421871  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.412745  1666 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.423410  1666 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:18.424237  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:18.382836  6647 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.005s	sys 0.018s
I20260709 13:03:18.384207  6326 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:18.400488  6544 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.013s	sys 0.028s
I20260709 13:03:18.359427  6743 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:18.350440  6756 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 13:03:18.350023  6757 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.097s
I20260709 13:03:18.439281  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:18.442608  1590 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:18.354252  6750 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 13:03:18.430991  1735 raft_consensus.cc:2376] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11. Re-sending same reply.
I20260709 13:03:18.446175  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.469126  1586 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:18.435060  6738 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:18.350775  6580 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.447365  2315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:18.471385  2315 raft_consensus.cc:606] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:18.427484  6583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.005s	sys 0.012s
I20260709 13:03:18.427500  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.351898  6745 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 13:03:18.458144  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.459203  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2728) took 2810 ms. Trace:
I20260709 13:03:18.475157  1742 rpcz_store.cc:276] 0709 13:03:15.648909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.042997 (+1394088us) service_pool.cc:224] Handling call
0709 13:03:17.322172 (+279175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:17.339649 (+ 17477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.450056 (+1110407us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54000df2d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:18.459093 (+  9037us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":641010048}
I20260709 13:03:18.353391  6755 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 3, Time since last communication: 0.074s
I20260709 13:03:18.459370  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:18.428814  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79076.6us
I20260709 13:03:18.462008  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:18.421356  6753 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:18.503731  1584 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:18.456502  6743 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:18.470129  2318 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:18.385919  6138 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.492933  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.487656  6738 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.460208  6307 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:18.501329  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.506177  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.429840  6611 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:18.424127  6498 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.430310  6595 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.398468  6570 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.438622  6752 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 13:03:18.510964  6516 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:18.514537  6716 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.230s	user 0.007s	sys 0.007s
I20260709 13:03:18.515146  6580 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:18.515975  6761 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
W20260709 13:03:18.508802  6751 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.243s	user 0.001s	sys 0.001s
I20260709 13:03:18.499500  6745 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:18.516695  6772 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:18.517594  6665 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:18.517266  6486 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:18.518442  6551 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:18.491850  6600 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:18.520304  6506 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.504771  6680 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:18.527961  6127 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.818s	user 0.011s	sys 0.000s
I20260709 13:03:18.535667  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.530874  1586 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:18.514957  6638 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:18.542654  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2070) took 1024 ms. Trace:
W20260709 13:03:18.661391  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 235818us
I20260709 13:03:18.549755  6566 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
W20260709 13:03:18.556207  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.076s	sys 0.146s
I20260709 13:03:18.567806  6754 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:18.601411  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:18.617435  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.619304  1584 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:18.543079  6765 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:18.651244  1666 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:03:18.661674  1738 rpcz_store.cc:276] 0709 13:03:17.518131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.518719 (+   588us) service_pool.cc:224] Handling call
0709 13:03:17.541259 (+ 22540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:17.715094 (+173835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.542345 (+827251us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:18.542586 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397925120}
I20260709 13:03:18.598562  6580 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:03:18.547343  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:18.667273  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.076s	sys 0.146s
I20260709 13:03:18.670432  1586 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
W20260709 13:03:18.670500  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 783178us
I20260709 13:03:18.667676  1665 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:18.669516  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:18.672127  1738 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:18.672760  6763 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
I20260709 13:03:18.676549  6754 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:18.677129  6754 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.677779  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2649) took 4157 ms. Trace:
I20260709 13:03:18.678175  1662 rpcz_store.cc:276] 0709 13:03:14.520410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:15.148319 (+627909us) service_pool.cc:224] Handling call
0709 13:03:15.148668 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:17.395631 (+2246963us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:17.395687 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:17.406162 (+ 10475us) log.cc:844] Serialized 26 byte log entry
0709 13:03:17.870973 (+464811us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:17.871015 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.677361 (+806346us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:18.677412 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.677458 (+    46us) raft_consensus.cc:1685] finished
0709 13:03:18.677472 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.677717 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":570615,"raft.queue_time_us":560382,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":18685,"spinlock_wait_cycles":3190353280,"thread_start_us":693503,"threads_started":2,"wal-append.queue_time_us":775492}
I20260709 13:03:18.679741  6754 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:18.680321  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:18.683586  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59393.5us
I20260709 13:03:18.684460  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:18.684719  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 149202us
I20260709 13:03:18.665764  6127 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.685493  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.615334  6600 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:18.684180  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:18.686909  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 262777us
I20260709 13:03:18.687399  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:18.690168  1663 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:18.690475  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:18.691031  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 369165us
I20260709 13:03:18.692619  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:18.654313  6516 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:18.697422  1591 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:18.698218  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:18.668743  1666 pending_rounds.cc:85] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 11
I20260709 13:03:18.699627  1666 pending_rounds.cc:107] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:03:18.699874  1666 raft_consensus.cc:2889] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:18.700068  1587 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:18.700762  1738 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:18.693398  2315 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:18.695525  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.702353  1738 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:18.703073  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.583566  6716 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.663688  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.658243  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:18.709946  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 283808us
I20260709 13:03:18.726866  1587 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:03:18.727272  1587 pending_rounds.cc:85] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:03:18.727495  1587 pending_rounds.cc:107] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:03:18.727775  1587 raft_consensus.cc:2889] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:18.712890  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:18.652529  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.004s	sys 0.006s
I20260709 13:03:18.711614  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.714577  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:18.688411  6638 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 13:03:18.693591  6760 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 13:03:18.743695  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.728849  1665 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:03:18.730648  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67030.4us
I20260709 13:03:18.721980  6307 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:18.750926  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:18.732043  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.004s	sys 0.006s
W20260709 13:03:18.751964  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:18.752523  6481 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.008s	user 0.016s	sys 0.000s
I20260709 13:03:18.754024  6481 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:03:18.696779  6773 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 13:03:18.730397  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:18.755960  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2073) took 1003 ms. Trace:
W20260709 13:03:18.756548  6774 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:03:18.756395  1736 rpcz_store.cc:276] 0709 13:03:17.752171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.757759 (+  5588us) service_pool.cc:224] Handling call
0709 13:03:17.758212 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:17.789087 (+ 30875us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:17.789136 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:17.810339 (+ 21203us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.343183 (+532844us) raft_consensus.cc:1645] Marking committed up to 3
0709 13:03:18.343246 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.343357 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.755750 (+412393us) raft_consensus.cc:1685] finished
0709 13:03:18.755771 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.755904 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30312,"raft.queue_time_us":39830,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":133,"thread_start_us":484528,"threads_started":1,"wal-append.queue_time_us":629723}
I20260709 13:03:18.758535  6571 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:18.743462  6714 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:18.735643  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2835) took 1320 ms. Trace:
I20260709 13:03:18.655371  6506 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:18.751421  6623 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.794s	user 0.012s	sys 0.004s
I20260709 13:03:18.669346  6580 consensus_queue.cc:1243] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:18.737931  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:18.738823  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.739240  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:18.764830  2318 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:18.693253  6780 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.711321  6779 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:18.761158  1593 rpcz_store.cc:276] 0709 13:03:17.415239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.415406 (+   167us) service_pool.cc:224] Handling call
0709 13:03:17.415885 (+   479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:17.444478 (+ 28593us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:17.444526 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:17.459423 (+ 14897us) log.cc:844] Serialized 52 byte log entry
0709 13:03:18.077895 (+618472us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:18.077959 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.078066 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.735397 (+657331us) raft_consensus.cc:1685] finished
0709 13:03:18.735424 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.735581 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28028,"raft.queue_time_us":28167,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":8429,"thread_start_us":563831,"threads_started":1,"wal-append.queue_time_us":826472}
I20260709 13:03:18.728905  6628 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:18.812626  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 73933.7us
W20260709 13:03:18.815146  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64316.8us
I20260709 13:03:18.815505  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:18.815963  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.710904  6750 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 13:03:18.817423  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:18.818372  1593 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:03:18.772871  6634 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.001s
I20260709 13:03:18.723132  6516 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.742533  6326 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:18.737516  6791 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:18.762817  6712 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 3, Time since last communication: 0.092s
I20260709 13:03:18.819727  6638 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:18.815344  6736 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.103s	user 0.009s	sys 0.007s
I20260709 13:03:18.694070  6611 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.779276  6779 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:18.858335  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.881682  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.861332  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.882254  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 796) took 2722 ms. Trace:
I20260709 13:03:18.882951  1738 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:18.769124  6623 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:18.866963  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 168840us
I20260709 13:03:18.840862  6781 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:18.741422  6756 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20260709 13:03:18.777181  6775 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:18.876153  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:18.868851  6628 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:18.728257  6766 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.135s
I20260709 13:03:18.862373  6572 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:18.869433  6580 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:18.761569  6745 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 13:03:18.890777  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:18.882707  1667 rpcz_store.cc:276] 0709 13:03:16.159457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.932098 (+772641us) service_pool.cc:224] Handling call
0709 13:03:16.932445 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:17.442140 (+509695us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:17.442189 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:17.482038 (+ 39849us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.284632 (+802594us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:18.284704 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.284823 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.882000 (+597177us) raft_consensus.cc:1685] finished
0709 13:03:18.882025 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:18.882186 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":174653,"raft.queue_time_us":178080,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":30079,"thread_start_us":1138726,"threads_started":2,"wal-append.queue_time_us":951169}
I20260709 13:03:18.844122  6634 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:18.892248  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.843397  6782 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 13:03:18.887264  6779 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:18.920313  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.897713  6791 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:18.923168  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:18.898995  6775 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:18.956125  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 224866us
I20260709 13:03:18.948196  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:18.951035  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:18.886811  6638 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.901993  6628 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:18.903357  6138 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:18.955313  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.883900  6326 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 13:03:18.915584  6583 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:18.904217  6738 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:18.871980  6786 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.096s
I20260709 13:03:18.951737  6781 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:18.949831  6791 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:18.972602  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:18.915050  6750 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:03:18.956406  6714 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:19.064013  1736 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:18.957167  6334 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:18.947487  6736 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:03:18.960318  6782 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:18.958216  6775 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:19.069624  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 177441us
I20260709 13:03:18.908828  6572 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:18.964028  6571 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:18.965854  6628 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:18.979581  6138 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:18.974514  6583 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:19.068257  6773 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 13:03:19.067500  6791 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:19.117277  6716 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.170382  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 726) took 1147 ms. Trace:
I20260709 13:03:19.171257  6572 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.120587  6782 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:19.173810  6782 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:19.175211  6782 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:19.074155  6456 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:19.156095  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:19.142781  6736 consensus_queue.cc:1243] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:19.182106  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:19.170869  1592 rpcz_store.cc:276] 0709 13:03:18.022946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.024267 (+  1321us) service_pool.cc:224] Handling call
0709 13:03:18.024582 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.541193 (+516611us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.541228 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.580841 (+ 39613us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.818379 (+237538us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:18.973759 (+155380us) log.cc:844] Serialized 14 byte log entry
0709 13:03:19.072872 (+ 99113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.073212 (+   340us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.170121 (+ 96909us) raft_consensus.cc:1685] finished
0709 13:03:19.170145 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.170313 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":158916,"prepare.queue_time_us":350988,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":570590,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":13779,"thread_start_us":702976,"threads_started":3,"wal-append.queue_time_us":540440}
I20260709 13:03:19.192246  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:19.185367  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:19.208892  1592 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:19.157801  6788 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:19.172744  6753 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.652s	user 0.006s	sys 0.003s
W20260709 13:03:19.217789  2315 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.516s	user 0.009s	sys 0.005s
I20260709 13:03:19.162844  6583 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:19.160127  6506 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.154102  6138 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:19.271791  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:19.218835  1592 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:19.184798  6801 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:19.278463  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.273249  2315 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:19.121301  6334 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.278980  1593 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
W20260709 13:03:19.707944  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 522673us
I20260709 13:03:19.714057  1667 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 14
I20260709 13:03:19.714468  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 964) took 1022 ms. Trace:
W20260709 13:03:19.601063  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:19.703074  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.883s	user 0.002s	sys 0.005s
W20260709 13:03:19.725253  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.002s	sys 0.005s
W20260709 13:03:19.725646  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 908949us
W20260709 13:03:19.746183  1738 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.002s	sys 0.007s
W20260709 13:03:19.746809  1738 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.002s	sys 0.007s
W20260709 13:03:19.277411  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.757486  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:19.215163  6572 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:19.763624  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:19.721566  2315 rpcz_store.cc:276] 0709 13:03:18.692190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.692330 (+   140us) service_pool.cc:224] Handling call
0709 13:03:19.714369 (+1022039us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:19.768606  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48802.4us
I20260709 13:03:19.231835  6127 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.786077  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:19.272472  6456 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:19.699960  6790 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.623s	user 0.001s	sys 0.000s
W20260709 13:03:19.581756  6758 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 13:03:19.614336  6800 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.450s	user 0.000s	sys 0.001s
W20260709 13:03:19.702545  6794 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.342s	user 0.001s	sys 0.001s
W20260709 13:03:19.470456  6651 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.365s	user 0.012s	sys 0.016s
W20260709 13:03:19.405223  6784 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.516s	user 0.000s	sys 0.002s
I20260709 13:03:19.272161  6516 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:19.577790  6793 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.658s	user 0.001s	sys 0.000s
W20260709 13:03:19.802127  1708 connection.cc:256] RPC call timeout handler was delayed by -1.7972709470000154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
I20260709 13:03:19.788453  6712 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:19.279826  6716 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:19.815140  1667 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:19.401435  6720 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:19.364061  6634 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:19.816229  1585 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.929s	user 0.004s	sys 0.016s
I20260709 13:03:19.406945  6623 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:19.781283  6754 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:19.850999  1585 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.004s	sys 0.018s
W20260709 13:03:19.839378  1586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.008s	sys 0.016s
W20260709 13:03:19.842459  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.886s	user 0.003s	sys 0.035s
I20260709 13:03:19.277376  6753 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:19.852662  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.897s	user 0.003s	sys 0.035s
I20260709 13:03:19.854091  6736 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.854688  6736 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.855427  1739 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:19.844864  6743 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.298s	user 0.013s	sys 0.001s
I20260709 13:03:19.813017  6127 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:19.855019  1586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.008s	sys 0.017s
W20260709 13:03:19.847812  6775 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.000s
W20260709 13:03:19.864230  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:19.783123  6801 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:19.845860  6307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.012s	sys 0.029s
I20260709 13:03:19.864727  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2786) took 1095 ms. Trace:
I20260709 13:03:19.841795  6712 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:19.449749  6714 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:19.850411  6720 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.855094  6753 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:19.812539  6138 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:19.865767  6743 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.863428  6754 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.834803  6651 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:19.879297  1586 rpcz_store.cc:276] 0709 13:03:18.769799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.769993 (+   194us) service_pool.cc:224] Handling call
0709 13:03:18.770424 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.770700 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:19.144646 (+373946us) log.cc:844] Serialized 14 byte log entry
0709 13:03:19.864296 (+719650us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:19.864355 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:19.864407 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.864479 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.864644 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1664,"prepare.queue_time_us":374558,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"thread_start_us":1044254,"threads_started":2}
I20260709 13:03:19.895001  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 732) took 1506 ms. Trace:
I20260709 13:03:19.866111  6506 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:19.923676  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2077) took 1264 ms. Trace:
I20260709 13:03:19.867785  6638 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.877624  6516 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.924779  1737 rpcz_store.cc:276] 0709 13:03:18.658991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.659155 (+   164us) service_pool.cc:224] Handling call
0709 13:03:18.659590 (+   435us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:18.666076 (+  6486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.666127 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:18.666913 (+   786us) log.cc:844] Serialized 52 byte log entry
0709 13:03:18.851168 (+184255us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:18.851290 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.851393 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.923329 (+1071936us) raft_consensus.cc:1685] finished
0709 13:03:19.923361 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.923587 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14768,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":122034,"threads_started":1,"wal-append.queue_time_us":371065}
I20260709 13:03:19.939333  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 805) took 1082 ms. Trace:
I20260709 13:03:19.940063  1737 rpcz_store.cc:276] 0709 13:03:18.856994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.938193 (+1081199us) service_pool.cc:224] Handling call
0709 13:03:19.938628 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:19.938871 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:19.938910 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:19.938974 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:19.939020 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.939102 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.939257 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:19.913199  1588 rpcz_store.cc:276] 0709 13:03:18.388792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.388959 (+   167us) service_pool.cc:224] Handling call
0709 13:03:18.389307 (+   348us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:18.421478 (+ 32171us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.421529 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:18.433133 (+ 11604us) log.cc:844] Serialized 52 byte log entry
0709 13:03:18.703850 (+270717us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:18.857649 (+153799us) log.cc:844] Serialized 14 byte log entry
0709 13:03:18.858173 (+   524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.858254 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.894663 (+1036409us) raft_consensus.cc:1685] finished
0709 13:03:19.894691 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:19.894919 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35247,"prepare.queue_time_us":275425,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":105712,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":48691,"thread_start_us":386603,"threads_started":2,"wal-append.queue_time_us":512377}
I20260709 13:03:19.944103  1737 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:19.945374  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 806) took 1088 ms. Trace:
I20260709 13:03:19.945966  1737 rpcz_store.cc:276] 0709 13:03:18.857347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.943077 (+1085730us) service_pool.cc:224] Handling call
0709 13:03:19.943552 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:19.944942 (+  1390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:19.945271 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:19.883323  6307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.013s	sys 0.029s
I20260709 13:03:19.884780  6810 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:19.954998  6810 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:19.883765  6712 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:19.901808  6818 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.409s	user 0.002s	sys 0.000s
W20260709 13:03:19.934610  6680 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.271s	user 0.018s	sys 0.004s
I20260709 13:03:19.884544  6803 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:19.886255  6816 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.899830  6334 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.885928  6720 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:19.995464  1742 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 12. (term mismatch)
I20260709 13:03:19.894104  6814 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.898677  6743 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:19.962311  6791 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.795s	user 0.011s	sys 0.002s
I20260709 13:03:19.871757  6775 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.027784  1742 pending_rounds.cc:85] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:03:20.039569  1742 pending_rounds.cc:107] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260709 13:03:20.040036  1742 raft_consensus.cc:2889] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:19.936100  6536 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:19.994588  6747 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 13:03:20.035172  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 807) took 1177 ms. Trace:
I20260709 13:03:20.049113  1737 rpcz_store.cc:276] 0709 13:03:18.862211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.991125 (+1128914us) service_pool.cc:224] Handling call
0709 13:03:20.034361 (+ 43236us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:20.034680 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:20.034719 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:20.034782 (+    63us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:20.034830 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.034909 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.035079 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:19.948348  6506 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.044837  6536 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:19.987866  6516 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.029018  6395 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:20.041193  6781 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:19.998390  6828 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.166s
I20260709 13:03:20.087105  1737 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:03:20.026690  6816 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.078487  6395 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:20.027029  6803 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.046523  6791 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.139367  6586 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.043119  6814 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.112782  6736 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.088114  6781 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.207679  6810 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.203575  6651 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.194957  6801 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.277807  6516 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.272418  6775 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:20.346448  6839 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:03:20.345073  6791 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:20.357012  6780 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.541s	user 0.007s	sys 0.007s
W20260709 13:03:20.368721  6779 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.413s	user 0.011s	sys 0.003s
I20260709 13:03:20.369977  6803 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.370795  6816 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:20.383325  6814 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:20.362962  6782 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.187s	user 0.014s	sys 0.004s
I20260709 13:03:20.411876  6779 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.415318  6779 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:20.480154  1559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.002s	sys 0.006s
I20260709 13:03:20.502050  1589 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:20.503118  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.004s	sys 0.010s
W20260709 13:03:20.588137  1708 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.004s	sys 0.008s
W20260709 13:03:20.591354  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:20.596832  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2749) took 1180 ms. Trace:
I20260709 13:03:20.598857  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 800) took 1743 ms. Trace:
W20260709 13:03:20.626411  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:20.628346  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2745) took 2662 ms (client timeout 1889 ms). Trace:
W20260709 13:03:20.639896  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2066) took 3274 ms (client timeout 2413 ms). Trace:
I20260709 13:03:20.641894  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2742) took 2677 ms. Trace:
I20260709 13:03:20.642866  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 798) took 1788 ms. Trace:
W20260709 13:03:20.643823  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2772) took 2240 ms (client timeout 1527 ms). Trace:
I20260709 13:03:20.644284  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2009) took 4671 ms. Trace:
I20260709 13:03:20.645627  6127 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:20.655704  6743 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:20.657437  6334 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:20.706753  6138 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:20.706522  6334 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:20.708606  1559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.003s	sys 0.007s
I20260709 13:03:20.403324  6780 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:20.709653  1708 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.005s	sys 0.008s
W20260709 13:03:20.663632  1585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.007s	sys 0.029s
W20260709 13:03:20.710481  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 871443us
W20260709 13:03:20.702800  1734 rpcz_store.cc:269] 0709 13:03:17.965649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.979482 (+ 13833us) service_pool.cc:224] Handling call
0709 13:03:20.627615 (+2648133us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540039e9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:20.627969 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493801856}
W20260709 13:03:20.695508  1736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.000s	sys 0.013s
W20260709 13:03:20.381368  6628 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.219s	user 0.012s	sys 0.003s
W20260709 13:03:20.705382  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.004s	sys 0.010s
W20260709 13:03:20.710996  6307 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.000s	sys 0.003s
W20260709 13:03:20.710614  1585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.007s	sys 0.029s
I20260709 13:03:20.695681  6712 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:20.703084  1741 rpcz_store.cc:269] 0709 13:03:17.365270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.365454 (+   184us) service_pool.cc:224] Handling call
0709 13:03:20.639163 (+3273709us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b540012ded4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:20.639609 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965859712}
I20260709 13:03:20.701905  1737 rpcz_store.cc:276] 0709 13:03:18.909576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:20.073713 (+1164137us) service_pool.cc:224] Handling call
0709 13:03:20.086730 (+ 13017us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:20.089139 (+  2409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.089459 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:20.719188  6628 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.720736  6743 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.703234  1739 rpcz_store.cc:276] 0709 13:03:17.964444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.252626 (+288182us) service_pool.cc:224] Handling call
0709 13:03:18.284948 (+ 32322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:19.855855 (+1570907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.641157 (+785302us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b540039dfd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:20.641626 (+   469us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3611543808}
W20260709 13:03:20.717145  1736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.001s	sys 0.013s
W20260709 13:03:20.703735  1591 rpcz_store.cc:269] 0709 13:03:18.403488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.403648 (+   160us) service_pool.cc:224] Handling call
0709 13:03:20.643109 (+2239461us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b54000fa854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:20.643444 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":670076800}
I20260709 13:03:20.703297  1742 rpcz_store.cc:276] 0709 13:03:18.854819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.858903 (+  4084us) service_pool.cc:224] Handling call
0709 13:03:18.859321 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:20.040723 (+1181402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.642179 (+601456us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:20.642621 (+   442us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2612130688}
I20260709 13:03:20.713936  6127 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.715808  6506 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.731323  6152 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:20.726217  6307 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.000s	sys 0.004s
W20260709 13:03:20.730916  1708 connection.cc:256] RPC call timeout handler was delayed by -1.4871601620000092s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
I20260709 13:03:20.740899  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 804) took 1884 ms. Trace:
I20260709 13:03:20.742782  1741 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:03:20.740720  6810 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.744204  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2750) took 1832 ms. Trace:
I20260709 13:03:20.744156  6810 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.746752  6765 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:20.741639  1736 rpcz_store.cc:276] 0709 13:03:18.856558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.798120 (+941562us) service_pool.cc:224] Handling call
0709 13:03:19.803443 (+  5323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:19.837918 (+ 34475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:19.837988 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:19.845151 (+  7163us) log.cc:844] Serialized 26 byte log entry
0709 13:03:20.740134 (+894983us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:20.740227 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.740379 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:20.740483 (+   104us) raft_consensus.cc:1685] finished
0709 13:03:20.740501 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.740812 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9554,"raft.queue_time_us":27549,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6172,"thread_start_us":849752,"threads_started":1,"wal-append.queue_time_us":800194}
I20260709 13:03:20.751377  6127 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.751670  6712 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.702265  1738 rpcz_store.cc:276] 0709 13:03:18.855464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.883770 (+ 28306us) service_pool.cc:224] Handling call
0709 13:03:18.884249 (+   479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:19.747594 (+863345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:19.747646 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:19.779263 (+ 31617us) log.cc:844] Serialized 52 byte log entry
0709 13:03:20.091790 (+312527us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:20.091850 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.091973 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:20.597963 (+505990us) raft_consensus.cc:1685] finished
0709 13:03:20.598044 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.598586 (+   542us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8410,"raft.queue_time_us":81469,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":5971,"thread_start_us":1110999,"threads_started":2,"wal-append.queue_time_us":428176}
I20260709 13:03:20.744819  1741 rpcz_store.cc:276] 0709 13:03:18.911721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:20.741586 (+1829865us) service_pool.cc:224] Handling call
0709 13:03:20.742310 (+   724us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:20.743344 (+  1034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.744105 (+   761us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:20.744751  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2788) took 1928 ms. Trace:
I20260709 13:03:20.749590  6743 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.748895  6810 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.760665  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2766) took 2402 ms. Trace:
I20260709 13:03:20.760061  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2753) took 1833 ms. Trace:
I20260709 13:03:20.763123  1736 rpcz_store.cc:276] 0709 13:03:18.923933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:20.750911 (+1826978us) service_pool.cc:224] Handling call
0709 13:03:20.751401 (+   490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:20.757295 (+  5894us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:20.757333 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:20.757377 (+    44us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:20.757414 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.757492 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.757675 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:20.765177  6152 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:20.767943  1708 connection.cc:256] RPC call timeout handler was delayed by -1.361300257000039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
W20260709 13:03:20.770145  1708 connection.cc:256] RPC call timeout handler was delayed by -1.1338138150000532s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
W20260709 13:03:20.768229  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.842s	user 0.016s	sys 0.011s
I20260709 13:03:20.761092  1590 rpcz_store.cc:276] 0709 13:03:18.356850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.358258 (+  1408us) service_pool.cc:224] Handling call
0709 13:03:18.358723 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.465133 (+106410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.758674 (+2293541us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:20.759118 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":192381184}
W20260709 13:03:20.773818  1706 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.852s (SENT) [suppressed 12 similar messages]
I20260709 13:03:20.774902  6498 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:20.776139  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.017s	sys 0.011s
I20260709 13:03:20.782665  6334 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.756906  1585 rpcz_store.cc:276] 0709 13:03:18.815882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.816094 (+   212us) service_pool.cc:224] Handling call
0709 13:03:18.816470 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.816706 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:19.851559 (+1034853us) log.cc:844] Serialized 14 byte log entry
0709 13:03:20.743894 (+892335us) log.cc:844] Serialized 14 byte log entry
0709 13:03:20.744199 (+   305us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:20.744261 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:20.744353 (+    92us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.744471 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.744676 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55881,"prepare.queue_time_us":354525,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":110,"raft.queue_time_us":23134,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":305,"thread_start_us":1834822,"threads_started":2,"wal-append.queue_time_us":812056}
I20260709 13:03:20.704504  1659 rpcz_store.cc:276] 0709 13:03:15.972393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:16.016523 (+ 44130us) service_pool.cc:224] Handling call
0709 13:03:16.544230 (+527707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:17.095622 (+551392us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 13:03:17.095978 (+   356us) log.cc:844] Serialized 14 byte log entry
0709 13:03:17.762716 (+666738us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b1800494b60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:18.068837 (+306121us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.070407 (+  1570us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.070609 (+   202us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:18.070647 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.071032 (+   385us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:18.366695 (+295663us) raft_consensus.cc:1685] finished
0709 13:03:18.382824 (+ 16129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.643707 (+2260883us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b540013ced0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:20.643972 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":540964,"prepare.queue_time_us":778,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":56,"raft.queue_time_us":718961,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":16672,"spinlock_wait_cycles":1228121216,"thread_start_us":972189,"threads_started":2,"wal-append.queue_time_us":419346,"wal-append.run_cpu_time_us":4863,"wal-append.run_wall_time_us":1887834}
W20260709 13:03:20.802487  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 949809us
W20260709 13:03:20.821676  6583 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.431s	user 0.012s	sys 0.009s
I20260709 13:03:20.822455  6583 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:20.826297  1590 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:20.801632  6032 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.839759  6752 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:20.825215  6583 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.829654  6628 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.861660  1588 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:20.862532  6033 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 13:03:20.863008  6033 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:20.858028  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:20.868562  6152 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.869593  6032 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.873812  6032 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:20.874369  6595 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:20.880316  1736 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:20.882242  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:20.883824  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:20.885038  1590 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:20.886942  6376 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.888567  6376 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:20.890921  6846 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:20.893076  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:20.889781  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:20.893369  6805 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.559s	user 0.006s	sys 0.019s
I20260709 13:03:20.895148  6846 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:20.900264  6541 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:20.900867  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2689) took 2268 ms. Trace:
I20260709 13:03:20.901340  1662 rpcz_store.cc:276] 0709 13:03:18.632703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.687527 (+ 54824us) service_pool.cc:224] Handling call
0709 13:03:18.687977 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.695573 (+  7596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.695624 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:18.728690 (+ 33066us) log.cc:844] Serialized 26 byte log entry
0709 13:03:18.929164 (+200474us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:18.929230 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.929336 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:20.892037 (+1962701us) spinlock_profiling.cc:243] Waited 3.76 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:20.900213 (+  8176us) raft_consensus.cc:1685] finished
0709 13:03:20.900243 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.900625 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":135,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":17944,"spinlock_wait_cycles":8655104,"thread_start_us":112440,"threads_started":1,"wal-append.queue_time_us":373386}
I20260709 13:03:20.905443  1590 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:20.911597  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:20.917143  1591 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:20.915653  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:20.899663  6846 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.912055  6138 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader election won for term 9
I20260709 13:03:20.943154  6033 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:20.961968  6580 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:20.998222  6580 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:20.998780  6536 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.920s	user 0.016s	sys 0.011s
I20260709 13:03:20.991746  6138 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:20.999814  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 808) took 1131 ms. Trace:
I20260709 13:03:21.000907  6580 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:20.964514  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:20.915704  6763 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.145s	user 0.010s	sys 0.007s
I20260709 13:03:20.993278  6506 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:20.993875  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:20.994063  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.782s (SENT)
I20260709 13:03:20.994561  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:20.995858  1634 connection.cc:256] RPC call timeout handler was delayed by -1.5827076289999695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
I20260709 13:03:21.000283  1738 rpcz_store.cc:276] 0709 13:03:19.868860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:20.753788 (+884928us) service_pool.cc:224] Handling call
0709 13:03:20.754617 (+   829us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:20.755647 (+  1030us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:20.755714 (+    67us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:20.803569 (+ 47855us) log.cc:844] Serialized 26 byte log entry
0709 13:03:20.850130 (+ 46561us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:20.850185 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:20.850278 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:20.999474 (+149196us) raft_consensus.cc:1685] finished
0709 13:03:20.999501 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:20.999714 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":257,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":243,"thread_start_us":22407,"threads_started":1,"wal-append.queue_time_us":101336}
W20260709 13:03:21.004114  1634 connection.cc:256] RPC call timeout handler was delayed by -1.3937076289999482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
W20260709 13:03:21.004701  1634 connection.cc:256] RPC call timeout handler was delayed by -1.3334076289999643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
I20260709 13:03:20.996538  6600 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:21.005023  1634 connection.cc:256] RPC call timeout handler was delayed by -1.2614076289999616s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
W20260709 13:03:21.005407  1634 connection.cc:256] RPC call timeout handler was delayed by -1.2569076289999543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
I20260709 13:03:21.006098  1733 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:21.006842  6595 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:21.007743  6498 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.008368  6498 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.009332  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2789) took 2167 ms. Trace:
W20260709 13:03:21.005754  1634 connection.cc:256] RPC call timeout handler was delayed by -1.2164076290000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260709 13:03:21.010576  1634 connection.cc:256] RPC call timeout handler was delayed by -1.1219076289999634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20390000000000003s for a response
W20260709 13:03:21.010979  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0859076289999621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20790000000000003s for a response
W20260709 13:03:21.011364  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0562076289999709s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
W20260709 13:03:21.011754  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0454076289999534s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
I20260709 13:03:21.011787  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2671) took 4828 ms. Trace:
W20260709 13:03:21.012241  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0382076289999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260709 13:03:21.012885  1634 connection.cc:256] RPC call timeout handler was delayed by -1.0130076289999579s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21600000000000003s for a response
I20260709 13:03:21.008387  6506 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.014168  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:21.016264  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.016788  6506 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:21.017794  6736 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:21.018843  6736 raft_consensus.cc:2764] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:21.006139  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:03:21.019462  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.002s (SENT)
I20260709 13:03:21.021332  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:21.007747  6595 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:21.023317  1708 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.008s (SENT)
I20260709 13:03:21.023664  6754 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:21.024513  6754 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:21.009392  6846 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:21.025961  6600 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:21.026527  1708 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.179s (SENT)
W20260709 13:03:21.027271  6824 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.953s	user 0.006s	sys 0.020s
I20260709 13:03:21.027141  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:21.028401  6754 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:21.030125  6595 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.003021  6583 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.012392  1664 rpcz_store.cc:276] 0709 13:03:16.182823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.680202 (+1497379us) service_pool.cc:224] Handling call
0709 13:03:18.364533 (+684331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:18.435066 (+ 70533us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:18.435107 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:18.435910 (+   803us) log.cc:844] Serialized 52 byte log entry
0709 13:03:18.772688 (+336778us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:18.772752 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:18.772851 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:19.921622 (+1148771us) raft_consensus.cc:1685] finished
0709 13:03:19.921669 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:21.011159 (+1089490us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b54001a77d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:21.011552 (+   393us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4123,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":12825,"spinlock_wait_cycles":1573084032,"thread_start_us":276624,"threads_started":1,"wal-append.queue_time_us":390547,"wal-append.run_cpu_time_us":10330,"wal-append.run_wall_time_us":2088570}
I20260709 13:03:21.034623  6714 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:21.009908  1593 rpcz_store.cc:276] 0709 13:03:18.842020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.842236 (+   216us) service_pool.cc:224] Handling call
0709 13:03:18.842628 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:19.713199 (+870571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.008640 (+1295441us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54003b79d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:21.009110 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002959232}
I20260709 13:03:21.037406  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 803) took 3874 ms. Trace:
W20260709 13:03:21.038053  1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2785) took 2294 ms (client timeout 2887 ms). Trace:
I20260709 13:03:21.040098  1588 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:21.040560  6754 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.047586  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.048700  1591 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:21.046276  1587 rpcz_store.cc:269] 0709 13:03:18.743275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.743430 (+   155us) service_pool.cc:224] Handling call
0709 13:03:21.037620 (+2294190us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54000fa354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:21.037851 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302890880}
I20260709 13:03:21.000358  6138 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.046181  1668 rpcz_store.cc:276] 0709 13:03:17.162651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.763815 (+601164us) service_pool.cc:224] Handling call
0709 13:03:18.380040 (+616225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.410966 (+ 30926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.029288 (+2618322us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b540030b950. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:21.037196 (+  7908us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1416279936}
I20260709 13:03:21.060263  6570 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.045778  6714 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:20.999295  6536 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.081267  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:21.086532  6580 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 13:03:21.099771  6714 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:21.102717  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2740) took 3138 ms (client timeout 2222 ms). Trace:
I20260709 13:03:21.103607  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:21.104391  2315 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:21.103281  1735 rpcz_store.cc:269] 0709 13:03:17.963625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.156559 (+192934us) service_pool.cc:224] Handling call
0709 13:03:21.102278 (+2945719us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:21.102464 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":452652800}
I20260709 13:03:21.081032  6541 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:03:21.110234  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:21.111073  6541 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:21.111593  6541 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:21.112192  6665 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 13:03:21.117372  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.078037  6583 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.113104  6665 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.201370  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.202747  1668 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:03:21.203222  1668 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:21.203536  1668 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:03:21.203848  1668 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:21.210296  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.190724  6714 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:21.191412  6665 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.214511  2315 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:21.217748  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.218694  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.118292  6754 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.113142  6541 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:21.223313  6748 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.903s	user 0.016s	sys 0.031s
I20260709 13:03:21.224709  1666 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:21.221297  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.225327  1666 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:21.228276  1659 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:21.229408  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.230281  6536 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:21.235561  6786 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.241909  6786 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.243016  2316 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:21.240387  6743 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:21.244115  6743 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.246199  1661 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:21.246730  1661 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:21.248108  1661 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.224624  6583 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:21.263244  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.273298  1661 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:21.275553  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 259537us
I20260709 13:03:21.276482  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2050) took 5337 ms. Trace:
I20260709 13:03:21.279708  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.276916  1740 rpcz_store.cc:276] 0709 13:03:15.900511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.104492 (+1203981us) service_pool.cc:224] Handling call
0709 13:03:17.138055 (+ 33563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:18.318448 (+1180393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.191602 (+2873154us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540012e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:21.237471 (+ 45869us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2649738496}
I20260709 13:03:21.288986  1666 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.305699  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.330197  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:21.347767  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.350206  1666 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:21.348802  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.317242  1661 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:03:21.322620  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.333611  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.307147  6785 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:21.356788  6422 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:21.339725  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.359778  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.369966  1667 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:21.364722  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2733) took 3406 ms (client timeout 2112 ms). Trace:
I20260709 13:03:21.365937  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2780) took 2863 ms. Trace:
I20260709 13:03:21.366971  2315 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:21.374552  1664 raft_consensus.cc:2368] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13.
W20260709 13:03:21.370887  1733 rpcz_store.cc:269] 0709 13:03:17.958221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:17.959972 (+  1751us) service_pool.cc:224] Handling call
0709 13:03:21.364456 (+3404484us) spinlock_profiling.cc:243] Waited 2.92 s on lock 0x7b54003520d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:21.364638 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6724893440}
I20260709 13:03:21.371671  1584 rpcz_store.cc:276] 0709 13:03:18.502465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.504481 (+  2016us) service_pool.cc:224] Handling call
0709 13:03:18.521465 (+ 16984us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:18.668582 (+147117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.365359 (+2696777us) spinlock_profiling.cc:243] Waited 97.5 ms on lock 0x7b540031e554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:21.365742 (+   383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":224198400}
I20260709 13:03:21.378921  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.379248  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.380115  1659 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:21.380894  6422 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.381428  1587 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:21.381914  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.382669  1660 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.383845  6422 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:21.384675  6806 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.385512  1665 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:21.388592  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.388695  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.389627  1661 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:21.390254  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.390513  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.392040  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.388854  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.393487  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2694) took 2085 ms. Trace:
I20260709 13:03:21.393828  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.393924  1663 rpcz_store.cc:276] 0709 13:03:19.191134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.191845 (+   711us) service_pool.cc:224] Handling call
0709 13:03:19.192272 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:19.192621 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:19.275015 (+ 82394us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.015902 (+1740887us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b180043d800. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff05e5 00007fcdedff0568
0709 13:03:21.034913 (+ 19011us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.195808 (+160895us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.214505 (+ 18697us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:21.219576 (+  5071us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:21.226647 (+  7071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.240124 (+ 13477us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:21.275943 (+ 35819us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1606399,"prepare.queue_time_us":1825269,"prepare.run_cpu_time_us":144,"prepare.run_wall_time_us":145,"raft.queue_time_us":111,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":19316,"spinlock_wait_cycles":38235136,"thread_start_us":150048,"threads_started":1,"wal-append.queue_time_us":196921,"wal-append.run_cpu_time_us":6910,"wal-append.run_wall_time_us":1536474}
I20260709 13:03:21.395871  1585 raft_consensus.cc:2410] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:03:21.396586  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.397789  6600 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:21.397620  6820 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 13:03:21.345335  6743 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.403527  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.369346  6785 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:21.405951  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.408001  6786 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:21.408883  1665 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:21.385387  6806 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.411339  6806 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.417703  1662 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.418434  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.419587  1585 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:21.398805  6768 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:21.420537  6395 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.421128  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:21.421734  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.422043  2316 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:21.423235  1557 connection.cc:256] RPC call timeout handler was delayed by -1.6510637499999916s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2413s for a response
I20260709 13:03:21.424276  6828 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.424937  6828 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.426294  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.427253  6828 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.427821  6033 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 8 FOLLOWER]: Advancing to term 9
I20260709 13:03:21.428606  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.428730  6623 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:21.429414  6623 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.430720  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.430951  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.431537  2312 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:21.431893  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.432139  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.417333  1734 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: [7.5-11.6]
I20260709 13:03:21.433847  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:21.434085  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2752) took 2512 ms. Trace:
I20260709 13:03:21.434266  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:21.434455  1734 rpcz_store.cc:276] 0709 13:03:18.921430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.416377 (+2494947us) service_pool.cc:224] Handling call
0709 13:03:21.416880 (+   503us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:21.433714 (+ 16834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.433998 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:21.398618  6570 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.436053  6570 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.438321  6570 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.439415  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:21.440205  1659 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:21.431941  6623 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.420802  6768 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.452124  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.452642  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.427914  6828 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.472707  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:21.474256  6757 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
W20260709 13:03:21.478828  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.459491  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.483975  1668 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:21.459652  6395 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:21.456717  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.462805  6785 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.490797  2310 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:21.453001  6754 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:21.498777  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.474331  1661 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:21.510126  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76379.4us
I20260709 13:03:21.512219  1660 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:21.514366  2309 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:21.516505  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2790) took 2666 ms. Trace:
I20260709 13:03:21.516963  1589 rpcz_store.cc:276] 0709 13:03:18.849838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:18.849987 (+   149us) service_pool.cc:224] Handling call
0709 13:03:18.850383 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:20.707882 (+1857499us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.492577 (+784695us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b54003b1fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:21.516427 (+ 23850us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3796263680}
I20260709 13:03:21.518872  6785 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:21.522024  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.524214  6768 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.527120  6498 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:21.527547  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 232206us
I20260709 13:03:21.528666  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:21.532624  2316 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:21.538959  1662 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:21.466467  6828 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.584209  6541 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:21.559051  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.561900  6665 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.574472  6781 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:21.532109  6765 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:21.584403  6828 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.593454  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.557826  2318 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:21.594239  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:21.598106  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.547205  1667 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:21.591392  1584 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:21.577302  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:21.609932  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.612823  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.614048  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.618044  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 735) took 2610 ms. Trace:
I20260709 13:03:21.615924  1585 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:21.621603  1591 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:21.603123  6736 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:21.612151  6856 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 13:03:21.645726  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.627599  2309 raft_consensus.cc:2376] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14. Re-sending same reply.
I20260709 13:03:21.632817  1590 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:21.623404  1592 rpcz_store.cc:276] 0709 13:03:19.007608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.778805 (+771197us) service_pool.cc:224] Handling call
0709 13:03:19.779225 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:19.886516 (+107291us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:19.886748 (+   232us) log.cc:844] Serialized 14 byte log entry
0709 13:03:20.749455 (+862707us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:20.749611 (+   156us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:20.749693 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.608336 (+858643us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54001c4154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:21.608416 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:21.617966 (+  9550us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17585,"prepare.queue_time_us":14748,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":18371,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"spinlock_wait_cycles":246151424,"thread_start_us":828374,"threads_started":1,"wal-append.queue_time_us":377006,"wal-append.run_cpu_time_us":2452,"wal-append.run_wall_time_us":1082597}
I20260709 13:03:21.638831  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.640153  6745 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.646942  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.592200  6765 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:21.650931  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:21.645604  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 129) took 19501 ms (client timeout 15000 ms). Trace:
I20260709 13:03:21.603223  6820 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:03:21.656121  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.665096  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.666582  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:21.668088  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 139654us
I20260709 13:03:21.656860  6765 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.671316  2311 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:21.670938  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:21.674580  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:21.675310  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:21.679356  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.680913  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.683938  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.683966  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.684733  1591 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:21.685752  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.686115  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.686702  1661 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:21.686903  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.687685  1666 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:21.689296  1666 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:21.692787  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.696444  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:21.702586  6857 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 13:03:21.703172  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.711751  1666 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:21.714306  1661 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:21.718555  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.719331  1557 leader_election.cc:400] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14.
I20260709 13:03:21.720134  1557 leader_election.cc:403] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:03:21.722561  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.694481  6768 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:21.726298  1737 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 13:03:21.734977  1587 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:21.735369  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.673902  6745 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:21.745262  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.894s (SENT)
I20260709 13:03:21.744911  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:21.752158  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.727335  6768 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.754463  1585 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:21.759514  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70875.7us
I20260709 13:03:21.761025  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:21.761624  6811 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.762995  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.768189  6745 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:21.770119  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.775153  1740 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:21.758111  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.773332  1734 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 13:03:21.788018  1738 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:21.791047  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.722025  6842 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.157s
I20260709 13:03:21.793476  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.773695  6768 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:21.794503  1587 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:21.796319  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 736) took 2209 ms. Trace:
I20260709 13:03:21.797101  1661 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:21.802345  1733 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [11.7-13.8]
W20260709 13:03:21.803159  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70118.5us
I20260709 13:03:21.803751  6852 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:21.785897  6665 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:21.815132  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:21.815783  1705 connection.cc:256] RPC call timeout handler was delayed by -1.0125814019999666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17920000000000003s for a response
I20260709 13:03:21.816579  1736 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
W20260709 13:03:21.821005  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 105672us
I20260709 13:03:21.797704  6745 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:21.825325  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:21.829787  1591 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:21.831367  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.834172  1593 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:21.837299  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 163138us
I20260709 13:03:21.799276  1586 rpcz_store.cc:276] 0709 13:03:19.463372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:19.904310 (+440938us) service_pool.cc:224] Handling call
0709 13:03:19.904791 (+   481us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:19.910440 (+  5649us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:19.948787 (+ 38347us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.363573 (+1414786us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b18002d7300. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:03:21.449698 (+ 86125us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:21.643932 (+194234us) log.cc:844] Serialized 26 byte log entry
0709 13:03:21.644155 (+   223us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:21.644194 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.644268 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:21.649172 (+  4904us) raft_consensus.cc:1685] finished
0709 13:03:21.655552 (+  6380us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:21.672498 (+ 16946us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32820,"prepare.queue_time_us":37947,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":8810,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":26040,"spinlock_wait_cycles":31125376,"thread_start_us":22194,"threads_started":1,"wal-append.queue_time_us":1456852}
I20260709 13:03:21.840202  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.842460  1587 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:21.843571  6622 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:21.847090  1734 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [10.6-12.7]
I20260709 13:03:21.848488  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:21.654817  1486 rpcz_store.cc:269] 0709 13:03:02.143905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:02.145658 (+  1753us) service_pool.cc:224] Handling call
0709 13:03:21.645182 (+19499524us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:12.284953 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:12.285770 (+   817us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:12.285823 (+    53us) write_op.cc:435] Acquired schema lock
0709 13:03:12.285841 (+    18us) tablet.cc:662] Decoding operations
0709 13:03:12.287127 (+  1286us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:12.287235 (+   108us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:12.287272 (+    37us) tablet.cc:685] Acquiring locks for 13 operations
0709 13:03:12.288587 (+  1315us) tablet.cc:701] Row locks acquired
0709 13:03:12.288642 (+    55us) write_op.cc:260] PREPARE: finished
0709 13:03:12.288829 (+   187us) write_op.cc:270] Start()
0709 13:03:12.295220 (+  6391us) write_op.cc:276] Timestamp: P: 1783602192288756 usec, L: 0
0709 13:03:12.295264 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 13:03:12.296210 (+   946us) log.cc:844] Serialized 5322 byte log entry
0709 13:03:13.622420 (+1326210us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:15.090919 (+1468499us) op_driver.cc:464] REPLICATION: finished
0709 13:03:15.345945 (+255026us) write_op.cc:301] APPLY: starting
0709 13:03:15.346063 (+   118us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:15.707026 (+360963us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:15.707084 (+    58us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:15.736764 (+ 29680us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:15.737267 (+   503us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 13:03:15.737366 (+    99us) write_op.cc:312] APPLY: finished
0709 13:03:15.750872 (+ 13506us) log.cc:844] Serialized 121 byte log entry
0709 13:03:15.751134 (+   262us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:15.751351 (+   217us) write_op.cc:454] Released schema lock
0709 13:03:15.751449 (+    98us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:15.752041 (+   592us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:15.752089 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1085822,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":254658,"apply.run_cpu_time_us":24454,"apply.run_wall_time_us":406024,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58493,"num_ops":13,"prepare.queue_time_us":243610,"prepare.run_cpu_time_us":82863,"prepare.run_wall_time_us":171338,"raft.queue_time_us":764048,"raft.run_cpu_time_us":47405,"raft.run_wall_time_us":301202,"replication_time_us":2741975,"spinlock_wait_cycles":63276544,"thread_start_us":1229276,"threads_started":4,"wal-append.queue_time_us":115961,"wal-append.run_cpu_time_us":42626,"wal-append.run_wall_time_us":4359412}]]}
I20260709 13:03:21.851294  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.858094  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.859133  6623 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.859537  6623 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.859894  6623 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.860265  2314 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:21.861289  6395 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.861716  6395 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.862051  6395 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.863490  6485 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.863997  6485 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.841651  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:21.874715  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 113881us
I20260709 13:03:21.876832  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.877490  1591 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:03:21.878835  6861 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 13:03:21.882129  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.882750  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.884963  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.876140  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:21.881238  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.892194  1735 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:03:21.893702  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2751) took 2980 ms. Trace:
I20260709 13:03:21.894186  1739 rpcz_store.cc:276] 0709 13:03:18.913739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.521986 (+2608247us) service_pool.cc:224] Handling call
0709 13:03:21.522536 (+   550us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:21.523109 (+   573us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:21.523144 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:21.523920 (+   776us) log.cc:844] Serialized 26 byte log entry
0709 13:03:21.738803 (+214883us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:21.758196 (+ 19393us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.783174 (+ 24978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.783264 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:21.891252 (+107988us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b1800242200. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:21.893330 (+  2078us) raft_consensus.cc:1685] finished
0709 13:03:21.893457 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:21.893610 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19144,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":29625,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":6320,"spinlock_wait_cycles":3823872,"thread_start_us":214453,"threads_started":1,"wal-append.queue_time_us":269372}
I20260709 13:03:21.897671  1733 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:21.897969  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.899441  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.900666  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.901458  1585 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:03:21.903258  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:21.903898  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:21.905325  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:21.872176  6786 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:21.915711  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.840601  6811 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:21.876289  6570 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:21.923612  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.917269  2311 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:21.928509  2311 raft_consensus.cc:606] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.932193  1587 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:21.933532  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.934551  1659 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:21.935621  1663 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:21.937683  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.937284  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:21.938670  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.922183  6138 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.941815  1665 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:21.943706  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:21.917915  6786 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:21.929040  6828 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:21.954064  1667 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:21.949260  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.957077  1660 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 9.
I20260709 13:03:21.959547  1593 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:03:21.960314  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:21.961334  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:21.946790  6486 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:21.961163  1735 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:21.970973  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:21.971303  2316 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:21.971776  1593 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:21.943470  6138 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:21.974076  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.974803  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:21.978008  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.991436  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.972258  6486 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.006009  1584 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:21.994547  6485 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.011806  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.012655  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:21.978246  6138 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.016170  2314 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:22.017184  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.017374  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.018872  1734 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:22.022270  1584 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:22.024295  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:21.996071  6842 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:22.044620  2316 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:22.009624  6541 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:22.069761  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:22.048862  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90956.9us
W20260709 13:03:22.073550  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 75192.7us
W20260709 13:03:22.082610  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.014020  6486 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.084100  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:22.013197  6806 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14.
I20260709 13:03:22.086578  1733 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:03:22.070565  6033 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.614s	user 0.013s	sys 0.007s
W20260709 13:03:22.099326  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147867us
I20260709 13:03:22.093963  6422 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:22.102909  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.104959  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.032130  6828 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:22.106905  1734 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:22.107985  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.108201  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.108747  1733 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:22.108997  1734 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:22.112208  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.113067  6609 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.139s
W20260709 13:03:22.113946  6865 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260709 13:03:22.118347  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:22.119781  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.106814  6422 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.105949  2312 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 9.
I20260709 13:03:22.081447  6541 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:22.129635  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:22.019279  6745 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:22.133751  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:22.028854  6395 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:22.152711  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53499.8us
I20260709 13:03:22.155488  1733 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:22.151685  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.158893  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.160224  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:22.159865  1733 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 9.
I20260709 13:03:22.163424  1590 raft_consensus.cc:2376] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14. Re-sending same reply.
I20260709 13:03:22.163239  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.166218  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.167467  1734 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:22.167665  1586 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:03:22.147538  6541 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.169140  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.171604  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.173671  1588 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:22.174165  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:22.175853  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.156944  6376 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:22.181589  1741 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 9. Current term is 11. Ops: [9.9-9.9]
I20260709 13:03:22.184053  6422 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.173573  6628 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:22.185873  6628 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:22.186271  6628 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:22.187182  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:22.176594  1589 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:22.120499  6781 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 13:03:22.178702  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.207963  1586 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:22.210366  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.216459  1593 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:22.177874  6376 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:22.194834  6852 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:22.195003  6745 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.226552  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.238000  6786 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:22.241547  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:22.242287  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:22.243013  1632 connection.cc:256] RPC call timeout handler was delayed by -1.999765970999988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1889s for a response
W20260709 13:03:22.243551  1632 connection.cc:256] RPC call timeout handler was delayed by -1.9592659710000362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260709 13:03:22.244012  1632 connection.cc:256] RPC call timeout handler was delayed by -1.7990659710000045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
W20260709 13:03:22.244369  1632 connection.cc:256] RPC call timeout handler was delayed by -1.700065971000015s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2222s for a response
I20260709 13:03:22.250658  6714 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.251302  6714 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:22.232247  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:03:22.245334  1632 connection.cc:256] RPC call timeout handler was delayed by -1.1437820700000429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17820000000000003s for a response
I20260709 13:03:22.253993  6376 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.259250  6376 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:22.267520  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 122153us
I20260709 13:03:22.173527  6811 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:22.279805  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:22.301213  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:22.280541  6867 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
W20260709 13:03:22.272557  6765 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.002s
I20260709 13:03:22.338550  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:22.324885  6745 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:22.331861  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:22.332834  1734 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:22.327647  6768 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.340596  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:22.330308  6714 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.344623  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.266185  6828 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.365857  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.366781  1738 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:22.376513  6628 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.378554  1591 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:22.381862  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.335798  6765 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.410233  6736 raft_consensus.cc:2764] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:22.411418  6736 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:22.411779  6736 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:22.412129  6736 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:22.414508  6859 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.620s	user 0.003s	sys 0.005s
I20260709 13:03:22.416858  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2082) took 3227 ms. Trace:
I20260709 13:03:22.418247  6628 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:22.417235  1742 rpcz_store.cc:276] 0709 13:03:19.189835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.397347 (+2207512us) service_pool.cc:224] Handling call
0709 13:03:21.398147 (+   800us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:21.415852 (+ 17705us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:21.416918 (+  1066us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.628906 (+211988us) log.cc:844] Serialized 14 byte log entry
0709 13:03:21.650278 (+ 21372us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:21.690107 (+ 39829us) log.cc:844] Serialized 52 byte log entry
0709 13:03:21.690293 (+   186us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:21.690329 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:21.690409 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:21.846099 (+155690us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b1800366d60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f
0709 13:03:22.413906 (+567807us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:22.416593 (+  2687us) raft_consensus.cc:1685] finished
0709 13:03:22.416623 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:22.416777 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":835,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":86,"raft.queue_time_us":49043,"raft.run_cpu_time_us":3220,"raft.run_wall_time_us":195646,"spinlock_wait_cycles":13941504,"thread_start_us":210959,"threads_started":1,"wal-append.queue_time_us":315767}
I20260709 13:03:22.420933  6852 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.363499  6395 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:22.424304  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 140002us
I20260709 13:03:22.431807  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:22.340113  6138 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:22.436311  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 151301us
I20260709 13:03:22.437808  1592 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:22.425312  6623 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.565s	user 0.013s	sys 0.000s
I20260709 13:03:22.439457  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.443826  6623 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.449616  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 974) took 1007 ms. Trace:
I20260709 13:03:22.453085  2312 rpcz_store.cc:276] 0709 13:03:21.442113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.451745 (+  9632us) service_pool.cc:224] Handling call
0709 13:03:22.449267 (+997522us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:22.449540 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501790976}
W20260709 13:03:22.454058  6785 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.934s	user 0.016s	sys 0.000s
I20260709 13:03:22.450265  2316 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 13:03:22.456022  1668 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 13:03:22.456390  1668 pending_rounds.cc:85] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:22.456606  1668 pending_rounds.cc:107] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:03:22.433799  6486 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.394147  6541 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.456902  1668 raft_consensus.cc:2889] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:22.464994  6541 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:22.466897  6138 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:22.470661  6765 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.471000  6665 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.471590  6665 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.472734  1739 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:22.473883  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.474558  1738 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:22.475633  1733 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:22.475884  1740 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:22.476639  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 374820us
I20260709 13:03:22.476969  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.477746  1737 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:22.478018  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 308666us
I20260709 13:03:22.479068  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:22.480545  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.481199  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:22.482270  6486 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:22.492261  2309 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:22.482180  6811 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:22.482824  6883 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:22.520975  6811 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.522436  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.523337  1733 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.484251  6736 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.493367  6623 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:22.493625  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:22.494422  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:22.529732  1660 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:22.495203  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:22.530964  1667 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:22.496515  6842 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:22.496737  6745 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:22.497221  6852 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.548856  6852 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.550446  6736 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.551275  1661 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:22.551520  6852 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:22.497933  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:22.551800  1488 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:03:22.552807  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54986.5us
I20260709 13:03:22.553747  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:22.554473  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:22.555137  1661 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:22.557297  6892 raft_consensus.cc:2764] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 13:03:22.498423  6768 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.499387  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:22.502065  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:22.561782  1665 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:22.503420  1659 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:22.563189  1659 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:22.482519  6754 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:22.504422  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46496us
I20260709 13:03:22.504582  6884 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:22.565301  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2048) took 1221 ms. Trace:
I20260709 13:03:22.565781  1668 rpcz_store.cc:276] 0709 13:03:21.344135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:21.485014 (+140879us) service_pool.cc:224] Handling call
0709 13:03:21.485402 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:22.518638 (+1033236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:22.565066 (+ 46428us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b5400014ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:22.565242 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2231494912}
I20260709 13:03:22.566267  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:22.505796  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:22.568035  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 62338us
I20260709 13:03:22.570528  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:22.571367  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:22.572139  2310 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:22.571885  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:22.573024  2316 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:22.573230  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:22.479069  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:22.516130  1664 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 13:03:22.574748  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95753.5us
I20260709 13:03:22.575707  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:22.576289  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.577020  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:22.528477  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:22.577693  6883 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:22.577862  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 96370us
I20260709 13:03:22.579308  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:22.579954  6422 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:22.571406  1586 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:22.530249  1663 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:22.530486  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:22.581334  1665 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:22.535486  6785 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 13:03:22.536132  6842 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.541738  6811 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:22.543802  6894 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:03:22.546228  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:22.603948  1660 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.546975  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:22.605484  1662 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 13:03:22.547462  6665 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:22.570749  1592 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:22.534047  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:22.577436  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:22.587898  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88611.6us
I20260709 13:03:22.591989  1589 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:22.577458  1659 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.607988  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:22.560712  6892 raft_consensus.cc:2764] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:03:22.614130  1733 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:22.612434  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:22.615948  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:22.621304  2312 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:22.587954  6536 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.633698  2310 raft_consensus.cc:2368] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:22.607514  6884 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:22.614250  6883 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.607728  6736 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 13:03:22.624629  6754 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:22.638191  6781 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:22.648552  1661 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:22.662355  1660 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:22.618589  6852 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20260709 13:03:22.671156  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75457.9us
I20260709 13:03:22.675163  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:22.632139  6745 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 13:03:22.650511  6876 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.668586  6781 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:22.650712  6884 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:22.681713  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 154870us
I20260709 13:03:22.665182  6536 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:22.656425  6785 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.693126  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:22.683018  6736 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:22.676518  6665 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.701629  6736 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:22.702245  6881 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:22.703011  6869 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.283s	user 0.004s	sys 0.000s
I20260709 13:03:22.703694  6883 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.684916  6754 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.722426  6881 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:22.723006  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.691181  6781 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.723745  6422 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:22.725786  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79736.9us
W20260709 13:03:22.728705  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 151374us
I20260709 13:03:22.729879  6898 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:22.713011  6736 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:22.737354  6899 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:22.672946  6885 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.028s
W20260709 13:03:22.729679  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102956us
I20260709 13:03:22.743434  1659 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:22.744995  1488 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:03:22.746970  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72547.5us
I20260709 13:03:22.706239  6806 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:22.692436  6876 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.752147  6891 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:22.761600  6881 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:22.731812  6884 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.773027  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:22.762562  6785 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:03:22.837752  6785 consensus_queue.cc:1243] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:22.774255  6736 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.804437  6908 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.771775  6899 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:22.760555  6665 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 13:03:22.809017  6754 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.814880  1488 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:22.765179  6898 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:22.827096  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.840194  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:22.839121  6736 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:22.817593  6891 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:22.859565  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 120063us
W20260709 13:03:22.842025  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 228319us
W20260709 13:03:22.840026  6900 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
W20260709 13:03:22.855288  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 82373.6us
I20260709 13:03:22.855901  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:22.856375  6152 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:22.856602  6902 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:22.865989  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.867991  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:22.870339  1736 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:22.876325  6481 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:22.876855  6801 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:22.877533  6801 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:22.878814  1706 leader_election.cc:400] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:22.878475  6801 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.879158  1706 leader_election.cc:403] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 13:03:22.882385  6898 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.883800  6898 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:22.847472  6541 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:22.894990  1705 leader_election.cc:400] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13.
I20260709 13:03:22.895524  1705 leader_election.cc:403] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:03:22.867919  6422 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.901461  6622 raft_consensus.cc:993] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:22.902419  6902 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:22.903434  6902 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.904096  6622 raft_consensus.cc:1081] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:22.905854  2309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:22.906984  1488 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 1 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:22.908149  6152 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:22.909336  6152 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.876952  6481 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.912470  6891 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.912829  6745 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:22.914800  6891 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:22.921367  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125374us
I20260709 13:03:22.847259  6899 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:22.909183  1488 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 5, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 5 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:22.928269  1736 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:22.862571  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:22.929852  1659 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:22.936045  2309 raft_consensus.cc:606] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:22.937942  1736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
I20260709 13:03:22.938928  1736 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:22.939399  1736 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:22.919909  6786 raft_consensus.cc:993] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
W20260709 13:03:22.956743  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 272293us
I20260709 13:03:22.899277  6541 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.958566  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:22.958527  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58445.2us
W20260709 13:03:22.965886  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 103409us
I20260709 13:03:22.976675  6903 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:22.977442  6903 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:22.978451  6903 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.958882  6745 consensus_queue.cc:1243] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:22.979856  6903 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:22.983445  6842 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:22.984804  2314 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:22.985488  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:22.991307  6498 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 10. Current term is 11.
I20260709 13:03:22.992666  6498 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:22.877060  6781 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:22.995363  6781 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:22.954172  6536 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13.
W20260709 13:03:22.997293  6912 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:03:22.999396  6905 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:22.999893  6905 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:23.000279  6905 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:22.968051  6786 raft_consensus.cc:1081] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:03:23.004349  6876 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:22.868613  6908 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.007294  6908 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:23.010471  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.014050  1736 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:23.028038  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105932us
I20260709 13:03:23.028983  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:23.044551  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.024778  6785 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:23.056172  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 161490us
I20260709 13:03:23.059386  1736 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:23.059840  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 181200us
I20260709 13:03:23.060994  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:23.061287  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.062636  1735 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:23.062777  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.063422  1666 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.064877  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:23.065356  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106980us
I20260709 13:03:23.066524  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:23.065552  1735 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:23.074278  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.043118  6745 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:23.076345  2310 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:23.079136  2312 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:23.081863  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:23.082715  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70388.7us
I20260709 13:03:23.083359  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:23.083997  6842 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.089979  6811 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:23.090690  6811 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.091166  6811 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:23.094872  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52542.3us
I20260709 13:03:23.070925  6852 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.053403  6541 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.104278  1735 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:23.112525  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.104745  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.131083  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.129468  1488 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 9, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 9 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:23.144867  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.142992  6806 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:23.146845  6806 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:23.147229  6806 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:23.149309  6481 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.135973  6891 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:23.153741  6891 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:23.158953  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:23.162689  6898 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:23.109216  6921 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:23.165798  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.117578  6422 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 13:03:23.168298  1735 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:23.168864  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64219.6us
I20260709 13:03:23.172171  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.170073  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.110433  6852 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.174407  1666 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:23.143438  6665 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.163498  6898 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.134424  6922 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 13:03:23.157917  6876 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:03:23.170312  6884 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:23.177762  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:23.178423  6919 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:03:23.192488  6898 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.193109  6876 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:23.193878  6876 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:23.194391  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.196031  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.196199  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:23.197695  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.190023  1488 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 8, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 8 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:23.221776  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.222352  6923 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 13:03:23.222818  6923 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.224632  6923 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
W20260709 13:03:23.224658  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58990.8us
I20260709 13:03:23.225795  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.226637  6920 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.214576  6891 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.139928  6785 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:23.228861  6785 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.229725  6915 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.206111  6896 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:03:23.223522  6801 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 13:03:23.197199  6884 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:23.235687  6907 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:23.236241  6907 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:23.239145  6899 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.240999  6907 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.189672  6852 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.242586  6907 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.188179  6881 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.218963  6913 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.245704  6920 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.246457  6920 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:23.186270  6918 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 13:03:23.252866  1708 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.535s (SENT)
I20260709 13:03:23.253957  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:23.256254  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58628us
I20260709 13:03:23.255609  1488 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 6, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 6 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:03:23.263109  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.265950  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.266767  1736 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.268095  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:23.270373  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74238.3us
I20260709 13:03:23.245253  6881 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:23.274935  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49242.5us
I20260709 13:03:23.272693  1488 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:23.293396  1708 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.674s (SENT)
I20260709 13:03:23.297001  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.298969  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.255153  6892 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.260730  6885 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:23.261171  6915 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.303795  6828 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:23.260950  6884 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:23.311158  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:23.289629  6852 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:23.327667  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64785.2us
I20260709 13:03:23.330653  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.307554  6876 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:23.337766  1708 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.677s (SENT)
I20260709 13:03:23.338796  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.345613  6876 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:23.346253  6876 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.348946  6665 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:23.350646  6811 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.310158  6736 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.353892  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.359144  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.317898  6921 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.356742  1736 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:23.307914  6908 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 13:03:23.326982  6899 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:23.327736  6852 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:23.328661  6885 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:23.359427  6785 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.366734  6928 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.367174  6904 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:03:23.364874  6891 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:23.371086  6929 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:23.414345  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:23.415335  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50011.5us
W20260709 13:03:23.381395  6924 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 13:03:23.382719  2316 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:23.416662  2316 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:23.389719  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.390630  6884 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.391644  6915 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:23.392279  6896 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:23.394519  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.430212  1666 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:23.395481  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.431257  1659 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:23.396672  6920 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:23.398030  6498 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:23.311251  6892 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.398850  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.406054  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.359875  6883 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.360260  6781 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:23.415568  1665 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 13:03:23.379992  6811 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:23.419171  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.419695  1488 catalog_manager.cc:5697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:23.420783  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90848.5us
W20260709 13:03:23.455202  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:23.455756  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.803s (SENT)
I20260709 13:03:23.456209  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.421196  6891 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:23.414584  6929 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.462795  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.463687  1667 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:23.462795  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.422384  2316 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.426994  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:23.432003  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 135092us
I20260709 13:03:23.405207  6852 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:23.444818  6927 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:23.445828  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:23.458900  6930 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 13:03:23.478901  1664 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.459156  6883 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.480237  6781 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.481389  6781 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.482093  6883 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:23.483323  1488 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:23.439818  6498 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.485281  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:23.486203  6908 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:23.486069  6891 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.486330  6498 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.487920  6498 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.472751  6929 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.477285  1668 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:23.493249  1668 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:23.494115  6765 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:03:23.400068  6736 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:23.461217  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97787.4us
I20260709 13:03:23.439397  6913 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:23.465430  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.473052  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.473877  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.474911  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.477174  2316 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.412895  6928 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.508585  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.496933  1488 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:23.544201  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78871.5us
I20260709 13:03:23.545192  1666 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:23.545855  1742 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:23.547962  1634 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.518855  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:23.500295  6903 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.520s	user 0.015s	sys 0.000s
W20260709 13:03:23.511374  6905 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.003s
I20260709 13:03:23.564370  1667 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:23.564994  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.561736  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.497035  6765 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.565704  1734 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.577409  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:23.584136  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.549736  6913 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:23.587194  1665 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:23.590480  1739 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:23.591171  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.594642  1737 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:23.549525  6885 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.597397  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.594349  6913 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.598714  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.600489  2316 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:23.614332  1741 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:03:23.602322  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.597579  6736 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.622931  1666 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:23.623593  1668 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 13:03:23.588085  6925 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 13:03:23.612464  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:23.612120  6929 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.553056  6828 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:23.616457  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.598750  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.630803  1742 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 10: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:23.605571  6923 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:23.625099  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47857.6us
I20260709 13:03:23.637225  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.639524  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.626187  6935 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:23.626295  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.626899  6904 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:23.643625  1734 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:23.625942  6842 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 13:03:23.592686  6892 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.654188  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.655081  1734 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:23.657475  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:23.658668  1666 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 13:03:23.566977  6905 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.661394  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.670819  1660 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:23.671692  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.673327  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.671826  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.679625  6935 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:23.680234  6935 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.680634  6935 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:23.680718  6899 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:23.681661  6899 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.644718  6765 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.682649  6899 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.683429  6929 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.683938  6765 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.684623  6765 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:23.689014  6908 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:23.683960  6899 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:23.690126  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 142375us
I20260709 13:03:23.701323  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:23.701864  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.702821  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.704485  1584 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:23.705814  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.706908  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.707612  1585 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:23.707672  6935 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.708830  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.709883  1736 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:23.710741  1589 raft_consensus.cc:2368] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:23.660368  6933 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:23.714038  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.714531  6754 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:23.715036  1584 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:23.714985  6754 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.654564  6934 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:23.646201  6903 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:23.713446  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.716385  6903 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.716044  6754 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.716945  6903 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:23.716183  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.643973  6904 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:23.659927  6932 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 13:03:23.714386  6933 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:23.681110  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.627925  6876 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.661976  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.680527  6905 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.717486  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:23.759303  1590 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:23.759537  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.760169  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.761080  1740 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:23.719218  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:23.762290  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:23.725526  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:23.763343  1742 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:23.729851  1588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:46322
I20260709 13:03:23.736771  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.764428  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.740011  6541 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:23.726915  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:23.723047  6811 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:23.741724  6876 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.769541  6876 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:23.742726  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:23.770133  1591 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.717478  6754 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:23.742101  6943 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 13:03:23.771066  6943 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:23.771519  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.772259  1734 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:23.744015  6929 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:23.719975  6904 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:23.774400  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:23.765192  1733 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:23.777429  6906 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:23.778146  6767 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:23.777861  6745 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:23.764876  1588 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:23.779482  6745 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:23.743196  6928 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.778055  6906 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:23.780155  6745 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:23.770979  6736 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:23.743373  6892 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:23.782254  6892 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.735438  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.784389  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:23.716038  6934 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.785357  6892 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.785602  6934 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:23.757445  1736 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:23.781242  6906 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.768801  6811 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.734972  6933 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:23.779839  1588 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.790234  1737 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:23.745859  6908 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:23.754848  1735 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:23.791939  6908 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.739573  1659 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:23.788748  6811 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.745091  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:23.794268  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49283.1us
I20260709 13:03:23.795023  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.795922  6736 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:23.796299  6736 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:23.796703  6736 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:23.797055  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.774430  6929 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.718413  6935 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:23.770242  6876 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:23.748346  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:23.752746  1738 raft_consensus.cc:2368] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14.
I20260709 13:03:23.804284  6876 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:23.749848  6806 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.006s
I20260709 13:03:23.805814  6806 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.808470  6806 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:23.802462  6152 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:23.810818  6786 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:23.810691  6152 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:23.811398  6786 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:23.812371  6786 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.814975  1585 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:23.819458  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82806.9us
I20260709 13:03:23.819427  1591 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:23.772403  6943 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:23.824647  6902 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:03:23.825912  6842 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:23.830802  1587 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:03:23.831600  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57705.9us
I20260709 13:03:23.833309  2316 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:03:23.843729  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46749.5us
I20260709 13:03:23.844092  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:23.840937  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 350162us
I20260709 13:03:23.845713  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:23.846490  1585 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:03:23.839321  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71369.7us
I20260709 13:03:23.847579  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:23.853893  1708 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.852s (SENT)
W20260709 13:03:23.861586  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 342874us
I20260709 13:03:23.869969  1741 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:23.871958  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.873160  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:23.838886  6905 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:23.892722  1667 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:23.895824  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:23.896839  1735 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:23.897081  1666 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 13:03:23.917236  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:23.918007  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:23.886672  6485 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:23.885696  6928 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:23.931588  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:23.843158  6883 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:03:23.932824  1660 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:23.928473  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:23.891772  6892 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.898752  6905 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:23.901630  6923 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:23.839098  6745 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.5, Last appended by leader: 5, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.916605  6781 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:23.927969  6485 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.931618  6928 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:23.900516  6481 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 13:03:23.901376  6881 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 13:03:23.956277  6929 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:23.953438  6033 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:23.954555  6905 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:23.953127  6898 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:23.906185  6376 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:03:23.935266  6828 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.960508  6892 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.980515  6485 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:23.982295  6891 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:23.927620  6885 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 13:03:24.006542  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:23.971081  6937 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:23.960762  6920 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 13:03:23.985345  6928 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:24.021179  6891 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:24.038710  6891 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:23.980912  6781 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.042900  6892 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.043723  6781 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:24.028131  6907 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.785s	user 0.008s	sys 0.010s
I20260709 13:03:24.045667  6940 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 13:03:23.981298  6498 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 13:03:24.028872  2316 raft_consensus.cc:606] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:23.993440  6806 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:23.997970  6898 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.052364  6928 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.006759  6033 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.022946  6745 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:24.068181  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 195152us
I20260709 13:03:24.103001  6033 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:24.104266  6947 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.105080  6947 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.108146  6947 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:24.076526  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 120624us
I20260709 13:03:24.098485  1588 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.015347  6884 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.050s
W20260709 13:03:24.065469  6852 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.576s	user 0.024s	sys 0.005s
W20260709 13:03:24.115593  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.000s	sys 0.026s
W20260709 13:03:24.115963  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.000s	sys 0.026s
W20260709 13:03:24.116276  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 557742us
W20260709 13:03:24.067734  6785 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.656s	user 0.010s	sys 0.012s
I20260709 13:03:24.127513  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:24.128556  2311 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:24.131976  1667 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:24.097180  6806 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.134953  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:24.141479  1660 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:24.142403  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:24.144004  2309 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:24.138131  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 266311us
I20260709 13:03:24.140306  1588 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:24.111037  6898 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:24.110667  6885 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:24.149886  1659 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:24.145444  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:24.049355  6903 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.061332  6913 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.150976  6806 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:24.128341  6785 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:24.153159  6885 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:24.434041  1661 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 13:03:24.433439  6885 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:24.216565  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71265.2us
I20260709 13:03:24.124208  6852 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.210906  6947 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:24.410326  6957 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.261s	user 0.007s	sys 0.007s
W20260709 13:03:24.410950  6942 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.389s	user 0.000s	sys 0.007s
W20260709 13:03:24.411518  6934 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.625s	user 0.011s	sys 0.014s
I20260709 13:03:24.215863  6891 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.454574  6891 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.412902  6955 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:03:24.415894  6736 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.619s	user 0.017s	sys 0.004s
I20260709 13:03:24.210413  6892 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.460568  6892 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.463557  6892 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:24.417795  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 480376us
I20260709 13:03:24.465731  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:24.417824  6903 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:24.420053  6828 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:24.467856  6884 raft_consensus.cc:2764] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:03:24.468147  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.469949  1591 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:24.471457  6736 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.473367  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:24.474349  6955 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:24.460033  6934 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.477828  6955 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:24.480448  6891 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:24.421466  6754 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.651s	user 0.015s	sys 0.008s
I20260709 13:03:24.153072  6883 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 13:03:24.232004  6941 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.109s
I20260709 13:03:24.484076  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:24.424878  6933 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.004s
I20260709 13:03:24.237998  6937 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:24.425454  6905 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:24.424263  6899 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.734s	user 0.018s	sys 0.008s
I20260709 13:03:24.429704  1662 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:24.431566  6904 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.009s
W20260709 13:03:24.431883  6950 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.193s	user 0.000s	sys 0.006s
I20260709 13:03:24.491245  6904 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.212847  6929 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.492449  6933 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.437397  1634 leader_election.cc:400] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
W20260709 13:03:24.432134  6541 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.666s	user 0.007s	sys 0.010s
W20260709 13:03:24.406031  6908 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.045s	sys 0.054s
I20260709 13:03:24.433656  6876 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:24.459326  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.037s	sys 0.083s
W20260709 13:03:24.485620  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:24.608165  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.037s	sys 0.083s
I20260709 13:03:24.490339  6905 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.427397  6928 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.495873  1634 leader_election.cc:403] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:24.493844  6929 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.500427  1592 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:24.585058  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 466250us
I20260709 13:03:24.525458  6885 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:24.502059  6908 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.046s	sys 0.055s
I20260709 13:03:24.450048  6609 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.503162  1663 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:24.683930  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2730) took 1107 ms. Trace:
I20260709 13:03:24.620841  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:24.685050  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64434.6us
W20260709 13:03:24.652815  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.452229  6947 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:24.654337  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 217190us
I20260709 13:03:24.692811  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:24.654404  6905 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:24.632587  6941 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:24.612874  6915 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.663689  6736 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:24.675338  1556 connection.cc:256] RPC call timeout handler was delayed by -2.175496800000019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:03:24.431644  6913 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.726416  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.728197  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.728971  1584 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:24.621893  6937 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:24.729893  6937 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:24.616632  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 132721us
I20260709 13:03:24.687348  1739 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:24.684660  1664 rpcz_store.cc:276] 0709 13:03:23.576797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.630769 (+ 53972us) service_pool.cc:224] Handling call
0709 13:03:23.646773 (+ 16004us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:23.935763 (+288990us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:23.935799 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:23.955947 (+ 20148us) log.cc:844] Serialized 26 byte log entry
0709 13:03:24.100462 (+144515us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:24.100519 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.122247 (+ 21728us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:24.683540 (+561293us) raft_consensus.cc:1685] finished
0709 13:03:24.683566 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:24.683819 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7456,"raft.queue_time_us":96512,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":390163,"threads_started":2,"wal-append.queue_time_us":163372}
I20260709 13:03:24.606287  6785 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:24.734203  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:24.734368  6785 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:24.697912  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74507.2us
I20260709 13:03:24.704849  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.713040  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.740762  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.721662  1591 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:24.725484  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.725941  6754 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.744414  6754 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.689370  6908 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:24.725392  6915 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.749555  6908 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:24.752738  6964 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 13:03:24.663048  6485 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.633s	user 0.015s	sys 0.000s
I20260709 13:03:24.693742  6928 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:24.753755  6928 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:24.753628  6485 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.754247  6928 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:24.725484  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.755303  1586 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:24.752614  6915 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:24.757968  6754 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:24.762768  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.770823  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.771477  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:24.771500  1588 raft_consensus.cc:2376] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12. Re-sending same reply.
I20260709 13:03:24.773113  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.774149  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.775169  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.776367  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.783267  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.784695  6975 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:24.787461  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:24.788223  1664 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:24.788266  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.789530  6745 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.790241  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.790668  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2718) took 1357 ms. Trace:
I20260709 13:03:24.790895  1586 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:24.791076  1662 rpcz_store.cc:276] 0709 13:03:23.431746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.431928 (+   182us) service_pool.cc:224] Handling call
0709 13:03:23.450565 (+ 18637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:24.491088 (+1040523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.789290 (+298202us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:24.789465 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2251072896}
I20260709 13:03:24.792311  6975 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:24.792699  6975 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:24.790089  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.790067  6745 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.797062  6975 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.798323  6933 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.799146  6975 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:24.802464  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:24.803233  1662 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:24.804577  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:24.804773  1593 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:24.805240  1659 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:24.780882  1587 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:24.806250  1587 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:24.797137  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 843) took 1393 ms. Trace:
I20260709 13:03:24.807633  1667 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:24.807788  1663 rpcz_store.cc:276] 0709 13:03:23.403407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.403562 (+   155us) service_pool.cc:224] Handling call
0709 13:03:23.403942 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:24.693737 (+1289795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.796882 (+103145us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:24.797073 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2518809344}
I20260709 13:03:24.808665  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:24.808938  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:24.714418  6904 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:24.761023  6947 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.807178  1587 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.809270  1667 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:24.845391  1587 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:24.846385  1662 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:24.809890  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.812987  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.762216  6892 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:24.861526  6754 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:24.775722  6934 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:24.847476  1659 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:24.863413  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:24.866205  6967 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:24.866657  1659 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:24.847152  6960 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:24.868340  6960 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.850404  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.856731  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:24.869784  6785 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.847162  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:24.860908  2312 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:24.761178  6767 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:24.772997  6937 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:24.865657  6955 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:24.874236  6955 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:24.875227  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.875329  6955 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.875957  1588 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:24.851511  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:24.877055  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:24.878698  6767 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.879940  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2802) took 1180 ms. Trace:
I20260709 13:03:24.880112  2311 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:24.880348  1592 rpcz_store.cc:276] 0709 13:03:23.699924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.705413 (+  5489us) service_pool.cc:224] Handling call
0709 13:03:24.879703 (+1174290us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:24.879851 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1825208704}
I20260709 13:03:24.880848  6904 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:24.882536  6915 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:24.883141  6883 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:24.848132  1593 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:24.883441  6915 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.865824  6892 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:24.894327  2311 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 13:03:24.869968  1584 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:24.894521  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.888975  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.934966  1592 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:24.881357  6904 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:24.831722  6943 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:24.903179  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:24.844364  6908 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:24.912325  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:24.701851  6609 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:24.961362  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.961668  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:24.916771  6969 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.106s	user 0.000s	sys 0.003s
I20260709 13:03:24.919899   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:03:24.895970  6903 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:24.936463  2314 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:24.941988  1587 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:03:24.967700  1587 pending_rounds.cc:85] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:03:24.947044  1659 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:24.968055  1587 pending_rounds.cc:107] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:03:24.968410  1587 raft_consensus.cc:2889] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:24.928085  6898 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.716s	user 0.016s	sys 0.000s
I20260709 13:03:24.895874  6915 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.972976  6908 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:24.973769  6908 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.974673  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.975579  6767 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:24.976087  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:24.942246  6935 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.043s	user 0.011s	sys 0.009s
I20260709 13:03:24.982182  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.985699  1588 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:03:24.998288  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:24.987926  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:24.977401  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:24.971060  6903 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:24.982978  6898 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:24.993883  6915 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:24.993930  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:24.999214  1741 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:24.999589  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.013990  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:25.000223  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.000239  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.000862  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:24.963336  6904 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:25.008738  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.021266  1742 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:25.014823  1737 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:24.996987  6935 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.011s	sys 0.009s
I20260709 13:03:25.009886  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.025476  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.021942  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.018447  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:24.963001  6943 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.027311  1734 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:25.023005  1741 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:25.022758  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.027865  1738 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:25.039868  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:25.042127  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.042316  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.038522  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:25.000389  6934 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:25.008410  6908 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:25.060477  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 289093us
I20260709 13:03:25.061560  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.070978  1584 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:25.019634  6767 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:25.005931  6811 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.065598  1733 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:25.066344  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.077817  1741 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:25.072072  1590 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 13:03:25.021389  6953 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:25.073639  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:25.086647  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:25.050875  6736 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:25.089658  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 177469us
I20260709 13:03:25.056265  6905 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:25.023757  6937 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
W20260709 13:03:25.095193  6970 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 13:03:25.120509  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99711.8us
I20260709 13:03:25.126212  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:25.076153  6767 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.095654  6876 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:25.051869  6972 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 13:03:25.125216  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:25.155683  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:25.182128  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:25.183146  1588 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:25.192777  1662 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 13:03:25.199908  1662 pending_rounds.cc:85] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 6
I20260709 13:03:25.200212  1662 pending_rounds.cc:107] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:25.190996  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.152035  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:25.210034  1587 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:25.160274  6943 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.160595  6485 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.220484  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:25.220774  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 249933us
I20260709 13:03:25.162669  6745 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.149000  6960 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.166771  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.166945  6898 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.168931  1592 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:25.157413  6767 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:25.074342  6934 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.022084  6892 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.118604  6811 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.200518  1662 raft_consensus.cc:2889] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:25.211445  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86336.2us
I20260709 13:03:25.183743  6903 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.214877  1584 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:25.200526  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:25.161929   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:25.223127  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.125218  6828 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:25.229627  1742 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:25.155951  6905 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:25.233561  1586 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:25.234319  1593 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:25.236137  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.183399  6609 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.191186  6876 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.173053  6928 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.174820  6929 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.243836  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.203835  6541 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.220220  6967 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:25.181440  6806 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.684s	user 0.008s	sys 0.004s
I20260709 13:03:25.221983  6979 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:25.239419  6745 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.281070  1557 leader_election.cc:400] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14.
I20260709 13:03:25.295811  1663 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:03:25.303179  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.305053  1662 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:25.301054  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:25.310286  1742 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [12.7-15.8]
W20260709 13:03:25.303602  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 149237us
I20260709 13:03:25.311663  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.311354  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.325780  1660 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:03:25.330782  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.335337  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.336381  1740 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:25.338028  6927 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:25.299944  6745 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.343760  1662 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:03:25.344707  6745 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.345863  1667 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:25.346696  6915 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:25.347131  1740 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:25.347431  6915 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:25.348295  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2123) took 1778 ms (client timeout 2191 ms). Trace:
W20260709 13:03:25.348852  1739 rpcz_store.cc:269] 0709 13:03:23.572677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.653840 (+ 81163us) service_pool.cc:224] Handling call
0709 13:03:25.348092 (+1694252us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b5400284454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:25.348213 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1713430912}
I20260709 13:03:25.350499  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2715) took 1955 ms. Trace:
I20260709 13:03:25.350929  1661 rpcz_store.cc:276] 0709 13:03:23.394872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.405621 (+ 10749us) service_pool.cc:224] Handling call
0709 13:03:25.350191 (+1944570us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:25.350390 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270019456}
I20260709 13:03:25.352739  1668 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:25.353626  1665 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260709 13:03:25.353992  1665 pending_rounds.cc:85] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 8
I20260709 13:03:25.354247  1665 pending_rounds.cc:107] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 13:03:25.354480  1665 raft_consensus.cc:2889] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:25.348366  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.345224  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.356280  1663 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:25.286305  6876 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:25.332062  1584 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:25.278352  6955 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 13:03:25.364959  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.368813  1739 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [14.8-17.10]
I20260709 13:03:25.374020  6915 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.379556  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.380685  1660 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:25.392087  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 238421us
W20260709 13:03:25.390544  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68885.5us
I20260709 13:03:25.329586  6929 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:25.398041  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 148101us
I20260709 13:03:25.400990  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.405941  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.418030  1663 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:25.407375  1668 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:25.408145  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:25.409413  1660 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 13:03:25.295708  6806 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.421718  1662 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:25.426769  1668 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [11.10-14.11]
I20260709 13:03:25.299434  6541 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.425683  1661 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:25.429026  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.300202  6967 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.307428  6785 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.091s
I20260709 13:03:25.369191  6898 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:25.389408  6884 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.378450  6979 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.448192  1736 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:25.450984  1739 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:25.458843  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.459700  1668 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:25.460088  6974 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:25.463243  6974 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:25.464442  6979 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.467332  6979 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.416596  6904 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.468819  1735 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:25.470026  6984 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:03:25.471386  6996 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:25.471781  6996 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:25.476174  6953 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:25.477012  6953 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.473201  6996 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.483340  6967 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.422683  6929 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.485381  6929 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:25.490532  6975 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.008s
I20260709 13:03:25.491348  2310 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:25.493223  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.470105  1740 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:25.500708  1589 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:25.421411  6536 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:25.519435  1661 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:25.550092  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:25.550343  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:25.549324  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 105759us
I20260709 13:03:25.560381  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:25.563614  2283 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:25.519923  6754 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.470036  6904 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:25.535717  6968 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.477451  6953 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.496783  6884 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.559864  6960 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.495481  6898 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.517644  6736 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:25.576864  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:25.527956  6947 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 13:03:25.536285  6955 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:25.588023  1663 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:25.537570  6536 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:25.574957  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 76503us
I20260709 13:03:25.589253  2314 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
W20260709 13:03:25.570546  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 319982us
I20260709 13:03:25.454035  6905 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.602818  1662 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:03:25.602223  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 184100us
I20260709 13:03:25.625123  1661 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:25.628055  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:25.601007  6955 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:25.603739  6152 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:25.573279  6754 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:25.650128  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48164.8us
I20260709 13:03:25.589118  6968 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:25.648854  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:25.652020  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:25.607211  6934 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.602981  6998 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:25.624035  6947 consensus_queue.cc:1243] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:25.596938  6960 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.660033  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:25.647358  6842 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:25.678817  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:25.647691  6928 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:25.666123  6992 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.269s	user 0.000s	sys 0.002s
I20260709 13:03:25.630582  6935 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 13:03:25.659411  7000 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:25.744617  7006 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:25.745201  6935 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:25.745762  6935 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:25.635251  6940 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 13:03:25.728850  6806 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.679749  6541 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:25.748114  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88164us
I20260709 13:03:25.762970  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:25.765069  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:25.679541  6998 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:25.733170  6842 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:25.744733  6884 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:25.767239  6990 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.222s	user 0.001s	sys 0.001s
I20260709 13:03:25.680078  6767 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.824753  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.825788  1587 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:25.772329  7006 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:25.810627  7004 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.141s	user 0.000s	sys 0.002s
W20260709 13:03:25.815284  7003 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
W20260709 13:03:25.848657  7007 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
W20260709 13:03:25.849745  6988 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 13:03:25.850148  6842 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.852640  6961 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.853196  6961 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.844703  6536 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.844192  6953 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.777019  6934 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.841105  6998 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:25.858146  6934 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.779800  6955 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.861315  6961 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.862848  6955 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:25.840941  6935 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.866627  6935 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:25.865058  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.870430  1660 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 10: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:25.871714  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:25.845238  6968 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.825038  6908 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.877192  6908 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.879623  6908 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:25.881922  6934 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:25.885653  7001 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
W20260709 13:03:25.885766  7005 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.205s	user 0.000s	sys 0.010s
I20260709 13:03:25.887954  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:25.894979  6898 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.898165  6754 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:25.898818  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:25.899415  6991 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 13:03:25.900415  6937 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:25.900883  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.903618  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.814263  6904 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:25.905598  6904 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.907743  1587 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:25.900972  6937 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:25.908484  6937 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:25.910679  1587 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:25.930879  6745 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:25.931874  6745 consensus_queue.cc:1243] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 13:03:25.938468  7013 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.184s	user 0.001s	sys 0.000s
I20260709 13:03:25.770871  6828 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.068s
I20260709 13:03:25.942137  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2751) took 1110 ms. Trace:
I20260709 13:03:25.944195  1659 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:25.947739  6767 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:25.942761  1664 rpcz_store.cc:276] 0709 13:03:24.831610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.871451 (+ 39841us) service_pool.cc:224] Handling call
0709 13:03:24.871935 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:24.872406 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:25.000994 (+128588us) log.cc:844] Serialized 14 byte log entry
0709 13:03:25.304413 (+303419us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:25.328665 (+ 24252us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.332932 (+  4267us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:25.333025 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.333130 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:25.941847 (+608717us) raft_consensus.cc:1685] finished
0709 13:03:25.941875 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:25.942050 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3191,"prepare.queue_time_us":203355,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":6054,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":24902,"thread_start_us":375867,"threads_started":2,"wal-append.queue_time_us":373818}
I20260709 13:03:25.907819  6904 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:25.978044  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:25.988657  6745 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.991008  6745 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:25.995195  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:25.995802  6811 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:25.997443  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2894) took 1226 ms. Trace:
I20260709 13:03:25.997605  1591 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:25.997926  1589 rpcz_store.cc:276] 0709 13:03:24.770712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.770881 (+   169us) service_pool.cc:224] Handling call
0709 13:03:25.997202 (+1226321us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54003b1ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:25.997385 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1646217984}
I20260709 13:03:25.998768  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2904) took 1134 ms. Trace:
I20260709 13:03:25.998955  6745 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:25.999140  1591 rpcz_store.cc:276] 0709 13:03:24.864828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.061149 (+196321us) service_pool.cc:224] Handling call
0709 13:03:25.997372 (+936223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.998013 (+   641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.998598 (+   585us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b54003b1ad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:25.998712 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2121986432}
I20260709 13:03:25.929271  6898 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:03:25.999334  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2817) took 1282 ms. Trace:
I20260709 13:03:26.013774  1590 rpcz_store.cc:276] 0709 13:03:24.717119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.717371 (+   252us) service_pool.cc:224] Handling call
0709 13:03:24.717872 (+   501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.078458 (+360586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:25.999128 (+920670us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b540031c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:25.999284 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":792680832}
I20260709 13:03:26.020392  1590 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:26.012166  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:26.024446  6876 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.000s
I20260709 13:03:26.015503  6967 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:26.053934  6967 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:26.055079  6967 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.037779  6968 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:25.977890  6806 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 13:03:26.045867  6996 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:26.052265  6929 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.000s
I20260709 13:03:26.031021  6745 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.013s
I20260709 13:03:26.037529  6876 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:26.074024  1666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.010s	sys 0.013s
W20260709 13:03:26.087071  1666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.010s	sys 0.013s
I20260709 13:03:26.092991  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:26.094518  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116152us
I20260709 13:03:26.094961  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.095628  1667 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:26.095628  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:26.097201  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.098227  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.067263  6968 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.120204  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:26.120965  1662 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:26.122659  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.124089  1584 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:26.124840  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.125825  1664 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 13:03:26.124493  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:26.133193  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.133705  2314 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:26.134909  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.137549  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.139570  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2731) took 2275 ms. Trace:
I20260709 13:03:26.140793  6968 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.124459  7010 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 13:03:26.148273  6852 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.149714  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2740) took 2034 ms. Trace:
I20260709 13:03:26.140214  1665 rpcz_store.cc:276] 0709 13:03:23.864156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:23.864400 (+   244us) service_pool.cc:224] Handling call
0709 13:03:25.353170 (+1488770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.367366 (+ 14196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.139279 (+771913us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001a86d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:26.139481 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3354363136}
I20260709 13:03:26.153635  2309 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:26.155473  2314 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:26.150234  1666 rpcz_store.cc:276] 0709 13:03:24.115247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.115388 (+   141us) service_pool.cc:224] Handling call
0709 13:03:24.134312 (+ 18924us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:24.209609 (+ 75297us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:25.340344 (+1130735us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b1800445d80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:03:25.421134 (+ 80790us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.088347 (+667213us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:26.110139 (+ 21792us) log.cc:844] Serialized 26 byte log entry
0709 13:03:26.110365 (+   226us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:26.110415 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.111718 (+  1303us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.149223 (+ 37505us) raft_consensus.cc:1685] finished
0709 13:03:26.149362 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.149625 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34722,"prepare.queue_time_us":865802,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":321,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":57810,"spinlock_wait_cycles":4501248,"thread_start_us":676379,"threads_started":2,"wal-append.queue_time_us":634125}
I20260709 13:03:26.173003  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:26.173779  1660 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:26.117000  6953 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:26.182125  1662 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:26.094341  6929 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.148834  7010 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:26.193698  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:26.207513  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 442599us
I20260709 13:03:26.209009  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:26.130821  7020 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 13:03:26.126405  6806 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.058s
W20260709 13:03:26.222635  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.224309  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.228065  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:26.209632  6973 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.132s	user 0.011s	sys 0.012s
I20260709 13:03:26.236186  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.237048  1734 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:26.237644  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.238936  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.239780  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.240211  1733 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:26.240919  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.241288  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:26.238346  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:26.217343  6541 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 13:03:26.206468  7010 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:26.254693  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 155999us
I20260709 13:03:26.256968  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:26.258646  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.256212  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.262678  1733 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:26.215955  7000 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:26.262913  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53970us
I20260709 13:03:26.264151  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.257416  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.185588  6953 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:26.243453  6973 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.169s	user 0.012s	sys 0.012s
I20260709 13:03:26.269793  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:26.243631  7018 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 13:03:26.268226  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.275094  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.276141  1734 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:26.294618  1742 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:26.281594  1708 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.232s (SENT)
I20260709 13:03:26.290804  1660 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:26.235764  6904 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.292484  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.280061  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.285265  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.307184  6955 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:26.309170  2314 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 13:03:26.320973  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91008us
I20260709 13:03:26.321748  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.313686  1733 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:26.178107  6968 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.098s
I20260709 13:03:26.315517  1734 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:26.266876  6536 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:26.283951  7006 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:26.324676  1737 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:26.330193  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:26.316411  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.331909  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.353466  1734 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:26.332435  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:26.333375  2316 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:26.334715  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53240.6us
W20260709 13:03:26.354205  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.317278  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:26.340157  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.354314  1740 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:26.293776  6745 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:26.342031  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:26.293550  7000 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:26.355805  2313 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:26.319020  6955 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:26.328509  6905 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:26.331867  6967 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:26.366264  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.357368  6541 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:26.377233  1737 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:26.322597  6904 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.380137   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:26.381400  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.383885  1737 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:26.387693  1740 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 13:03:26.392024  6971 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.487s	user 0.008s	sys 0.000s
I20260709 13:03:26.368512  6967 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:26.368652  6905 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:26.421463  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.421545  1740 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:26.423475  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:26.377893  6876 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.428586  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.428769  1741 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:26.425395  6905 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.365679  6745 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.434020  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.399869  6973 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:26.393872  6541 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:26.386870  6665 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:03:26.446990  7025 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:26.432343  6899 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:26.438306  6934 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:26.439587  6811 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:26.456880  6929 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:26.479681  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 162505us
I20260709 13:03:26.482124  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.482013  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:26.486296  1591 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:26.479755  6937 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.570s	user 0.014s	sys 0.000s
I20260709 13:03:26.455813  6536 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
W20260709 13:03:26.498381  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 176696us
I20260709 13:03:26.479619  7038 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:26.527668  7038 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.529567  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2759) took 1102 ms. Trace:
I20260709 13:03:26.529824  7043 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:26.530532  7043 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.531576  6968 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:26.532124  6968 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:26.533094  6904 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:26.488127  7039 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:26.534075  6892 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:26.534593  6892 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:26.535218  7044 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.535745  7044 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.535598  6892 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.500983  6541 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:26.501914  7041 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 13:03:26.502735  7033 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 13:03:26.501047  7035 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:03:26.514134  7037 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 13:03:26.455632  6955 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.461078  7010 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.459545  7000 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.517949  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 785) took 2057 ms. Trace:
W20260709 13:03:26.518771  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76140.8us
I20260709 13:03:26.519327  6908 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:26.534098  7039 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.522456  6975 raft_consensus.cc:993] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:26.523015  6745 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:26.524968  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.537356  6892 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.472388  6967 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.525640  6903 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.562582  6908 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:26.558506  7044 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.563499  7006 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 13:03:26.498549  6937 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.565712  7050 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:26.565791  6828 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 13:03:26.588953  7050 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:26.590588  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.530076  1661 rpcz_store.cc:276] 0709 13:03:25.427491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.545366 (+117875us) service_pool.cc:224] Handling call
0709 13:03:25.545938 (+   572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.629153 (+ 83215us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.529238 (+900085us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:26.529473 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140369792}
I20260709 13:03:26.499759  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.539767  1591 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:26.547683  7048 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.597638  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:26.598414  1590 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:26.598585  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98958.7us
I20260709 13:03:26.550330  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:26.552122  6995 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.987s	user 0.004s	sys 0.000s
W20260709 13:03:26.555152  2310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 819) took 1706 ms (client timeout 1580 ms). Trace:
I20260709 13:03:26.561753  1585 rpcz_store.cc:276] 0709 13:03:24.461058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.462120 (+  1062us) service_pool.cc:224] Handling call
0709 13:03:24.462524 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:24.462892 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:24.462922 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:24.463496 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 13:03:24.466192 (+  2696us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:24.466242 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:24.466418 (+   176us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.095300 (+1628882us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:26.499690 (+404390us) raft_consensus.cc:1685] finished
0709 13:03:26.499720 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.517860 (+ 18140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1312,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":182,"spinlock_wait_cycles":59067136,"thread_start_us":2071,"threads_started":1,"wal-append.queue_time_us":172433}
I20260709 13:03:26.524978  6960 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.575346  1584 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:26.601720  2310 rpcz_store.cc:269] 0709 13:03:24.846732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.846862 (+   130us) service_pool.cc:224] Handling call
0709 13:03:26.553383 (+1706521us) spinlock_profiling.cc:243] Waited 618 ms on lock 0x7b5400572e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:26.553507 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1421416832}
I20260709 13:03:26.568742  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2835) took 1375 ms. Trace:
I20260709 13:03:26.604296  1593 rpcz_store.cc:276] 0709 13:03:25.193204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.475986 (+282782us) service_pool.cc:224] Handling call
0709 13:03:25.476390 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:25.625936 (+149546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:25.625981 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:25.629938 (+  3957us) log.cc:844] Serialized 26 byte log entry
0709 13:03:25.727914 (+ 97976us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:25.727965 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.568250 (+840285us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400442d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:26.568312 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:26.568367 (+    55us) raft_consensus.cc:1685] finished
0709 13:03:26.568383 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:26.568668 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4933,"raft.queue_time_us":611,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":8272,"spinlock_wait_cycles":299953920,"thread_start_us":62342,"threads_started":1,"wal-append.queue_time_us":159887}
I20260709 13:03:26.576572  2312 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:26.595075  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.550014  6979 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.607384  6745 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:26.608158  6745 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:26.609447  6745 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.599462  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.590085  6937 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.556442  7025 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:26.621467  6985 raft_consensus.cc:1081] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:26.621903  7025 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:26.622609  7025 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:26.581873  7038 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.533267  6968 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:26.617915  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67715.9us
I20260709 13:03:26.527310  6905 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.567384  6842 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:26.587493  6967 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:26.604272  6536 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:26.570518  7035 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:26.615656  7048 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.588240  7006 consensus_queue.cc:1243] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:26.620656  7043 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:26.625914  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53723.2us
I20260709 13:03:26.626713  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.638265  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.658443  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.583559  6908 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:26.644511  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:26.583256  7039 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.651633  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:26.652657  1592 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:26.659487  1736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
W20260709 13:03:26.667898  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 185893us
I20260709 13:03:26.660259  1590 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:26.668322  1736 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:26.668756  1736 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.670794  1584 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:26.667362  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44458.6us
I20260709 13:03:26.674372  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.675209  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.628849  6665 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:26.607611  6979 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.628979  7050 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.629630  6905 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.667224  6811 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:26.604120  6904 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
I20260709 13:03:26.628670  6927 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:26.681272  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.624533  7052 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:26.691138  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2070) took 1482 ms. Trace:
I20260709 13:03:26.707000  1659 rpcz_store.cc:276] 0709 13:03:25.208886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.276470 (+ 67584us) service_pool.cc:224] Handling call
0709 13:03:25.943911 (+667441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.944743 (+   832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:26.690857 (+746114us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b540013ced0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:26.691055 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1494274944}
I20260709 13:03:26.643944  6806 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:26.666446  7038 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:26.701737  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.686457  7039 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:26.686457  6937 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:26.678308  6536 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:26.682237  6754 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:26.679915  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.672330  6940 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 13:03:26.719549  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.719784  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:26.723345  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:26.698381  6811 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.724526  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.726794  6947 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:26.709725  6904 consensus_queue.cc:1243] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:03:26.685871  7043 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 13:03:26.704352  6927 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:26.749897  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:26.692695  6883 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 13:03:26.734874  1736 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:26.773195  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48766.1us
I20260709 13:03:26.708940  6935 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:26.775377  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:26.766892  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.737985  7050 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:26.770601  7044 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:26.716444  6905 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.776739  2309 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:03:26.769896  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46632.6us
I20260709 13:03:26.814982  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:26.817164  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.818044  1667 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:26.822835  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.780943  1736 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.783788  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:26.784106  7053 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:26.784389  7049 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:26.785656  6536 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.751986  6906 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 13:03:26.743479  6934 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:26.799199  6979 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:26.748392  6811 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.800602  7050 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.744879  7039 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:26.824522  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:26.881099  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 137712us
I20260709 13:03:26.881553  1667 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:26.768663  6940 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:26.819440  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:26.827072  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.828827  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:26.744197  6937 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:26.842965  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:26.882045  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:26.888980  2316 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:26.873643  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.816550  7044 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:26.843420  6988 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 13:03:26.835506  7054 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 13:03:26.842721  6934 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:26.877965  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:26.891513  2316 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:26.820502  6929 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:26.889225  1668 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.903782  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:26.889988  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61250.9us
I20260709 13:03:26.777520  6956 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.128s
I20260709 13:03:26.909884  1659 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
W20260709 13:03:26.909785  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94912us
I20260709 13:03:26.985920  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:26.882023  7050 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.911727  2316 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.871775  6754 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:26.743860  7035 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.003110  7053 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:27.002765  6754 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.004132  7053 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:26.836910  6935 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:03:26.820792  6947 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:27.006446  6947 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.008740  6947 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:26.837447  6828 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 13:03:26.899106  6979 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:26.844447  7052 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.141s
W20260709 13:03:26.987185  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:26.826422  6905 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:26.992764  2316 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:26.994365  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:26.995450  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:26.916774  6934 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.011040  1660 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:27.011101  1667 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
W20260709 13:03:26.982820  7057 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:03:26.843824  7043 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 13:03:27.023125  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:27.123557  1592 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:27.029191  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.080119  1663 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:27.060133  7048 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:27.023159  7054 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 13:03:26.845222  7049 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.062795  1587 raft_consensus.cc:2368] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
W20260709 13:03:27.083675  7062 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 13:03:27.028533  7061 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.171s	user 0.006s	sys 0.004s
I20260709 13:03:27.087888  6937 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.091869  6899 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 13:03:27.109269  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.055097  7044 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:27.153477  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.862s (SENT) [suppressed 23 similar messages]
I20260709 13:03:27.164559  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:27.173892  7045 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.171s	user 0.000s	sys 0.003s
I20260709 13:03:27.121642  6745 raft_consensus.cc:2764] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 13:03:27.168089  7060 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.368s	user 0.000s	sys 0.002s
I20260709 13:03:27.201719  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:27.180419  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:27.174142  6971 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.163s	user 0.001s	sys 0.002s
I20260709 13:03:27.187325  7070 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:27.174301  6899 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:27.168589  7048 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.209084  7044 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:27.221000  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 57444.9us
I20260709 13:03:27.231143  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.233738  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.235098  7039 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:03:27.179841  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.220727  7070 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:27.230163  6899 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:27.236287  7000 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:27.231966  6979 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:27.304086  7039 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:03:27.315512  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 329738us
I20260709 13:03:27.317344  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:27.167557  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.306741  1586 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:27.231433  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:27.332404  6960 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.007s	sys 0.005s
I20260709 13:03:27.332494  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:27.333202  6960 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.007s	sys 0.005s
I20260709 13:03:27.334170  6960 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:27.334720  6960 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:27.335021  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 431378us
I20260709 13:03:27.336601  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:27.323668  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 92381.7us
I20260709 13:03:27.341162  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:27.342398  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:27.345444  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.346696  1667 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:27.348637  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.003s	sys 0.012s
W20260709 13:03:27.349107  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.003s	sys 0.012s
I20260709 13:03:27.337760  7078 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:27.335258  6960 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:27.236050  7048 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.315410  6984 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.363385  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.308542  7000 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:27.330444  7076 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:27.311460  7070 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:27.315770  7039 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:27.395663  7073 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:27.398391  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.400575  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:27.356119  6956 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:27.355337  7043 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.409264  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.350541  6828 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.362095  7053 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.409978  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.410269  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:27.388099  7075 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:27.355918  7006 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
W20260709 13:03:27.389787  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 161103us
I20260709 13:03:27.437650  6956 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:27.439270  6842 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.797s	user 0.016s	sys 0.000s
I20260709 13:03:27.439218  6956 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.440199  7039 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.441656  7039 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:27.445042  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.445863  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 10.
I20260709 13:03:27.400228  7076 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:27.354331  6883 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:27.445153  6541 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.449055  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:27.370793  2316 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:27.449900  1590 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
W20260709 13:03:27.409233  7069 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.386s	user 0.004s	sys 0.000s
I20260709 13:03:27.451879  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:27.467602  1586 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:27.447145  6903 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.865s	user 0.008s	sys 0.010s
I20260709 13:03:27.473539  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:27.451190  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 479997us
I20260709 13:03:27.466604  1590 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:27.412982  7068 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:27.469527  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 860) took 1948 ms (client timeout 1725 ms). Trace:
I20260709 13:03:27.479532  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:27.490304  1737 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 13:03:27.490921  7048 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.507972  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 861) took 1986 ms. Trace:
I20260709 13:03:27.402971  6984 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:27.524230  1735 rpcz_store.cc:269] 0709 13:03:25.521209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.521418 (+   209us) service_pool.cc:224] Handling call
0709 13:03:27.469263 (+1947845us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400434c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:27.469427 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287798656}
I20260709 13:03:27.493219  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2844) took 1302 ms. Trace:
W20260709 13:03:27.525568  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48097.7us
I20260709 13:03:27.529783  2316 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 13:03:27.544816  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 149885us
I20260709 13:03:27.433856  7043 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.492534  6884 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:27.492293  7068 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:27.459018  7076 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.459647  6828 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.570477  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.581310  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:27.589994  1735 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:27.522429  1739 rpcz_store.cc:276] 0709 13:03:25.521556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:25.522527 (+   971us) service_pool.cc:224] Handling call
0709 13:03:25.550997 (+ 28470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:25.630776 (+ 79779us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:25.649737 (+ 18961us) log.cc:844] Serialized 14 byte log entry
0709 13:03:26.227914 (+578177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:26.228005 (+    91us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:26.228073 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.507524 (+1279451us) spinlock_profiling.cc:243] Waited 79.4 ms on lock 0x7b5400352ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:27.507612 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.507870 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":269706,"prepare.queue_time_us":19427,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":3425,"raft.queue_time_us":278657,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":13413,"spinlock_wait_cycles":182727552,"thread_start_us":325709,"threads_started":1,"wal-append.queue_time_us":467428,"wal-append.run_cpu_time_us":1135,"wal-append.run_wall_time_us":1026225}
W20260709 13:03:27.587218  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.465732  6541 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.597455  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.598510  1663 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:03:27.554373  7025 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.931s	user 0.008s	sys 0.012s
I20260709 13:03:27.588086  7071 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.607611  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.608318  1660 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:27.614883  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.452080  7053 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.618053  1660 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:27.622431  7077 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.224s
I20260709 13:03:27.618789  7071 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.627578  7071 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:27.629591  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:27.630190  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:27.631064  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:27.631811  1738 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:27.632552  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:27.640120  7053 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.641882  7076 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.643644  1735 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:27.644416  7076 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.633890  6935 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.532173  7048 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:27.651262  1586 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:27.667735  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:27.668093  7087 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.201s	user 0.001s	sys 0.000s
I20260709 13:03:27.673768  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 878) took 1015 ms. Trace:
I20260709 13:03:27.674345  1740 rpcz_store.cc:276] 0709 13:03:26.658252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.659269 (+  1017us) service_pool.cc:224] Handling call
0709 13:03:26.678255 (+ 18986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.671902 (+993647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:27.671955 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:27.672018 (+    63us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:27.672064 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.673401 (+  1337us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b540012d9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:27.673459 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.673680 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2284687744}
W20260709 13:03:27.675985  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.685537  7047 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:27.686195  7047 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:27.652403  1592 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.528s	user 0.007s	sys 0.011s
I20260709 13:03:27.690357  1592 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:27.691437  7020 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:27.702608  7047 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.463811  7000 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.662173  6828 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:27.659763  7079 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 13:03:27.713737  6960 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.465766  6979 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.617247  7073 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.105s
W20260709 13:03:27.651577  6892 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.058s	user 0.007s	sys 0.011s
W20260709 13:03:27.651739  6967 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.988s	user 0.007s	sys 0.012s
I20260709 13:03:27.575963  7068 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.717427  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:27.736588  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.726266  6953 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.798933  1661 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 13:03:27.724162  6984 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.720598  6935 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.722103  7043 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.806026  7043 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:27.806602  7043 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.807170  6960 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.729274  6908 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:27.809463  6908 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:27.811635  6934 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.757s	user 0.009s	sys 0.009s
I20260709 13:03:27.725626  6541 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:27.718941  7028 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:27.813764  6968 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:27.816275  7066 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.525s	user 0.002s	sys 0.000s
I20260709 13:03:27.813758  7028 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:27.808800  7043 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:27.758852  7010 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:27.724351  7049 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:27.844758  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 127453us
W20260709 13:03:27.847267  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 373870us
I20260709 13:03:27.800314  7047 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:27.856593  7083 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:27.618521  7025 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.819419  6876 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.825150  7028 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:27.820212  6828 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:27.824307  6967 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.852918  6953 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:27.878820  1660 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:27.829244  1668 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.917s	user 0.007s	sys 0.013s
I20260709 13:03:27.883922  1668 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:03:27.884701  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2091) took 1069 ms. Trace:
I20260709 13:03:27.884814  1659 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:27.874888  7050 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.850s	user 0.015s	sys 0.005s
W20260709 13:03:27.886279  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 437431us
I20260709 13:03:27.863940  7083 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:27.893822  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44924.3us
I20260709 13:03:27.823722  6934 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:27.892902  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:27.908771  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.897970  1668 rpcz_store.cc:276] 0709 13:03:26.815703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.819183 (+  3480us) service_pool.cc:224] Handling call
0709 13:03:27.884596 (+1065413us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:27.855274  7043 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 13:03:27.912674  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.882373  6935 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:27.892982  6786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.295s	user 0.006s	sys 0.012s
W20260709 13:03:27.879805  6754 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.000s	sys 0.012s
I20260709 13:03:27.888052  6953 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:27.873528  6967 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:27.932252  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:27.874025  6876 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:27.892351  7064 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.006s	sys 0.006s
I20260709 13:03:27.900353  6929 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:27.878178  6828 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:27.959581  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.965318  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:27.939195  6786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.343s	user 0.006s	sys 0.013s
W20260709 13:03:27.973181  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.010s	sys 0.018s
W20260709 13:03:27.942169  6754 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.001s	sys 0.012s
W20260709 13:03:27.977008  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.530s	user 0.011s	sys 0.018s
I20260709 13:03:27.983537  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:27.972651  6937 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.021s	sys 0.080s
I20260709 13:03:27.906777  7050 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:27.955865  7064 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.006s	sys 0.007s
W20260709 13:03:27.998085  1708 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.000s	sys 0.037s
I20260709 13:03:28.000752  1668 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 10. Current term is 11. Ops: []
W20260709 13:03:28.008818  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:28.011822  1708 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.000s	sys 0.037s
W20260709 13:03:28.012615  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 859201us
I20260709 13:03:28.016103  6754 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.017115  6754 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.023602  7025 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.024413  6967 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:27.948017  7094 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
W20260709 13:03:27.997143  6937 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.022s	sys 0.080s
I20260709 13:03:28.030064  7095 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:27.947340  7083 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:28.013128  6899 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.003s
W20260709 13:03:28.034469  7092 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 13:03:28.048185  1663 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:03:28.043377  7072 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:28.045095  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:28.045985  1558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.023s	sys 0.026s
I20260709 13:03:28.041322  6953 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.5, Last appended by leader: 5, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:28.052029  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 150310us
W20260709 13:03:28.053786  1558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.024s	sys 0.026s
I20260709 13:03:28.058310  1668 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:28.025082  6967 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 13:03:28.066819  1593 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:28.066839  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.065526  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:28.069711  1585 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:03:28.071156  1590 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.7-13.7]
W20260709 13:03:28.070442  7044 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.834s	user 0.007s	sys 0.007s
I20260709 13:03:28.074074  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.074790  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.082185  1587 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:28.083680  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:28.084406  1589 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:28.086882  6828 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.049378  6541 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.094841  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.095757  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:28.102725  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.106472  1584 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 13:03:28.128201  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58903.7us
I20260709 13:03:28.088104  6935 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.142371  6935 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:28.147293  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.132470  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:28.088304  6883 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:28.053277  6899 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.090384  6536 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:28.099970  6485 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:28.158692  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 85531us
I20260709 13:03:28.159653  1591 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:28.167912  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:28.160189  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:28.099622  6956 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:28.127085  6811 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.234s	user 0.021s	sys 0.004s
I20260709 13:03:28.052904  7095 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.112920  7089 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:28.118664  6754 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.184124  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.182432  6956 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:28.191710  7070 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.782s	user 0.011s	sys 0.002s
I20260709 13:03:28.060222  7072 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.182650  6811 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.199537  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:28.199584  7099 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.109s	user 0.001s	sys 0.001s
I20260709 13:03:28.167887  6934 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.151600  6968 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 13:03:28.177961  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:28.180132  7111 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:28.212455  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44606.4us
I20260709 13:03:28.243019  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:28.224326  6536 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:28.113960  6541 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.232779  1664 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:28.234480  1592 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:28.176498  7094 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:28.221580  6963 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:28.251427  1736 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:28.252408  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:28.263182  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:28.265426  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:28.224992  1589 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
W20260709 13:03:28.239804  6927 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.446s	user 0.015s	sys 0.010s
I20260709 13:03:28.175604  6485 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.271050  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:28.274331  1589 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:28.283635  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 871) took 2056 ms. Trace:
I20260709 13:03:28.284497  1742 rpcz_store.cc:276] 0709 13:03:26.226815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.295674 (+ 68859us) service_pool.cc:224] Handling call
0709 13:03:26.295942 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:26.359948 (+ 64006us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.359979 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:26.360019 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:26.360060 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.283414 (+1923354us) spinlock_profiling.cc:243] Waited 63.8 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:28.283456 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:28.283581 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146665728}
I20260709 13:03:28.228495  7070 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.276402  6536 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:28.287616  6665 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:28.307404  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2921) took 1307 ms. Trace:
I20260709 13:03:28.309556  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:27.527714  1588 rpcz_store.cc:276] 0709 13:03:26.191164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.191365 (+   201us) service_pool.cc:224] Handling call
0709 13:03:26.191906 (+   541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:26.192378 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:26.192423 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:26.193068 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 13:03:27.470364 (+1277296us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b18003243a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1
0709 13:03:27.470591 (+   227us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:27.470650 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:27.470755 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:27.470834 (+    79us) raft_consensus.cc:1685] finished
0709 13:03:27.470854 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:27.493112 (+ 22258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":363,"raft.queue_time_us":127735,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":3272,"spinlock_wait_cycles":5049216,"thread_start_us":125021,"threads_started":1,"wal-append.queue_time_us":222421,"wal-append.run_cpu_time_us":2083,"wal-append.run_wall_time_us":1068580}
I20260709 13:03:28.309911  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:28.322832  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2804) took 2033 ms. Trace:
I20260709 13:03:28.335574  1735 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:28.336135  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.335615  1737 rpcz_store.cc:276] 0709 13:03:26.290367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.392569 (+102202us) service_pool.cc:224] Handling call
0709 13:03:27.469425 (+1076856us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.520827 (+ 51402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.322586 (+801759us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400434c50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:28.322758 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2475486976}
I20260709 13:03:28.309464  7070 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.326607  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:28.340324  6905 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.316201  1586 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:28.321409  1589 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:28.259617  7083 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:28.359935  6955 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:28.268265  6754 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 13:03:28.299000  6956 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.269225  6934 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:28.302178  6485 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.304693  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:28.304949  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:28.304455  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:28.306316  7039 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.864s	user 0.015s	sys 0.000s
I20260709 13:03:28.386265  7050 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.388568  6541 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.290728  7089 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:28.307998  1591 rpcz_store.cc:276] 0709 13:03:27.000409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.175968 (+175559us) service_pool.cc:224] Handling call
0709 13:03:28.307205 (+1131237us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:28.307337 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2015248768}
I20260709 13:03:28.407444  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:28.273486  7094 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:28.415928  1734 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:28.418283  1739 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:28.426119  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.431502  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.435408  1586 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:03:28.442051  1586 pending_rounds.cc:85] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:03:28.442368  1586 pending_rounds.cc:107] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:03:28.442615  1586 raft_consensus.cc:2889] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:28.442055  2311 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:28.437042  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:28.437532  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.438107  1589 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:28.445883  7094 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.450328  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.451242  1666 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:28.452718  1666 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:28.427009  6934 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:28.453902  6934 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
W20260709 13:03:28.453264  7065 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.375s	user 0.005s	sys 0.000s
I20260709 13:03:28.455547  7042 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:28.456001  7042 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:28.270115  6963 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.459257  6963 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:28.463719  2312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 820) took 2221 ms (client timeout 1979 ms). Trace:
I20260709 13:03:28.400601  7071 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:28.464432  2312 rpcz_store.cc:269] 0709 13:03:26.241894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.242066 (+   172us) service_pool.cc:224] Handling call
0709 13:03:28.463396 (+2221330us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003ef154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:28.463580 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":543669632}
I20260709 13:03:28.273070  6927 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.475629  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:28.413214  7111 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:28.479941  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 249840us
W20260709 13:03:28.481388  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44434.6us
W20260709 13:03:28.482396  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:28.484055  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.486743  1665 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:28.484650  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53336.7us
I20260709 13:03:28.492640  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.475097  6899 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.503271  1666 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:28.470407  7039 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.505147  1667 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:28.489060  6996 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:28.487560  6828 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:28.515581  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.527933  1665 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:28.413763  7074 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.096s
I20260709 13:03:28.530805  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.532924  1589 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:28.504802  6485 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.534705  6485 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.540791  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.558032  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.559290  1666 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:28.562328  7072 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.566310  6996 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:28.575709  7115 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:03:28.577678  6485 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.585554  6996 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.586055  7043 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:28.477573  7071 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.595007  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.488085  7050 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 13:03:28.517560  6994 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:28.520452  7039 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.620004  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.623222  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:28.624389  1592 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:28.537731  7074 consensus_queue.cc:1243] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:28.538812  6828 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:28.646687  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.648386  1665 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:28.609251  6908 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.799s	user 0.009s	sys 0.009s
I20260709 13:03:28.618476  7050 consensus_queue.cc:1243] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:03:28.754006  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 473392us
I20260709 13:03:28.762609  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.508987  7111 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.661741  6828 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:28.778617  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.784004  1665 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:28.797098  1733 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:03:28.797690  1733 pending_rounds.cc:85] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 10
I20260709 13:03:28.797973  1733 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 13:03:28.798461  1733 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:28.781960  7010 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.937s	user 0.016s	sys 0.003s
I20260709 13:03:28.765681  7110 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.636240  6994 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.812474  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:28.767490  7072 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.820139  1588 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
W20260709 13:03:28.822074  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:28.786171  7044 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.829396  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.807791  7095 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.837968  1665 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
W20260709 13:03:28.836884  6979 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.022s	sys 0.038s
I20260709 13:03:28.821072  7103 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.832835  6956 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:28.825327  7075 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:28.862805  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:28.809769  6541 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.041s
W20260709 13:03:28.863489  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 12 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:28.864873  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:28.876010  2311 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:28.828318  7010 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.796922  6883 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
W20260709 13:03:28.859023  6979 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.986s	user 0.023s	sys 0.039s
I20260709 13:03:28.823122  7072 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.862833  6956 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:28.836685  7044 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:28.866015  6811 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.865834  6927 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.882917  6541 consensus_queue.cc:1243] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:28.907610  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:28.914675  1665 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:28.918900  2311 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:28.823544  7110 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.899516  6956 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:28.940213  6934 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:28.910161  7068 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.056s	user 0.015s	sys 0.005s
I20260709 13:03:28.946273  7072 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:28.949343  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:29.022634  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.006s	sys 0.045s
W20260709 13:03:29.022974  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.006s	sys 0.045s
I20260709 13:03:28.951238  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.023900  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:28.855610  7095 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:28.962350  6955 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.549s	user 0.015s	sys 0.000s
W20260709 13:03:28.970410  7120 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 13:03:28.973064  7039 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.979499  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2106) took 1404 ms. Trace:
I20260709 13:03:28.979986  7071 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:28.982414  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.054718  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:28.983013  6994 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:28.988569  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:28.990042  1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2930) took 1605 ms (client timeout 1879 ms). Trace:
W20260709 13:03:29.081594  1587 rpcz_store.cc:269] 0709 13:03:27.384653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.394455 (+  9802us) service_pool.cc:224] Handling call
0709 13:03:28.989828 (+1595373us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54001def54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:28.989941 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1491281152}
I20260709 13:03:29.082201  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:28.992528  6927 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:28.994304  7117 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.460s	user 0.001s	sys 0.005s
I20260709 13:03:28.994185  7050 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:28.969269  6884 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:28.995833  6968 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
W20260709 13:03:28.998993  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:29.095880  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:29.097018  2313 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.022s	sys 0.039s
W20260709 13:03:29.097407  2313 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.022s	sys 0.039s
I20260709 13:03:29.006450  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.098814  1584 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:29.099426  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2856) took 1006 ms. Trace:
I20260709 13:03:29.099880  1584 rpcz_store.cc:276] 0709 13:03:28.092745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.264711 (+171966us) service_pool.cc:224] Handling call
0709 13:03:29.006200 (+741489us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b5800000098. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:03:29.099335 (+ 93135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6304640}
I20260709 13:03:29.008611  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 880) took 2301 ms. Trace:
I20260709 13:03:29.102988  1733 rpcz_store.cc:276] 0709 13:03:26.707044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.707227 (+   183us) service_pool.cc:224] Handling call
0709 13:03:26.707610 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.799944 (+2092334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.008213 (+208269us) spinlock_profiling.cc:243] Waited 2.09 s on lock 0x7b540012ded4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:29.008516 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4804680576}
I20260709 13:03:28.882380  7075 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.874873  7103 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:28.884505  7010 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:29.026525  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.024s	sys 0.028s
I20260709 13:03:29.026649  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:29.112953  1663 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:29.112044  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:29.029978  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:27.877363   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:29.052728  1585 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 13:03:29.051782  1660 rpcz_store.cc:276] 0709 13:03:27.574967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.621467 (+ 46500us) service_pool.cc:224] Handling call
0709 13:03:27.624412 (+  2945us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.879619 (+255207us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.979067 (+1099448us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54003c8f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:28.979399 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584605952}
W20260709 13:03:29.111991  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.024s	sys 0.029s
W20260709 13:03:29.116266  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 633926us
I20260709 13:03:29.061256  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.083204  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.081657  1589 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:29.022325  6974 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:29.044389  6736 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.006s	sys 0.049s
I20260709 13:03:29.114502  1585 pending_rounds.cc:85] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 9
W20260709 13:03:29.049273  7028 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.175s	user 0.005s	sys 0.017s
I20260709 13:03:29.137813  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.137673  7028 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.138516  1660 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:29.139565  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.140470  7028 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.140828  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.118193  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:29.145709  7020 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:29.145792  7052 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:29.146304  7020 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.146715  7020 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:29.148324  7052 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:29.148732  7052 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:29.157023  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.119310  2318 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:29.132643  1585 pending_rounds.cc:107] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:03:29.135658  1584 pending_rounds.cc:85] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 7
I20260709 13:03:29.136528  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.025039  7068 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.089440  7071 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.181638  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.198446  1584 pending_rounds.cc:107] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:03:29.164105  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.198907  1584 raft_consensus.cc:2889] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:29.198997  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 10.
I20260709 13:03:29.194581  2314 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:29.182135  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.200845  2309 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:29.181738  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.202832  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.201869  1733 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:29.204556  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.204928  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.207528  1585 raft_consensus.cc:2889] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:29.210043  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.210626  6940 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:29.211202  6940 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.211539  6935 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.212822  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:29.213188  7111 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.213716  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.214293  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2769) took 1803 ms. Trace:
I20260709 13:03:29.215021  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:29.214956  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.214708  1667 rpcz_store.cc:276] 0709 13:03:27.410723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.410921 (+   198us) service_pool.cc:224] Handling call
0709 13:03:27.412346 (+  1425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.511687 (+1099341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.214015 (+702328us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540013ced4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:29.214216 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2512939776}
I20260709 13:03:29.215857  1738 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:29.216099  1740 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:29.216522  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.218236  1667 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:29.218169  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:29.219022  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2779) took 1127 ms. Trace:
I20260709 13:03:29.219401  1667 rpcz_store.cc:276] 0709 13:03:28.091590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.217491 (+1125901us) service_pool.cc:224] Handling call
0709 13:03:29.217884 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:29.218776 (+   892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.218968 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:29.219987  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2094) took 2403 ms. Trace:
I20260709 13:03:29.219260  2311 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:29.220360  1661 rpcz_store.cc:276] 0709 13:03:26.816600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:26.818675 (+  2075us) service_pool.cc:224] Handling call
0709 13:03:26.818939 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.801204 (+982265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.219745 (+1418541us) spinlock_profiling.cc:243] Waited 979 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:29.219930 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2252758912}
I20260709 13:03:29.222187  1661 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:29.222865  1585 raft_consensus.cc:2368] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:29.224663  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.226075  6967 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.226717  6967 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.226850  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.159832  6665 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.228609  6940 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.228670  6665 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.231412  7075 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.145411  6811 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.232582  6811 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.234434  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.235396  1591 raft_consensus.cc:2368] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:29.236342  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:29.236696  6811 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.050940  7095 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.097788  6884 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:29.254325  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:29.132540  6736 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.008s	sys 0.052s
I20260709 13:03:29.127342  6883 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.244403  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.240149  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2775) took 1373 ms. Trace:
I20260709 13:03:29.245555  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:29.208130  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:29.249140  1660 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:29.252710  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.157430  7072 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:29.237480  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.255370  1734 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:29.096024  6968 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:29.258325  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:29.268289  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.267336  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:29.318176  1590 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:29.318746  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2777) took 1425 ms. Trace:
I20260709 13:03:29.244280  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.097523  6927 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.299243  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.301156  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:29.302310  1661 raft_consensus.cc:2410] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:29.330168  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.112672  7050 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.285307  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.299124  6935 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:29.262912  6665 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.242648  7091 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:29.307516  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:29.284730  1664 rpcz_store.cc:276] 0709 13:03:27.866780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.257556 (+390776us) service_pool.cc:224] Handling call
0709 13:03:28.278902 (+ 21346us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:28.279382 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:28.279421 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:28.280112 (+   691us) log.cc:844] Serialized 52 byte log entry
0709 13:03:28.819222 (+539110us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:28.847909 (+ 28687us) log.cc:844] Serialized 14 byte log entry
0709 13:03:28.848635 (+   726us) log.cc:844] Serialized 14 byte log entry
0709 13:03:28.850233 (+  1598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.850360 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:29.239933 (+389573us) raft_consensus.cc:1685] finished
0709 13:03:29.239953 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:29.240086 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1417,"prepare.queue_time_us":19395,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":75,"raft.queue_time_us":1773,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":6351,"thread_start_us":493424,"threads_started":1,"wal-append.queue_time_us":760402}
I20260709 13:03:29.281935  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:29.323961  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 477146us
I20260709 13:03:29.241904  7010 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:29.329957  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84525.9us
I20260709 13:03:29.344869  1660 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:03:29.263944  6934 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.201292  7052 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.353865  1591 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 13:03:29.164898  6974 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.247325  7071 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.358454  7075 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:29.309902  6883 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.360240  7020 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.361624  7119 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.362414  7020 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.330773  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.240422  7068 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.258512  7111 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:29.365283  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2857) took 1272 ms. Trace:
I20260709 13:03:29.334123  1665 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:29.335791  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.365801  1593 rpcz_store.cc:276] 0709 13:03:28.093248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.312549 (+219301us) service_pool.cc:224] Handling call
0709 13:03:28.313029 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:28.405486 (+ 92457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:28.405580 (+    94us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:28.411481 (+  5901us) log.cc:844] Serialized 26 byte log entry
0709 13:03:28.826552 (+415071us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:28.826596 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:28.826683 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:29.365026 (+538343us) raft_consensus.cc:1685] finished
0709 13:03:29.365050 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:29.365199 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3002,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":117,"thread_start_us":342344,"threads_started":1,"wal-append.queue_time_us":880641}
I20260709 13:03:29.338035  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:29.339443  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:29.340624  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:29.368266  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:29.368863  1740 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:29.341600  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.371368  1663 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:29.341908  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:29.342310  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:29.307360  7134 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 13:03:29.344941  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:29.362157  7119 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.299896  6152 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 13:03:29.383345  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:29.364329  1589 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 5. Preceding OpId from leader: term: 14 index: 6. (index mismatch)
I20260709 13:03:29.254905  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:29.378293  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:29.388707  1664 raft_consensus.cc:2410] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 13:03:29.372905  1733 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:29.378683  1663 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:29.379841  1708 leader_election.cc:400] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:29.389322  1708 leader_election.cc:403] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:03:29.390261  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:29.343801  7011 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:29.391642  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:29.343420  7091 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:29.406077  1708 leader_election.cc:400] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:29.389840  6974 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.410585  1663 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:03:29.415620  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.416762  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.409873  6541 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:29.418126  7064 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:29.418681  6376 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:29.371364  7070 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.372466  6904 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.424801  1666 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:29.329401  7072 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:29.455118  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 355051us
I20260709 13:03:29.385789  6996 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:29.475417  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:29.393930  7011 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:29.395654  6954 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:29.394228  7010 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.410399  7075 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:29.454659  6376 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:29.418150  6883 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.410531  6935 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20260709 13:03:29.457556  6906 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:03:29.519526  7042 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.520920  7042 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:29.463565  7089 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 13:03:29.524433  6903 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:29.524968  6903 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:29.526033  6903 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.527382  6903 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:29.530185  7072 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.494489  7010 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.533011  7010 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.485733  6842 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.479692  7011 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:29.482380  6954 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:29.486311  6665 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:29.544312  1740 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:29.488857  7064 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:29.496176  6376 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:29.534631  6828 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.721s	user 0.015s	sys 0.003s
I20260709 13:03:29.469641  7070 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:29.474961  6904 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.551627  6954 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:29.574111  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 822) took 1262 ms. Trace:
I20260709 13:03:29.553049  6934 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:29.575862  1586 rpcz_store.cc:276] 0709 13:03:28.324523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.345028 (+ 20505us) service_pool.cc:224] Handling call
0709 13:03:28.345364 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.443185 (+ 97821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.573108 (+1129923us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:29.573711 (+   603us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":205903104}
I20260709 13:03:29.554877  7071 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.558622  7064 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:29.571651  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 159985us
I20260709 13:03:29.554250  6927 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.553437  6842 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:29.578581  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 246939us
W20260709 13:03:29.577319  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 233688us
I20260709 13:03:29.606483  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.588020  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2115) took 1064 ms. Trace:
I20260709 13:03:29.565305  6828 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.632505  7064 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:29.706843  6956 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.680s	user 0.008s	sys 0.004s
I20260709 13:03:29.698865  7010 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.719723  1587 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:03:29.707885  7071 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:29.731437  7136 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 13:03:29.732357  7119 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:29.724053  7071 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.733558  7071 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:29.722101  6956 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.716024  7050 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.735504  1735 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:29.736394  7110 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:29.770354  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:29.778939  7143 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:03:29.779327  6842 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:29.779964  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 364484us
I20260709 13:03:29.781170  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.782406  7078 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:03:29.787513  7139 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:03:29.823937  7145 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:29.824807  7145 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.826822  7110 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:29.827555  7110 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.830642  6876 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:29.834543  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 438800us
I20260709 13:03:29.835981  7149 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:29.867827  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.686347  1665 rpcz_store.cc:276] 0709 13:03:28.523385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.366343 (+842958us) service_pool.cc:224] Handling call
0709 13:03:29.370671 (+  4328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:29.587433 (+216762us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:29.587484 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:29.587557 (+    73us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:29.587610 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.587729 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:29.587930 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6787,"raft.queue_time_us":251655,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4668,"thread_start_us":204706,"threads_started":1}
W20260709 13:03:29.910924  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 129847us
I20260709 13:03:29.912957  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:29.912469  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 306102us
I20260709 13:03:29.915771  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:29.917867  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:29.918576  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:29.937932  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:29.940749  1584 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 13:03:29.942210  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:29.945973  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1005) took 1413 ms. Trace:
I20260709 13:03:29.728018  7010 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:29.912426  6876 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.926956  7083 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:29.942682  7078 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:29.943203  6956 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:29.952538  2313 rpcz_store.cc:276] 0709 13:03:28.519643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.519923 (+   280us) service_pool.cc:224] Handling call
0709 13:03:28.563512 (+ 43589us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.563814 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:29.097865 (+534051us) log.cc:844] Serialized 14 byte log entry
0709 13:03:29.931204 (+833339us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:29.931276 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:29.931326 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.932343 (+  1017us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:29.932499 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60185,"prepare.queue_time_us":614779,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":30,"raft.queue_time_us":41156,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":13104,"thread_start_us":1337041,"threads_started":3,"wal-append.queue_time_us":491267}
I20260709 13:03:29.941720  7150 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:29.964350  6828 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:29.966872  7153 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:29.991312  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2772) took 2556 ms. Trace:
I20260709 13:03:29.942271  7152 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:03:29.946982  6927 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 13:03:30.005733  1659 rpcz_store.cc:276] 0709 13:03:27.434474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:27.434690 (+   216us) service_pool.cc:224] Handling call
0709 13:03:27.884547 (+449857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:27.898260 (+ 13713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.991036 (+2092776us) spinlock_profiling.cc:243] Waited 418 ms on lock 0x7b540013e7d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:29.991230 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":961262336}
I20260709 13:03:29.945787  6665 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 13:03:30.007645  2311 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:30.007438  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.012178  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:29.978698  7078 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:29.977569  7163 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.022405  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:30.031800  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.033399  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:30.017942  1742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.014s	sys 0.051s
W20260709 13:03:30.034608  1742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.014s	sys 0.051s
I20260709 13:03:30.034860  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:29.983680  7075 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:29.975397  6876 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:30.069599  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 823) took 1744 ms (client timeout 2122 ms). Trace:
W20260709 13:03:30.071257  1588 rpcz_store.cc:269] 0709 13:03:28.324833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.335507 (+ 10674us) service_pool.cc:224] Handling call
0709 13:03:30.069387 (+1733880us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b540025ce54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:30.069535 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1111462784}
I20260709 13:03:30.072695  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.073458  1736 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:30.075485  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.076120  1740 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 13:03:30.077033  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:30.078961  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44239.9us
I20260709 13:03:29.986622  7089 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 13:03:29.993897  7047 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.045s
W20260709 13:03:30.083256  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 137653us
W20260709 13:03:30.083212  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 169225us
I20260709 13:03:30.080132  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:30.019055  7134 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:30.008620  7070 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.026819  7145 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.014098  7083 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:03:30.030157  7078 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:30.080701  6828 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:30.109297  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:29.988198  7150 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.122426  1733 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:30.037631  7119 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 13:03:30.104507  7151 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:30.145097  6927 consensus_queue.cc:1243] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:30.104112  7137 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:30.110524  7158 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.108901  6979 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:30.113803  7039 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:30.160125  1736 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:30.162459  7160 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 13:03:30.066432  7094 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20260709 13:03:30.158716  6541 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.701s	user 0.006s	sys 0.013s
W20260709 13:03:30.170117  6884 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.871s	user 0.015s	sys 0.007s
I20260709 13:03:30.186951  7091 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.187773  7150 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.192730  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.225219  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:30.136345  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 193993us
I20260709 13:03:30.038908  7163 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.200320  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:30.228472  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.172094  6979 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:30.232601  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:30.196403  6996 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.717s	user 0.017s	sys 0.004s
I20260709 13:03:30.238970  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:30.182853  7110 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.245054  6736 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:30.172883  7158 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.212915  6541 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14.
I20260709 13:03:30.209824  7075 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.171552  7137 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.235230  6979 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:30.287756  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.288583  1738 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:30.285008  6876 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:30.291250  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52413.1us
I20260709 13:03:30.242756  6927 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:30.298472  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.267571  6996 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:30.303573  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.211102  6884 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.314147  6736 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:30.313925  7168 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.143s
I20260709 13:03:30.314961  7170 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:03:30.315442  7164 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:03:30.315546  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:30.316185  7162 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:03:30.318715  7158 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.322253  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.322671  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.280864  7091 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:30.330569  7000 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.350462  1584 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:30.331418  6904 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:30.332428  7095 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.335210  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 879) took 1887 ms. Trace:
I20260709 13:03:30.362332  1667 rpcz_store.cc:276] 0709 13:03:28.447668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.220541 (+772873us) service_pool.cc:224] Handling call
0709 13:03:29.303805 (+ 83264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:29.366566 (+ 62761us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:29.366601 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:29.366644 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:29.366675 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.334808 (+968133us) spinlock_profiling.cc:243] Waited 62.5 ms on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:30.334881 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.335132 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":143798656}
I20260709 13:03:30.340808  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.364339  1736 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.341691  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.242594  7151 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 13:03:30.296289  7149 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:30.347293  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.354329  1592 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:30.374032  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.375735  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.377424  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.378945  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.380153  1584 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:03:30.361235  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.361379  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.381031  1740 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:30.321426  7010 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:30.369316  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 228873us
I20260709 13:03:30.372215  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.390264  1662 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:30.381862  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.391522  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:30.391803  1663 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:30.392458  1736 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:30.394932  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.395066  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:30.342803  7170 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:30.398504  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:30.398782  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.400405  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.403429  1584 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:30.401918  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.404551  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.405705  1736 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:30.406987  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:30.407222  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.407900  1659 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:30.407979  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.357116  6736 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:30.408113  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.328557  6876 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:03:30.407021  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.412823  1661 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:30.418737  1665 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:30.419221  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:30.417376  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.419854  2311 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:30.414855  1736 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:03:30.415246  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:30.415390  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 193555us
I20260709 13:03:30.413866  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.422210  6828 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:30.423446  1742 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 13:03:30.423367  6828 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.398890  7155 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:30.432616  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.426414  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:30.428323  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.425127  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.431540  1734 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:30.418220  6903 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.887s	user 0.021s	sys 0.005s
I20260709 13:03:30.412416  7095 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.439419  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:30.427479  6376 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.867s	user 0.004s	sys 0.021s
I20260709 13:03:30.403009  7039 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:30.402066  7149 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:30.439164  1584 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:30.413314  6736 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:30.444114  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:30.442548  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.442984  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:30.439167  6954 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.847s	user 0.005s	sys 0.016s
W20260709 13:03:30.439544  7042 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.918s	user 0.009s	sys 0.017s
I20260709 13:03:30.444065  1739 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:30.451656  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.449328  1742 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:30.440201  7110 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:30.459911  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.430631  7175 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:30.439615  7150 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:30.425905  7010 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:30.466439  1584 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:30.455768  6736 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:30.473157  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.475075  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.441931  6985 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:30.445981  7167 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.481030  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.494297  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.495074  1584 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:30.496181  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.460726  7155 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:30.480079  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 119535us
I20260709 13:03:30.464819  7151 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.475401  7180 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 13:03:30.501957  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:30.503978  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.474327  7163 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.534200  7155 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.536996  7155 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.538173  6968 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.539496  6968 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:30.543081  6956 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:30.543670  6956 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:30.544596  6956 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.508433  7179 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:30.508939  7151 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.510413  7180 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:30.512576  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.569585  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:30.570241  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57755.4us
I20260709 13:03:30.571290  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.568408  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:30.513759  7174 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:30.503587  7178 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 13:03:30.514532  7149 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.515549  7177 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:03:30.446429  7095 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.587859  7095 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.516773  7137 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:30.519637  7103 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.483070  7150 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.590512  7150 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.486975  7010 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.592819  7010 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.476886  6903 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:30.593657  6903 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.584165  7094 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.010s
I20260709 13:03:30.482535  7039 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.489305  7044 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:30.513239  7011 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.957s	user 0.022s	sys 0.000s
I20260709 13:03:30.596778  6899 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:30.596961  7011 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.599346  7011 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:30.521145  7078 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.600508  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.580219  6935 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:30.601589  7078 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.579820  7145 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:30.527194  7071 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.793s	user 0.012s	sys 0.008s
I20260709 13:03:30.531253  7153 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:30.575687  7174 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:30.597379  6899 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.569303  7180 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:30.607312  7159 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.526s	user 0.004s	sys 0.000s
I20260709 13:03:30.448815  6376 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.607764  7180 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.608556  1737 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:30.483546  7175 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.468470  6954 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.548133  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.621176  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.622766  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.556221  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.505226  7167 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.558274  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.626154  1734 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:30.504709  1742 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:30.566679  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 886) took 2109 ms. Trace:
I20260709 13:03:30.627269  1660 rpcz_store.cc:276] 0709 13:03:28.456834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.364899 (+908065us) service_pool.cc:224] Handling call
0709 13:03:29.367393 (+  2494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:29.367694 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:29.367726 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:29.367771 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:29.367802 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.367878 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:30.566440 (+1198562us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b54003c9450. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:30.566590 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4825344}
I20260709 13:03:30.631819  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:30.634725  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.584889  7178 consensus_queue.cc:1243] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 13:03:30.570467  1667 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
W20260709 13:03:30.576720  1632 connection.cc:256] RPC call timeout handler was delayed by -1.4748043570000391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
W20260709 13:03:30.645875  1632 connection.cc:256] RPC call timeout handler was delayed by -1.4451043570000479s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20260709 13:03:30.646386  1632 connection.cc:256] RPC call timeout handler was delayed by -1.3821043570000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
I20260709 13:03:30.646462  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.601253  1733 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:30.552824  6979 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.467185  7170 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.653995  7175 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.654356  7170 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:30.654840  7072 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:30.655458  7072 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.656651  6954 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.596110  6903 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:30.635797  7095 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 13:03:30.606513  7174 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:30.658694  7072 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:30.545270  7064 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.828s	user 0.015s	sys 0.004s
I20260709 13:03:30.662533  7064 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.611276  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2867) took 1625 ms. Trace:
I20260709 13:03:30.611851  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.616624  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:30.665740  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:30.580214  7179 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:30.667475  7179 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.668140  6979 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.581140  7083 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:30.669793  7083 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:30.670470  7083 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.593290  7052 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.640436  7011 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.674348  1586 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:30.609108  7180 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:30.665143  1592 rpcz_store.cc:276] 0709 13:03:28.986137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.986461 (+   324us) service_pool.cc:224] Handling call
0709 13:03:28.987096 (+   635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.373651 (+1386555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.611018 (+237367us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540031c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:30.611207 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3143898880}
I20260709 13:03:30.573593  2318 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:30.681222  1667 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:30.629171  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:30.683749  1660 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:30.583698  6956 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:30.646873  1632 connection.cc:256] RPC call timeout handler was delayed by -1.2174043570000208s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260709 13:03:30.685086  1632 connection.cc:256] RPC call timeout handler was delayed by -1.1220043570000371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
I20260709 13:03:30.687428  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:30.596105  7151 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:30.648622  1742 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:30.650161  1734 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:30.651957  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.681226  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:29.319566  1668 rpcz_store.cc:276] 0709 13:03:27.893536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:28.445335 (+551799us) service_pool.cc:224] Handling call
0709 13:03:28.445790 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:28.446073 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:28.832116 (+386043us) log.cc:844] Serialized 14 byte log entry
0709 13:03:29.318204 (+486088us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:29.318291 (+    87us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:29.318367 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:29.318468 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:29.318639 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29012,"prepare.queue_time_us":70874,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":124,"thread_start_us":829127,"threads_started":2}
I20260709 13:03:30.700562  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.702117  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:30.700595  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.703855  1734 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:30.705212  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:30.714074  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:30.718204  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:30.719241  1742 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:30.722806  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.723869  1742 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:30.644573  7103 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:30.664613  6736 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.725819  1742 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:03:30.746214  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 134441us
I20260709 13:03:30.747586  1668 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:30.719681  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75560.8us
I20260709 13:03:30.728513  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.730197  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:30.749922  1584 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:30.730814  7184 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:30.743958  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:30.613395  7071 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:30.735453  7110 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:30.719265  7072 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:30.747496  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:30.803241  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55857.8us
W20260709 13:03:30.804245  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.754289  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.756443  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:30.732481  7167 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:30.763653  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.793862  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:30.815243  1589 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:30.794926  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46007.1us
W20260709 13:03:30.795282  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81370.5us
I20260709 13:03:30.692022  6754 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:30.695971  6376 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:30.804101  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.801929  1660 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:30.727236  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:30.818295  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:30.820379  1742 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:30.828806  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:30.741042  7103 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:30.829655  1660 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:30.830739  1667 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:30.798558  6968 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:30.718112  7169 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260709 13:03:30.831981  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:30.847404  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 892) took 1646 ms (client timeout 1735 ms). Trace:
W20260709 13:03:30.847895  1737 rpcz_store.cc:269] 0709 13:03:29.201048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.244020 (+ 42972us) service_pool.cc:224] Handling call
0709 13:03:30.847202 (+1603182us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001081d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:30.847334 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2963496832}
I20260709 13:03:30.850061  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:30.851038  1660 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:30.851402  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:30.851991  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:30.852584  1660 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:30.858361  1741 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:30.831239  1738 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:30.789309  1584 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:30.839555  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:30.840173  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 113033us
I20260709 13:03:30.860029  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:30.841192  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2172) took 1664 ms (client timeout 2135 ms). Trace:
W20260709 13:03:30.860699  1735 rpcz_store.cc:269] 0709 13:03:29.176348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.204293 (+ 27945us) service_pool.cc:224] Handling call
0709 13:03:30.840910 (+1636617us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:30.841110 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1218902272}
I20260709 13:03:30.842622  1739 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
W20260709 13:03:30.863812  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.802256  7184 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:30.872421  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.880393  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:30.813304  7095 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 13:03:30.895278  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55837.1us
I20260709 13:03:30.895807  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:30.755474  6954 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.084s
I20260709 13:03:30.797360  7022 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:03:30.897251  1593 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:30.899576  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:30.927702  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:30.913123  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:30.818190  6989 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 13:03:30.882730  6979 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:30.938324  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:30.938647  1737 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:30.883190  7151 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:30.891206  6935 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:30.919623  7043 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:30.919098  7110 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:30.892962  7167 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:30.921643  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:30.943210  7039 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:30.917356  7155 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:30.822733  7072 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.831032  6754 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:30.986969  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:30.935477  7168 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:30.943441  6968 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:30.929270  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:31.005890  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76729.8us
I20260709 13:03:30.988729  6935 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:31.010746  6935 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:31.013388  7157 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.895s	user 0.004s	sys 0.000s
W20260709 13:03:31.015075  7188 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:03:31.018285  6884 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:31.022015  7187 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 13:03:31.026721  6754 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:31.030299  7071 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:30.989076  7091 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.627s	user 0.012s	sys 0.000s
I20260709 13:03:30.965868  7043 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:30.982969  7153 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:31.048239  7043 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.051330  7184 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:31.052767  7155 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:31.053977  7050 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:31.054443  7050 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:31.055436  7050 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:30.995054  7039 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.057794  7039 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:31.064636  7072 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:31.065954  6899 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:31.066937  7174 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.068682  7064 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:30.987951  7103 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.070349  7044 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:31.070585  7103 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.070780  7044 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:31.071815  7044 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.073171  7044 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:31.045823  7163 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:31.046265  6968 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.077984  7153 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:30.976104  7110 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:31.069214  7174 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:31.088114  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81207.8us
I20260709 13:03:31.044159  7169 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:31.098541  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:31.259297  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 346145us
W20260709 13:03:31.260694  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 162121us
I20260709 13:03:31.270056  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:31.095359  7110 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:31.153309  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 319323us
W20260709 13:03:31.303982  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 382292us
I20260709 13:03:31.306097  7180 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:31.255864  1585 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:31.108726  6967 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.117153  6736 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:31.117372  7050 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:31.324426  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:31.328298  7083 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.657s	user 0.012s	sys 0.002s
I20260709 13:03:31.330109  7134 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:31.331432  7134 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.332836  7137 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:31.336596  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2955) took 1059 ms. Trace:
I20260709 13:03:31.336841  7137 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:31.155640  7145 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.553s	user 0.011s	sys 0.003s
I20260709 13:03:31.109465  7153 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:31.193153  7185 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
W20260709 13:03:31.189147  7189 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 13:03:31.181082  7195 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 13:03:31.158423  6968 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:31.302461  7011 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.628s	user 0.013s	sys 0.000s
I20260709 13:03:31.372357  7011 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:31.278064  6481 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:31.306943  7123 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.885s	user 0.005s	sys 0.015s
I20260709 13:03:31.316216  6828 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.317360  7167 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:31.317468  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:31.318755  7168 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260709 13:03:31.275532  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.337239  1592 rpcz_store.cc:276] 0709 13:03:30.276641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.729665 (+453024us) service_pool.cc:224] Handling call
0709 13:03:30.744997 (+ 15332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.745310 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:30.805085 (+ 59775us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.335545 (+530460us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.335690 (+   145us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:31.335854 (+   164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.335997 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.336260 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77138,"prepare.queue_time_us":408086,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":79513,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":208,"thread_start_us":452462,"threads_started":1,"wal-append.queue_time_us":358874}
I20260709 13:03:31.341696  7137 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:31.329097  7083 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.365461  6967 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.367453  6736 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:03:31.380681  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.382598  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1011) took 1959 ms. Trace:
I20260709 13:03:31.379312  6481 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.383424  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2798) took 1011 ms. Trace:
I20260709 13:03:31.390831  6967 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.392803  6481 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.391491  1664 rpcz_store.cc:276] 0709 13:03:30.371951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.372080 (+   129us) service_pool.cc:224] Handling call
0709 13:03:30.407341 (+ 35261us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:30.418109 (+ 10768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:30.418161 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:30.434356 (+ 16195us) log.cc:844] Serialized 52 byte log entry
0709 13:03:30.613432 (+179076us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:30.650970 (+ 37538us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.651247 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:30.651325 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.192306 (+540981us) spinlock_profiling.cc:243] Waited 7.49 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:31.372991 (+180685us) raft_consensus.cc:1685] finished
0709 13:03:31.373229 (+   238us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.383213 (+  9984us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34056,"raft.queue_time_us":2307,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":15228,"spinlock_wait_cycles":17219968,"thread_start_us":148235,"threads_started":1,"wal-append.queue_time_us":439175}
I20260709 13:03:31.394264  6967 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.394111  6828 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.395108  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2879) took 1129 ms. Trace:
I20260709 13:03:31.395491  1593 rpcz_store.cc:276] 0709 13:03:30.266151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.267906 (+  1755us) service_pool.cc:224] Handling call
0709 13:03:30.292654 (+ 24748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.931138 (+638484us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.394567 (+463429us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400442d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:31.394915 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1390029568}
I20260709 13:03:31.396659  6828 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.383916  7145 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:31.384490  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2810) took 1028 ms. Trace:
W20260709 13:03:31.384862  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67885.4us
I20260709 13:03:31.385957  7011 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:31.390797  7149 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:03:31.401059  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 2951) took 2265 ms (client timeout 2235 ms). Trace:
W20260709 13:03:31.401541  1585 rpcz_store.cc:269] 0709 13:03:29.135426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.257947 (+122521us) service_pool.cc:224] Handling call
0709 13:03:31.400771 (+2142824us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400002654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:31.400960 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4455235840}
I20260709 13:03:31.403123  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.405656  7011 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.406132  7011 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:31.405156  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.410352  1590 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:31.389835  2318 rpcz_store.cc:276] 0709 13:03:29.419881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.421296 (+  1415us) service_pool.cc:224] Handling call
0709 13:03:31.379008 (+1957712us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b5400719654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:31.379283 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005604736}
I20260709 13:03:31.398635  1741 rpcz_store.cc:276] 0709 13:03:30.334716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.337467 (+  2751us) service_pool.cc:224] Handling call
0709 13:03:30.858071 (+520604us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:30.883197 (+ 25126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.358661 (+475464us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b5400108bd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:31.362866 (+  4205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195959680}
W20260709 13:03:31.427815  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.430831  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 833) took 1145 ms. Trace:
I20260709 13:03:31.431658  6996 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:31.432873  6996 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:31.434846  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:31.436288  1664 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:31.407349  7149 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:31.437570  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.436641  6996 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.438485  1584 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:31.438686  7149 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.439803  6996 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:31.440709  7149 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:31.440793  7100 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:31.441383  7100 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:31.445870  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 170787us
I20260709 13:03:31.447086  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:31.448266  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2806) took 1073 ms. Trace:
W20260709 13:03:31.450897  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 181279us
I20260709 13:03:31.452392  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:31.453423  6376 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:31.454653  6485 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:31.455451  7083 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:31.455796  6376 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:31.456787  7167 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:31.457895  7172 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:31.458060  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2807) took 1082 ms. Trace:
I20260709 13:03:31.457475  6376 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.458644  7172 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:31.464313  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.440017  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.464674  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:31.466513  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44525us
I20260709 13:03:31.398654  7134 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:31.467711  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60844.9us
I20260709 13:03:31.458597  1659 rpcz_store.cc:276] 0709 13:03:30.375402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.408635 (+ 33233us) service_pool.cc:224] Handling call
0709 13:03:30.430975 (+ 22340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:30.443599 (+ 12624us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:30.802794 (+359195us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.446724 (+643930us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.453116 (+  6392us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:31.454214 (+  1098us) log.cc:844] Serialized 26 byte log entry
0709 13:03:31.454378 (+   164us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:31.454428 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.454689 (+   261us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.457458 (+  2769us) raft_consensus.cc:1685] finished
0709 13:03:31.457482 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.457750 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":235521,"prepare.queue_time_us":244064,"prepare.run_cpu_time_us":163,"prepare.run_wall_time_us":164,"raft.queue_time_us":81333,"raft.run_cpu_time_us":688,"raft.run_wall_time_us":33369,"thread_start_us":702747,"threads_started":2,"wal-append.queue_time_us":368008}
I20260709 13:03:31.469470  1584 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:31.467880  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.471678  7174 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:31.472153  7174 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:31.472327  6899 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.472543  7174 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.473169  6899 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.467819  7178 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:31.431526  1593 rpcz_store.cc:276] 0709 13:03:30.279776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.396488 (+1116712us) service_pool.cc:224] Handling call
0709 13:03:31.397057 (+   569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.397463 (+   406us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.397491 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.416316 (+ 18825us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:31.424491 (+  8175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.424582 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.424712 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.494094  7100 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.494943  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2962) took 1212 ms. Trace:
I20260709 13:03:31.495754  7100 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:31.495690  1593 rpcz_store.cc:276] 0709 13:03:30.282646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.489757 (+1207111us) service_pool.cc:224] Handling call
0709 13:03:31.494217 (+  4460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.494583 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.494609 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.494654 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:31.494686 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.494740 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.494855 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.501724  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 433832us
I20260709 13:03:31.502149  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2959) took 1223 ms. Trace:
I20260709 13:03:31.476258  7178 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:31.482234  6899 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:31.490216  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.504060  1586 rpcz_store.cc:276] 0709 13:03:30.277588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.331719 (+1054131us) service_pool.cc:224] Handling call
0709 13:03:31.333897 (+  2178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.334573 (+   676us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:31.353804 (+ 19231us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.492376 (+138572us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.492442 (+    66us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:31.492503 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.492578 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.501339 (+  8761us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18732,"prepare.run_wall_time_us":1971,"raft.queue_time_us":1519,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"spinlock_wait_cycles":2236288,"thread_start_us":110868,"threads_started":1}
I20260709 13:03:31.526000  1590 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:31.529744  1586 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 13:03:31.530536  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2964) took 1247 ms. Trace:
I20260709 13:03:31.530903  1586 rpcz_store.cc:276] 0709 13:03:30.283200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.528761 (+1245561us) service_pool.cc:224] Handling call
0709 13:03:31.529317 (+   556us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:31.530241 (+   924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.530455 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.532609  1586 pending_rounds.cc:85] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:31.532912  1586 pending_rounds.cc:107] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 13:03:31.533197  1586 raft_consensus.cc:2889] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:31.534004  1664 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:31.535279  1738 pending_rounds.cc:85] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 4
I20260709 13:03:31.535688  1738 pending_rounds.cc:107] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 13:03:31.535985  1738 raft_consensus.cc:2889] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:31.449186  1663 rpcz_store.cc:276] 0709 13:03:30.375083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.405394 (+ 30311us) service_pool.cc:224] Handling call
0709 13:03:30.405810 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:30.406297 (+   487us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:30.426720 (+ 20423us) log.cc:844] Serialized 14 byte log entry
0709 13:03:30.551694 (+124974us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b4800210258. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:03:30.620993 (+ 69299us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:30.622086 (+  1093us) log.cc:844] Serialized 52 byte log entry
0709 13:03:30.622298 (+   212us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:31.446184 (+823886us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b1800448600. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:03:31.446659 (+   475us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.447177 (+   518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.447310 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.447384 (+    74us) raft_consensus.cc:1685] finished
0709 13:03:31.447399 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.447975 (+   576us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43163,"prepare.queue_time_us":844822,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":2401,"raft.queue_time_us":13669,"raft.run_cpu_time_us":2709,"raft.run_wall_time_us":153337,"spinlock_wait_cycles":67325696,"thread_start_us":146576,"threads_started":1,"wal-append.queue_time_us":375841}
I20260709 13:03:31.546104  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2958) took 1268 ms. Trace:
I20260709 13:03:31.546741  1587 rpcz_store.cc:276] 0709 13:03:30.277345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.897982 (+620637us) service_pool.cc:224] Handling call
0709 13:03:30.943648 (+ 45666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:30.944020 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:30.944057 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:30.944767 (+   710us) log.cc:844] Serialized 26 byte log entry
0709 13:03:31.490372 (+545605us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:31.512167 (+ 21795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.523123 (+ 10956us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.545629 (+ 22506us) raft_consensus.cc:1685] finished
0709 13:03:31.545652 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.545873 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":436090,"threads_started":1,"wal-append.queue_time_us":458583}
I20260709 13:03:31.564337  1587 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:03:31.514088  7178 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.572371  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2190) took 1011 ms. Trace:
I20260709 13:03:31.577252  1742 rpcz_store.cc:276] 0709 13:03:30.560721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.832071 (+271350us) service_pool.cc:224] Handling call
0709 13:03:30.832398 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.079930 (+247532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.079967 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.080014 (+    47us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:31.080045 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.571852 (+491807us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:31.571922 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.572183 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568786176}
I20260709 13:03:31.573037  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:31.575219  1588 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 13:03:31.591724  6899 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:31.592653  6899 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.593676  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.594586  1662 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:31.597175  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:31.598091  1663 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.599035  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2969) took 1297 ms. Trace:
I20260709 13:03:31.599658  1587 rpcz_store.cc:276] 0709 13:03:30.284619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.548524 (+1263905us) service_pool.cc:224] Handling call
0709 13:03:31.549176 (+   652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:31.581811 (+ 32635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.582090 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.587599  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2972) took 1302 ms. Trace:
W20260709 13:03:31.604285  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 154926us
I20260709 13:03:31.602862  1588 rpcz_store.cc:276] 0709 13:03:30.285524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.573400 (+1287876us) service_pool.cc:224] Handling call
0709 13:03:31.573958 (+   558us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.580312 (+  6354us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.587499 (+  7187us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.614176  7044 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.520s	user 0.016s	sys 0.004s
W20260709 13:03:31.616145  7215 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
W20260709 13:03:31.617862  7043 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.014s
I20260709 13:03:31.618430  7043 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.624897  7172 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.624507  7110 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:31.631435  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 179323us
I20260709 13:03:31.626057  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:31.635254  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.637274  6481 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 13:03:31.642293  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:31.643290  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.641930  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:31.646497  1662 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:31.631436  7043 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:31.657220  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.658910  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:31.660388  7217 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 13:03:31.661471  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 834) took 1366 ms. Trace:
I20260709 13:03:31.654776  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:31.671487  1587 rpcz_store.cc:276] 0709 13:03:30.295353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.600783 (+1305430us) service_pool.cc:224] Handling call
0709 13:03:31.656345 (+ 55562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.661096 (+  4751us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.661124 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:31.661190 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:31.661227 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.661285 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.661411 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:31.663141  7214 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 13:03:31.663647  6954 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:31.663998  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "219129ac643e4650b611a85f565f5990"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:31.674844  2318 raft_consensus.cc:606] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:31.665318  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.665805  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.667193  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.667465  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2960) took 1389 ms. Trace:
I20260709 13:03:31.675815  1592 rpcz_store.cc:276] 0709 13:03:30.282040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.383046 (+1101006us) service_pool.cc:224] Handling call
0709 13:03:31.383682 (+   636us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:31.384454 (+   772us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.384497 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:31.385230 (+   733us) log.cc:844] Serialized 26 byte log entry
0709 13:03:31.492862 (+107632us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:31.492909 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.492986 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.667190 (+174204us) raft_consensus.cc:1685] finished
0709 13:03:31.667211 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.667371 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1228,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":221,"thread_start_us":105160,"threads_started":1,"wal-append.queue_time_us":251848}
I20260709 13:03:31.689610  1664 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:31.693284  1592 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:31.695279  1592 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260709 13:03:31.695461  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.696659  1592 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:31.675318  6481 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:31.701177  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2956) took 1424 ms. Trace:
I20260709 13:03:31.704366  1735 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:31.701624  1589 rpcz_store.cc:276] 0709 13:03:30.276904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.827490 (+550586us) service_pool.cc:224] Handling call
0709 13:03:30.892490 (+ 65000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:30.892747 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:30.892790 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:30.893420 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 13:03:31.491314 (+597894us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:31.520649 (+ 29335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.528938 (+  8289us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.700776 (+171838us) raft_consensus.cc:1685] finished
0709 13:03:31.700797 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.700982 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":425512,"threads_started":1,"wal-append.queue_time_us":593266}
I20260709 13:03:31.713876  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:31.714299  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 138226us
I20260709 13:03:31.712915  1742 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:31.696682  7202 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:31.719293  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:31.719360  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60647.9us
W20260709 13:03:31.720063  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45748.4us
I20260709 13:03:31.725674  1592 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:31.737270  1587 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:31.728026  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:31.719733  7202 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:31.728421  1588 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:31.748651  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:31.700879  6481 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.688145  7044 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:31.754942  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 835) took 1432 ms. Trace:
I20260709 13:03:31.756842  1588 rpcz_store.cc:276] 0709 13:03:30.316783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.646784 (+1330001us) service_pool.cc:224] Handling call
0709 13:03:31.707514 (+ 60730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.748638 (+ 41124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.748845 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.758131  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 836) took 1437 ms. Trace:
I20260709 13:03:31.758675  1587 rpcz_store.cc:276] 0709 13:03:30.317152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.672936 (+1355784us) service_pool.cc:224] Handling call
0709 13:03:31.724716 (+ 51780us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:31.747303 (+ 22587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.754235 (+  6932us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:31.762539  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.775260  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.778650  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.775207  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:31.783783  1664 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:03:31.747635  7137 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.794202  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.791973  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.790118  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.797223  1668 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:31.797689  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.799059  2310 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:31.800395  7050 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:31.806331  2318 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:31.808655  6899 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:31.813354  1589 raft_consensus.cc:2368] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:03:31.813695  6954 raft_consensus.cc:993] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:31.814208  6954 raft_consensus.cc:1081] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:31.814960  1589 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:31.815663  7202 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.817221  7202 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:31.817963  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.821722  7158 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:31.822247  7158 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:31.822631  7158 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.825009  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:31.830453  1661 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:31.804381  7110 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:31.825992  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55937.6us
I20260709 13:03:31.833585  7172 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:31.785602  7043 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:31.827713  7039 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.769s	user 0.012s	sys 0.009s
I20260709 13:03:31.826953  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.838006  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2965) took 1554 ms. Trace:
I20260709 13:03:31.845894  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.846638  1592 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:31.848219  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.849611  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.849825  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:31.850438  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56699.6us
I20260709 13:03:31.851281  7050 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:31.773722  7044 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.846442  1667 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:31.855306  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.812989  6376 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
W20260709 13:03:31.856525  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.842686  1586 rpcz_store.cc:276] 0709 13:03:30.283458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.531647 (+1248189us) service_pool.cc:224] Handling call
0709 13:03:31.532213 (+   566us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:31.537775 (+  5562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:31.537841 (+    66us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:31.539637 (+  1796us) log.cc:844] Serialized 78 byte log entry
0709 13:03:31.629194 (+ 89557us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:31.786424 (+157230us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.792523 (+  6099us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.792806 (+   283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.792883 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.832303 (+ 39420us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b180042f820. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdf1eb158f 00007fcdeaef0571 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd
0709 13:03:31.833131 (+   828us) spinlock_profiling.cc:243] Waited 2.46 ms on lock 0x7b180042f820. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:31.834018 (+   887us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b180042f820. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:31.837756 (+  3738us) raft_consensus.cc:1685] finished
0709 13:03:31.837777 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.837943 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1209,"prepare.queue_time_us":68581,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":25972,"raft.queue_time_us":2949,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":27853,"spinlock_wait_cycles":64565248,"thread_start_us":213693,"threads_started":2,"wal-append.queue_time_us":211169}
I20260709 13:03:31.862392  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:31.866662  7172 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:31.867766  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:31.869534  6956 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.159s	user 0.012s	sys 0.009s
W20260709 13:03:31.884884  7219 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 13:03:31.884693  7035 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:31.884501  1486 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:31.885608  7216 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.126s	user 0.002s	sys 0.000s
I20260709 13:03:31.885886  7100 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:31.886438  7100 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:31.887784  7100 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:31.889183  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 419970us
I20260709 13:03:31.892087  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:31.892465  7100 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.892946  1587 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:31.886723  1486 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:31.881189  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.856971  7050 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:31.901336  1663 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:31.930867  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 839) took 1292 ms. Trace:
I20260709 13:03:31.931555  1590 rpcz_store.cc:276] 0709 13:03:30.617849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:30.704860 (+ 87011us) service_pool.cc:224] Handling call
0709 13:03:31.882183 (+1177323us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b54003b15d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:31.910264 (+ 28081us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1619367680}
I20260709 13:03:31.904671  7167 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:31.944101  7083 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:31.947486  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:31.950630  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:31.857081  7039 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:31.952147  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:31.878129  7043 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:31.954388  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:31.948215  2310 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:03:31.957785  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95732.9us
I20260709 13:03:31.878458  7011 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:31.960654  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48183.5us
I20260709 13:03:31.955736  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:31.945283  7167 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:31.982638  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.974826  7075 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 13:03:31.975011  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.975416  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2872) took 2507 ms. Trace:
W20260709 13:03:31.977213  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.879s (SENT)
I20260709 13:03:31.986227  2312 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:31.991308  1590 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:31.972186  6481 raft_consensus.cc:2764] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 13:03:31.995056  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 58808us
I20260709 13:03:31.995440  7110 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:31.995852  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:31.996464  7110 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:31.998332  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 71680.7us
I20260709 13:03:31.972209  6736 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:31.999344  6736 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:31.999810  6736 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:31.999950  6485 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:32.000411  6485 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:32.001412  6485 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:32.003715  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70074.2us
I20260709 13:03:32.009675  6485 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:31.986986  7167 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:32.010581  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:32.012791  6481 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:32.013312  6481 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:32.014175  1486 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.014405  6481 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:32.006254  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80745.6us
I20260709 13:03:32.020419  7158 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.019872  1486 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.022526  1486 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.023355  7011 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.024194  2311 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:32.007627  7172 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:31.987270  7075 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.026103  7172 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.026696  7075 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.027913  7043 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.028403  7172 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.029001  7172 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:32.009310  7178 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.045715  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.046581  1588 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:32.046816  1584 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:32.048267  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2961) took 1765 ms. Trace:
I20260709 13:03:31.968035  7083 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:32.029222  7044 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:32.008163  7149 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.567s	user 0.018s	sys 0.000s
I20260709 13:03:32.007123  7039 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:31.985097  1591 rpcz_store.cc:276] 0709 13:03:29.386150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:29.386359 (+   209us) service_pool.cc:224] Handling call
0709 13:03:29.452594 (+ 66235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:29.453065 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:30.637552 (+1184487us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b18002bf300. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff05e5 00007fcdedff0568
0709 13:03:30.950067 (+312515us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.331331 (+381264us) log.cc:844] Serialized 14 byte log entry
0709 13:03:31.331683 (+   352us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:31.332708 (+  1025us) log.cc:844] Serialized 26 byte log entry
0709 13:03:31.332935 (+   227us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:31.332977 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:31.333141 (+   164us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:31.893350 (+560209us) raft_consensus.cc:1685] finished
0709 13:03:31.893380 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:31.893593 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5266,"prepare.queue_time_us":269506,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":133,"raft.queue_time_us":2486,"raft.run_cpu_time_us":719,"raft.run_wall_time_us":16413,"spinlock_wait_cycles":9653504,"thread_start_us":363623,"threads_started":2,"wal-append.queue_time_us":107920}
I20260709 13:03:32.024410  1486 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 11 to 13, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.048518  7158 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.051599  6828 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:32.056401  1486 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 9, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:32.005064  7110 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:32.041780  7227 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:03:32.042492  7050 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.051532  1634 leader_election.cc:400] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:03:32.060290  1634 leader_election.cc:403] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
W20260709 13:03:32.061296  7174 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.588s	user 0.016s	sys 0.000s
I20260709 13:03:32.048722  1584 rpcz_store.cc:276] 0709 13:03:30.282368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.493448 (+1211080us) service_pool.cc:224] Handling call
0709 13:03:31.518099 (+ 24651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.047265 (+529166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.047961 (+   696us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b54001c3c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:32.048193 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1215521664}
I20260709 13:03:32.062304  1590 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:32.052029  1585 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:31.990626  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:32.046593  7178 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.063419  7174 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.074028  1593 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 14. Ops: []
I20260709 13:03:32.087005  1661 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:32.100288  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:32.102303  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112717us
W20260709 13:03:32.103292  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.109977  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.112649  6956 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.113057  1659 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:32.113304  6956 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.114421  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.116016  7153 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.116791  7153 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.113788  1587 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:32.120872  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 2963) took 1837 ms. Trace:
W20260709 13:03:32.123793  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.112246  1486 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.121395  1593 rpcz_store.cc:276] 0709 13:03:30.282931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:31.496927 (+1213996us) service_pool.cc:224] Handling call
0709 13:03:31.497408 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.084911 (+587503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:32.095604 (+ 10693us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:32.120778 (+ 25174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1315610496}
W20260709 13:03:32.135229  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 136012us
I20260709 13:03:32.143581  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:32.155089  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 177969us
I20260709 13:03:32.159598  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.065795  7149 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 13:03:32.126925  7137 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:32.168299  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.169415  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.170985  1663 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:32.171698  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110528us
I20260709 13:03:32.097587  7158 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 13:03:32.180706  2312 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:32.181597  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:32.194852  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69854.7us
I20260709 13:03:32.191344  1659 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:32.182044  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.155570  6956 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:32.199788  7103 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.129s	user 0.007s	sys 0.018s
I20260709 13:03:32.200032  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:32.192055  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.904s (SENT) [suppressed 21 similar messages]
W20260709 13:03:32.203245  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47312.6us
I20260709 13:03:32.189131  1742 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
W20260709 13:03:32.227522  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.219141  2310 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:03:32.218228  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.216904  1486 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 8 to 16, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 16 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.246882  7174 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.172634  7235 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:32.167270  7011 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 13:03:32.253293  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.226744  7240 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 13:03:32.212909  7050 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.233819  7239 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:32.283381  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.217911  7233 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.221849  7237 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:03:32.288481  1733 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:32.287945  6985 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:32.296927  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 85075.7us
I20260709 13:03:32.260624  7039 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:32.300956  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 119686us
I20260709 13:03:32.266876  6934 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:32.311331  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.323930  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:32.327003  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.301437  6985 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:32.338814  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115473us
I20260709 13:03:32.333655  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.312942  6934 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:32.345489  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:32.346226  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 167860us
I20260709 13:03:32.274107  7083 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.344906  7149 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 13:03:32.362538  1664 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 11: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:32.333289  6985 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:32.296442  7239 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.349815  6934 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:32.370765  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:32.357383  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:32.372452  1588 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:32.303812  7235 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.357657  6904 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:32.387481  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:32.386477  1486 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 4 to 14. New cstate: current_term: 14 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.343840  6736 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.413949  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:32.425402  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 109554us
I20260709 13:03:32.368084  7240 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:32.426576  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:32.431353  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.429466  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79663.2us
I20260709 13:03:32.414337  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.429958  1663 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:32.390700  7103 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:32.400197  7100 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.507s	user 0.013s	sys 0.000s
I20260709 13:03:32.419610  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:32.436962  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 79700.2us
I20260709 13:03:32.401034  7035 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:32.393972  7172 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:32.420660  1486 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.389191  6904 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:32.445848  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.437578  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.396538  6899 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:32.410611  7064 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:32.439437  1664 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 13:03:32.453742  1668 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:32.429657  7207 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:32.414258  7153 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.462180  1660 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:03:32.378723  7231 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 13:03:32.454571  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:32.431960  7240 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:32.454181  6956 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:32.490494  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.491580  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.420737  7250 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 13:03:32.483690  1586 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:03:32.495461  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.462352  6904 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:32.463061  7064 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:32.487389  1486 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 8 to 16. New cstate: current_term: 16 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.458412  7172 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:32.506162  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.513820  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.507052  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.508046  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.513018  1590 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.7-12.7]
W20260709 13:03:32.522644  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:32.472779  7035 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:32.515887  1664 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:32.535430  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.536904  1584 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:32.496354  7103 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.542804  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.516016  7064 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:32.550285  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.551183  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.551492  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.551220  1486 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:32.566943  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.567751  1590 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:32.575543  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.533260  7137 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.578765  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.578204  1739 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:32.578642  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.579449  1591 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:32.591110  1740 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:32.591951  1735 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:32.558130  6956 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:32.593887  1660 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:03:32.594307  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.594511  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.610494  1663 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:32.558244  7207 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:32.581713  1736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "219129ac643e4650b611a85f565f5990"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
I20260709 13:03:32.566430  7251 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 13:03:32.601908  7237 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:32.602088  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:32.603760  1488 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 237) took 62357 ms (client timeout 15000 ms). Trace:
I20260709 13:03:32.603886  7247 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.147s
I20260709 13:03:32.604480  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.606305  1737 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 13:03:32.607502  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:32.612258  1736 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:32.612279  2311 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:32.616523  1593 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:32.618376  2316 raft_consensus.cc:606] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:32.631686  1736 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:32.632773  1666 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:32.613270  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:32.560372  6899 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:32.629356  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.609441  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:32.631218  1590 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:32.618808  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:32.628660  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:32.634054  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53405us
I20260709 13:03:32.637949  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.635340  1587 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:32.643503  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:32.622515  7075 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.595s	user 0.016s	sys 0.000s
I20260709 13:03:32.639811  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.656876  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.663256  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.668105  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:32.657816  1662 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:32.669860  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:32.644152  7243 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.275s	user 0.001s	sys 0.001s
I20260709 13:03:32.674079  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.677682  2312 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:32.672207  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.621233  7237 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:32.686820  7262 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:32.687584  7262 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.688967  2313 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:32.690411  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:32.690531  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.692377  1735 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:03:32.694011  1735 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:32.696086  7239 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.696846  1586 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:32.696583  6956 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.697654  7207 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.700248  1584 raft_consensus.cc:2376] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13. Re-sending same reply.
I20260709 13:03:32.613651  7172 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.702700  7172 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:32.702858  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:32.703552  1734 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:32.705624  1660 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 13:03:32.706123  7241 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 13:03:32.706545  1739 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:32.708812  1739 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:32.710598  6899 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.711239  7251 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:32.711162  6899 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.711887  7248 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:32.712574  7248 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:32.680495  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.682142  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.720275  1738 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:32.605233  7035 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.734593  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:32.735332  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 315853us
I20260709 13:03:32.617978  7137 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.740069  1737 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:32.741823  1664 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:03:32.720624  7039 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:32.645186  7174 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:03:32.755636  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.668885  7011 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 13:03:32.759627  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:32.724938  6899 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:32.723912  6736 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.664256  7075 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.624743  7240 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.6, Last appended by leader: 6, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.769145  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.770783  1735 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 13:03:32.770759  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:32.769054  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.711882  7251 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.752980  7235 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:32.786583  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:32.789129  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 119376us
I20260709 13:03:32.789211  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:32.791553  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 156640us
I20260709 13:03:32.790081  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:32.799253  1664 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:32.803344  1585 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:32.811203  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:32.814962  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.815888  1584 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:32.829377  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:32.830559  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.844990  7075 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.789042  7265 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 13:03:32.846786  7251 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.803562  7262 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:32.847584  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 91536.3us
I20260709 13:03:32.848515  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:32.849977  7234 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 13:03:32.850234  6899 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:32.850801  7137 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.827298  7235 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 13:03:32.859186  7244 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:32.859876  7244 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:32.862314  7245 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.330s	user 0.004s	sys 0.000s
I20260709 13:03:32.862676  7240 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:32.862632  7253 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.864769  7253 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.830639  1666 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:32.808967  7239 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.834344  1736 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.882663  1667 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:32.885736  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:32.884750  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:32.896318  1736 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:32.883673  7239 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:32.918030  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:32.921252  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:32.894624  7266 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:32.891548  7174 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:03:32.622115  1488 rpcz_store.cc:269] 0709 13:02:30.102015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:30.102403 (+   388us) service_pool.cc:224] Handling call
0709 13:03:32.451955 (+62349552us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:24.023592 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:24.026829 (+  3237us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:24.026895 (+    66us) write_op.cc:435] Acquired schema lock
0709 13:03:24.026914 (+    19us) tablet.cc:662] Decoding operations
0709 13:03:24.058761 (+ 31847us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:24.058956 (+   195us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:24.058987 (+    31us) tablet.cc:685] Acquiring locks for 12 operations
0709 13:03:24.062170 (+  3183us) tablet.cc:701] Row locks acquired
0709 13:03:24.062216 (+    46us) write_op.cc:260] PREPARE: finished
0709 13:03:24.062457 (+   241us) write_op.cc:270] Start()
0709 13:03:24.062772 (+   315us) write_op.cc:276] Timestamp: P: 1783602204062382 usec, L: 0
0709 13:03:24.062825 (+    53us) op_driver.cc:348] REPLICATION: starting
0709 13:03:24.211179 (+148354us) log.cc:844] Serialized 4795 byte log entry
0709 13:03:25.777350 (+1566171us) op_driver.cc:464] REPLICATION: finished
0709 13:03:26.380471 (+603121us) write_op.cc:301] APPLY: starting
0709 13:03:26.391307 (+ 10836us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:26.561216 (+169909us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:26.561359 (+   143us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:27.055732 (+494373us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:27.081401 (+ 25669us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0709 13:03:27.081478 (+    77us) write_op.cc:312] APPLY: finished
0709 13:03:27.082030 (+   552us) log.cc:844] Serialized 113 byte log entry
0709 13:03:27.403929 (+321899us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:27.409022 (+  5093us) write_op.cc:454] Released schema lock
0709 13:03:27.447790 (+ 38768us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:27.543837 (+ 96047us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:27.551289 (+  7452us) write_op.cc:454] Released schema lock
0709 13:03:28.298426 (+747137us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b18000995e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
Metrics: {"mutex_wait_us":406823,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":521792,"apply.run_cpu_time_us":49375,"apply.run_wall_time_us":1127132,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54311,"num_ops":12,"prepare.queue_time_us":337116,"prepare.run_cpu_time_us":120869,"prepare.run_wall_time_us":699036,"raft.queue_time_us":475660,"raft.run_cpu_time_us":16986,"raft.run_wall_time_us":452862,"replication_time_us":1680719,"spinlock_wait_cycles":18456960,"thread_start_us":1232159,"threads_started":5,"wal-append.queue_time_us":1031212,"wal-append.run_cpu_time_us":17678,"wal-append.run_wall_time_us":2615290}]]}
W20260709 13:03:32.929301  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44714.7us
I20260709 13:03:32.930308  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:32.940248  7256 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 13:03:32.941766  1742 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:32.942147  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.950068  1734 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:32.925300  7246 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:32.929402  7174 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:32.972123  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.973717  1736 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
W20260709 13:03:32.973840  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43608.2us
I20260709 13:03:32.831226  7248 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:32.989677  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.991043  1734 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:32.978514  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:32.974822  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:32.994899  1590 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:32.999145  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:32.999491  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.962945  7252 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:33.013218  1736 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 13:03:32.977957  7174 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:33.034431  1734 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:33.035388  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:32.991441  7244 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:32.936718  7266 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.005986  7247 raft_consensus.cc:2764] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:03:33.052735  7201 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 13:03:33.055639  7083 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:33.058838  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:33.058940  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.071875  1593 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:33.084547  1665 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:33.087446  1738 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:33.028220  6485 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:03:33.096351  1584 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:33.095100  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:33.088694  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 113992us
I20260709 13:03:33.095746  1739 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:33.102430  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.103197  1660 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:33.094369  7277 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:33.104432  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.104547  7277 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:33.105163  7277 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.106020  7277 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:33.089386  7233 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.766s	user 0.011s	sys 0.003s
I20260709 13:03:33.108031  7278 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.108505  7233 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:33.109927  7064 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.524s	user 0.013s	sys 0.000s
I20260709 13:03:33.110644  1733 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:33.108613  7278 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:33.114279  7267 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 13:03:33.084786  7083 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:33.117761  7083 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:33.118953  7083 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.109359  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.110425  7064 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.120428  7083 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:33.050897  7264 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:33.116242  7273 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:33.117390  1663 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:33.116799  7278 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.124068  1660 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:33.127194  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.027878  6904 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:33.132344  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 114574us
W20260709 13:03:33.132934  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 166195us
I20260709 13:03:33.140609  1660 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 13:03:33.141738  1660 pending_rounds.cc:85] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 8
I20260709 13:03:33.123055  7039 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 13:03:33.145139  7271 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 13:03:33.142500  1660 pending_rounds.cc:107] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 16.9
W20260709 13:03:33.146864  7275 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 13:03:33.146914  1660 raft_consensus.cc:2889] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:33.150338  7247 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:33.151458  7247 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 13:03:33.151846  7247 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:33.154521  7252 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.155252  7252 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.156661  1663 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 7. (term mismatch)
I20260709 13:03:33.159453  1663 pending_rounds.cc:85] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 6
I20260709 13:03:33.159682  1663 pending_rounds.cc:107] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:33.159899  1663 raft_consensus.cc:2889] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:33.160177  1663 pending_rounds.cc:107] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:03:33.160384  1663 raft_consensus.cc:2889] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:33.136965  6985 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.752s	user 0.006s	sys 0.007s
W20260709 13:03:33.136943  7167 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.126s	user 0.011s	sys 0.005s
W20260709 13:03:33.171169  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.170902  7277 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.176079  7277 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.179968  7252 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.140448  6485 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:33.182940  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.206032  1733 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:33.184213  7167 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.208565  1659 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:33.211285  7167 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.166131  7158 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.152901  7247 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:33.217375  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116272us
W20260709 13:03:33.196693  7273 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:03:33.130332  7264 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.163344  7266 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.200695  7253 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.223400  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:33.162565  7039 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 13:03:33.219460  7239 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.165195  6985 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.220316  7167 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:33.227150  7172 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.524s	user 0.014s	sys 0.000s
I20260709 13:03:33.224619  7277 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.244777  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:33.203822  6485 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:33.236157  7231 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.010s	sys 0.021s
I20260709 13:03:33.221985  7158 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.229749  7064 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.235620  7278 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:33.234949  7286 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:33.267158  7231 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.010s	sys 0.022s
I20260709 13:03:33.226095  7262 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:03:33.276561  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 905) took 1067 ms. Trace:
I20260709 13:03:33.282231  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.280982  1668 rpcz_store.cc:276] 0709 13:03:32.209006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.209232 (+   226us) service_pool.cc:224] Handling call
0709 13:03:32.249377 (+ 40145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.468241 (+218864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:33.276234 (+807993us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:33.276493 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":466908544}
I20260709 13:03:33.262856  7277 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.289691  7247 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:33.290086  7288 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:33.290683  7288 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:33.291704  7277 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.291736  7288 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.296164  1734 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:33.296680  7252 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.297549  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.268266  7172 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.236548  7266 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 13:03:33.309024  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:33.310313  1737 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:33.296041  6934 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.905s	user 0.006s	sys 0.009s
I20260709 13:03:33.270227  7279 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:33.320248  1665 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:33.249328  7167 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:33.336963  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.346254  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.348372  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.349874  7172 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.350422  6904 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:33.351053  7270 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.351600  7270 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.352257  7172 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.288700  7137 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 13:03:33.323179  7279 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.353639  7249 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.354269  7064 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:33.354347  7249 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.355113  7264 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:33.353263  7292 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:33.356169  7292 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:33.356710  7292 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:33.353740  7279 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:33.375145  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97128.3us
W20260709 13:03:33.384631  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.355139  7064 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:33.332032  6934 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.366343  6736 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.365418  6985 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.300457  7286 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.373306  7158 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:33.400094  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51886.4us
I20260709 13:03:33.419354  1741 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:33.400792  7064 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:33.432016  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:33.439198  7291 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:03:33.445672  7167 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.446849  7266 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:33.450405  7240 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.587s	user 0.008s	sys 0.008s
I20260709 13:03:33.452018  6934 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.454852  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.408850  6736 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.457953  6736 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.458796  7286 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:33.461066  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.463433  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.435256  7249 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:33.480944  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49095.8us
I20260709 13:03:33.487620  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.506098  7286 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.506800  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.510425  7286 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:33.514081  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 916) took 1246 ms. Trace:
I20260709 13:03:33.514564  1742 rpcz_store.cc:276] 0709 13:03:32.267296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.267455 (+   159us) service_pool.cc:224] Handling call
0709 13:03:32.309037 (+ 41582us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:32.942220 (+633183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:33.513802 (+571582us) spinlock_profiling.cc:243] Waited 632 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:33.513987 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1454740352}
I20260709 13:03:33.472304  7298 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:33.522646  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.497678  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 907) took 1215 ms. Trace:
I20260709 13:03:33.513898  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.509873  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:33.532141  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68871.6us
I20260709 13:03:33.529815  1659 rpcz_store.cc:276] 0709 13:03:32.282411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.282558 (+   147us) service_pool.cc:224] Handling call
0709 13:03:32.312853 (+ 30295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:33.209128 (+896275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:33.497167 (+288039us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54002f1a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:33.497558 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2059526400}
I20260709 13:03:33.502243  7235 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:33.538271  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.470140  7174 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:33.543650  7257 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.748s	user 0.004s	sys 0.000s
I20260709 13:03:33.478250  6934 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:33.544435  6934 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:33.544905  6934 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.544981  1586 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:33.546317  7235 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:33.546370  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:33.547170  1588 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.547501  7235 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.548944  7235 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.543813  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.550046  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.552758  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.549696  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.553736  1593 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:33.551541  1590 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:33.534192  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.558167  1592 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 13:03:33.560024  6736 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.560756  6736 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.558490  1585 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:33.563982  1589 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 10. Current term is 11. Ops: []
I20260709 13:03:33.565112  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.567781  7261 raft_consensus.cc:993] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:03:33.568084  7153 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:33.568765  7153 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:33.568310  7261 raft_consensus.cc:1081] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
W20260709 13:03:33.571928  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2137) took 1426 ms (client timeout 1671 ms). Trace:
I20260709 13:03:33.572753  7064 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.572947  7100 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:33.570062  7153 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.575652  7064 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:33.576526  7153 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:33.572404  1667 rpcz_store.cc:269] 0709 13:03:32.145844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.146028 (+   184us) service_pool.cc:224] Handling call
0709 13:03:33.571693 (+1425665us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:33.571828 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206648064}
I20260709 13:03:33.583544  7174 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.581822  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:33.585603  6485 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.586524  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.587567  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:33.588591  7239 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:33.588809  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.589588  1590 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:33.590245  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:33.586704  7244 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.590896  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:33.591257  7167 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:33.592104  7240 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:33.592446  7167 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:33.592815  7240 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:33.592960  7167 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.591274  7244 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.597050  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.598241  7167 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.599256  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.600869  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:33.593242  7240 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:33.601960  7286 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:03:33.602128  7240 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.597330  7244 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.605769  7103 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:33.606278  7248 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.606279  7103 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.607048  7103 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:33.608968  7237 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.609686  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.610147  7237 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.610345  2309 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.581102  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.611927  1735 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:33.583850  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.603344  7240 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:33.617465  1662 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:33.552102  7270 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.614841  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.626632  1663 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:33.627127  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.627923  1665 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:33.613948  1733 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:33.616494  1588 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:33.603622  7286 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:33.619180  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:33.630911  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.631565  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:33.637751  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 127959us
I20260709 13:03:33.638736  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.639096  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:33.639393  1665 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.638895  1663 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:33.638243  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:33.695961  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.728682  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.729188  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.730783  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:33.734160  1659 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:03:33.563251  6736 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.729790  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:33.749593  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.479549  7292 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.753376  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:33.745664  7100 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.755263  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.755509  2312 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 13:03:33.756028  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 221947us
I20260709 13:03:33.632619  7174 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:33.759536  7283 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:33.760610  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.761314  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:33.761652  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118620us
I20260709 13:03:33.784060  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:33.762530  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 143446us
I20260709 13:03:33.785256  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:33.786363  1659 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:33.770128  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.757005  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.768654  1665 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:33.780193  1588 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:46322
I20260709 13:03:33.770609  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:33.789376  1588 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:33.790069  1588 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:33.790907  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.791265  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.792068  2310 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:03:33.792440  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53958.5us
I20260709 13:03:33.793037  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:33.793262  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:33.740505  7244 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:33.801436  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:33.809558  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52697.4us
I20260709 13:03:33.809815  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.810089  2312 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:33.810511  1659 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.811057  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.811261  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.802994  7202 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.812282  7202 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.812981  7292 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:33.767851  7239 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:33.797617  7279 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.813884  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.814244  2310 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:33.815282  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.815390  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.792652  7039 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:03:33.816514  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.816888  7039 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:33.816951  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.803798  6736 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:33.827088  6736 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.827499  6736 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.828148  2309 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:33.829074  7202 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.831130  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:33.832180  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:33.817479  7039 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:33.762167  7178 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 13:03:33.817821  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:33.818090  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:33.819428  7247 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.529s	user 0.009s	sys 0.004s
I20260709 13:03:33.819283  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:33.820138  7252 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:33.813930  7239 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.820719  7172 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:33.798854  7137 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:33.812564  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:33.809166  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:33.791812  7100 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.767854  7249 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:33.887019  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:33.823102  1737 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:33.884155  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50386us
I20260709 13:03:33.826213  7290 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:33.903394  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:33.800209  7083 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.001s
I20260709 13:03:33.855356  2309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
W20260709 13:03:33.862229  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44525.8us
I20260709 13:03:33.820890  7244 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:33.880810  2312 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:33.882838  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:33.840304  6899 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:33.912801  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:33.870851  7174 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:33.838651  7288 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:33.904870  2309 raft_consensus.cc:606] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:33.912801  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.884825  1588 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:33.867046  7252 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:33.870280  7172 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.880925  7237 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:33.861708  7202 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
W20260709 13:03:33.930727  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85404.2us
I20260709 13:03:33.870337  7255 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 13:03:33.918900  7265 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:33.934417  1668 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:33.936316  7252 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:33.955654  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.956574  1668 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:33.957873  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:33.962812  7239 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:33.977244  1659 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:33.947691  7288 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:33.978901  7288 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:33.986172  7305 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20260709 13:03:33.993328  7309 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 13:03:33.996317  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 226288us
I20260709 13:03:33.997288  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:33.998227  7153 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:33.998737  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:33.999801  7153 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:33.939041  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 119880us
W20260709 13:03:33.912504  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:33.905403  7158 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:33.943456  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111355us
I20260709 13:03:33.950515  1588 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:33.920639  6899 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:33.913003  7266 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:03:33.933667  7303 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.456s	user 0.002s	sys 0.000s
I20260709 13:03:33.884685  7278 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:33.946848  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:33.955260  7137 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:03:34.031010  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.030619  2316 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:03:34.028245  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82039.8us
I20260709 13:03:33.940078  7172 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:34.222302  7103 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.003s
W20260709 13:03:34.054641  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 208754us
I20260709 13:03:34.108692  7167 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.238073  7167 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.000996  7265 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:34.239482  7265 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:33.937651  7174 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:34.242731  7174 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 14.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:34.243530  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2157) took 1317 ms (client timeout 1633 ms). Trace:
W20260709 13:03:34.244019  1662 rpcz_store.cc:269] 0709 13:03:32.926200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.970530 (+ 44330us) service_pool.cc:224] Handling call
0709 13:03:34.243327 (+1272797us) spinlock_profiling.cc:243] Waited 453 ms on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:34.243459 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1042964736}
W20260709 13:03:34.079093  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 268111us
I20260709 13:03:33.954260  7237 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:33.955495  7064 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.142s
I20260709 13:03:34.246676  7064 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:34.246582  7237 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.186828  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:34.058467  6899 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.249924  1665 raft_consensus.cc:2368] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:34.250084  1661 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:03:34.250968  7315 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:34.252465  7178 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:34.254469  7282 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:34.254889  7282 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:34.255395  7304 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.508s	user 0.001s	sys 0.000s
I20260709 13:03:34.255798  7282 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.185881  7100 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.199442  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:34.258183  6736 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:34.258980  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60070.2us
I20260709 13:03:34.260367  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:34.260810  6736 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.261374  1584 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 13:03:34.257246  7282 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:34.103955  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 226327us
W20260709 13:03:34.203732  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 256991us
I20260709 13:03:34.199198  7158 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:34.199435  7250 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:34.220037  2316 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:34.220440  6956 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:34.000320  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:34.221197  7239 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.232808  6485 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:34.220708  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:34.235535  6934 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.000s
I20260709 13:03:34.247215  7064 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:33.929682  7244 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.264804  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:34.271446  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:34.330430  2313 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:34.272934  2318 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:34.268281  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:34.296322  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 75743.2us
I20260709 13:03:34.330946  2313 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:34.261479  6736 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:34.337229  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.263599  7279 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.286864  7278 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:34.266539  7295 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:03:34.349043  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 348802us
I20260709 13:03:34.348055  1742 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
I20260709 13:03:34.350543  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:34.350827  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:34.351680  1663 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:34.245328  7172 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.325881  7292 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:34.338354  6956 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:34.295964  6485 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:34.348043  7248 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.734s	user 0.007s	sys 0.007s
I20260709 13:03:34.358162  2313 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.381848  2313 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:34.340493  7258 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:34.269102  7100 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 13:03:34.269930  7167 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 13:03:34.333142  6934 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:34.381808  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 117122us
I20260709 13:03:34.295460  7158 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.389834  1666 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 13:03:34.406198  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:34.262320  6899 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.376343  7207 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:34.396317  7240 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.769s	user 0.009s	sys 0.004s
I20260709 13:03:34.379462  7292 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:34.396968  7172 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.397872  7100 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:03:34.399842  7235 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.850s	user 0.013s	sys 0.000s
I20260709 13:03:34.402098  7278 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.402226  6485 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.327013  7039 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.351466  7317 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:34.306231  7250 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.326136  7239 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.349231  7295 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:34.418089  7207 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.406754  1666 pending_rounds.cc:85] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 12
I20260709 13:03:34.457901  1666 pending_rounds.cc:107] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 13:03:34.458400  1666 raft_consensus.cc:2889] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:34.395694  6956 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.405130  7255 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:34.390096  7083 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:34.421775  1736 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:34.426111  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 920) took 2003 ms. Trace:
I20260709 13:03:34.468482  1741 rpcz_store.cc:276] 0709 13:03:32.423095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.423241 (+   146us) service_pool.cc:224] Handling call
0709 13:03:32.436782 (+ 13541us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:33.426079 (+989297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:34.425711 (+999632us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b5400284e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:34.425993 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2259224960}
I20260709 13:03:34.376449  7259 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
W20260709 13:03:34.430799  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 162603us
I20260709 13:03:34.419173  7292 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:34.420101  7100 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:34.418033  7279 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:03:34.439069  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 179377us
I20260709 13:03:34.292122  7103 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.465809  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.465668  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.473325  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:34.478652  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:34.480000  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:34.459470  7247 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.488736  1740 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:34.435596  7317 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 13:03:34.501453  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:34.468010  7249 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.002s
I20260709 13:03:34.506928  1741 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:34.465688  7255 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:34.418533  7064 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:34.508030  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83180.2us
W20260709 13:03:34.485716  7252 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.003s
I20260709 13:03:34.520937  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:34.525370  6934 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:34.510387  7317 consensus_queue.cc:1243] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:34.514160  7249 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:34.510906  1708 connection.cc:256] RPC call timeout handler was delayed by -1.2224853010000061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
I20260709 13:03:34.530823  6899 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:34.519255  7258 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:34.532435  7158 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.532105  7258 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.532670  7250 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.506717  7270 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:34.533315  7270 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.533593  7295 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.534327  7270 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:34.534938  7295 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.535506  7258 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:34.489487  7237 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:34.512964  7247 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.538463  2317 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:34.539826  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.541972  7244 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:34.542954  7064 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.544130  7207 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:34.527554  7039 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.516589  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:03:34.565178  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.506935  7287 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:34.531406  7103 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:34.605661  7324 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.076s	user 0.003s	sys 0.000s
I20260709 13:03:34.557313  7290 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.105s
I20260709 13:03:34.606031  7100 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.522096  7255 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:34.557648  7252 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:34.606896  7319 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.095s	user 0.002s	sys 0.001s
I20260709 13:03:34.677183  7239 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.512760  7279 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:34.678741  7279 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.684283  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 519) took 1410 ms. Trace:
I20260709 13:03:34.684890  2311 rpcz_store.cc:276] 0709 13:03:33.273872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.274075 (+   203us) service_pool.cc:224] Handling call
0709 13:03:33.287106 (+ 13031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:33.287654 (+   548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:33.287699 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:33.314570 (+ 26871us) log.cc:844] Serialized 26 byte log entry
0709 13:03:34.266909 (+952339us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b18000e6c20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:34.479130 (+212221us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:34.479171 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:34.479336 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:34.683834 (+204498us) raft_consensus.cc:1685] finished
0709 13:03:34.683869 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:34.684148 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7348,"raft.queue_time_us":117,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":12667,"spinlock_wait_cycles":21919616,"thread_start_us":212022,"threads_started":1,"wal-append.queue_time_us":1367006}
W20260709 13:03:34.606711  7323 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.070s	user 0.003s	sys 0.000s
I20260709 13:03:34.669241  7100 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:34.701193  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 187684us
I20260709 13:03:34.703382  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:34.722157  7244 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.723121  7244 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.731824  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.737169  1737 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:34.738762  1741 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:34.701066  7290 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:34.749682  2313 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:34.750187  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:34.730073  7172 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:34.730796  7287 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.733722  7250 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:34.773319  1661 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:34.747105  7158 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.750070  7332 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.773332  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:34.776559  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:34.820396  1660 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:34.776126  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:34.781973  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:34.822718  1664 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:34.779065  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:34.821918  1662 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:34.759001  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 422225us
I20260709 13:03:34.774108  1738 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:34.826771  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:34.827585  1738 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:34.751179  7239 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.818029  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:34.833300  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:34.823786  1668 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:03:34.820406  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47177.1us
I20260709 13:03:34.837399  1742 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:34.837462  1741 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:34.756675  7172 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.755252  7321 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:34.851981  7321 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:34.843556  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 381772us
W20260709 13:03:34.854703  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.855034  7321 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.856371  6934 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:03:34.859128  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 96750.4us
I20260709 13:03:34.860206  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:34.815282  7250 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:34.765357  7244 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:34.761600  7279 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:34.767706  7240 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.865409  1660 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:03:34.790491  7290 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:34.868297  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:34.846839  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:34.823235  7288 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:34.887320  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:34.888691  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:34.840484  7174 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:34.863189  7172 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.891987  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.892486  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:34.900703  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:34.904779  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 383932us
I20260709 13:03:34.901319  1738 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:34.901433  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:34.897702  1662 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:03:34.897708  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 292774us
W20260709 13:03:34.909518  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:34.902567  1708 leader_election.cc:336] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.481s (SENT)
I20260709 13:03:34.910647  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:34.822057  7158 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.873973  7250 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:34.887974  7244 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:34.915786  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:34.786484  7287 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.891580  7279 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:34.899138  7248 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:34.915346  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:34.937970  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:34.867513  7103 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 13:03:34.903975  7288 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:34.847437  7239 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.007673  7239 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.904403  7321 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:34.941179  7320 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.222s	user 0.004s	sys 0.006s
I20260709 13:03:34.943554  7283 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:34.944779  7330 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:34.945930  7280 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.785s	user 0.013s	sys 0.007s
I20260709 13:03:35.022910  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.023895  1662 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:35.025559  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 132511us
I20260709 13:03:34.949599  7317 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:34.952766  7290 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.956375  7237 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:34.957203  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:34.957986  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:34.958498  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.035660  1592 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:34.958763  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:34.960897  7313 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.685s	user 0.006s	sys 0.013s
I20260709 13:03:34.963327  7178 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.965067  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.038497  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:34.966110  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.039201  1589 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:34.967099  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.039211  1667 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.040040  1587 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:34.967612  1742 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:34.969398  7235 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.984872  7035 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:34.993484  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:34.994272  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.051968  1593 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:34.994987  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.053853  1741 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:35.054903  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.050622  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:34.996588  7252 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.997829  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:34.920312  6956 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:35.050341  7103 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.028951  7239 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:34.999308  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.032605  7283 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:35.002513  2316 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.999251  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2851) took 1249 ms. Trace:
I20260709 13:03:35.075520  2316 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.003883  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.005015  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:35.016922  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:34.989209  7292 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:34.925037  7266 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:34.832593  7332 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.078111  7266 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.078653  7266 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.080096  7332 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:34.912174  7174 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.082077  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.082526  7174 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.013290  7279 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:35.083571  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.083724  7279 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.054503  7178 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.020267  7167 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:35.029757  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.085951  1733 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:35.085811  7167 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:34.891286  7240 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.086851  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2856) took 1496 ms. Trace:
I20260709 13:03:35.087371  7178 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.087239  1736 rpcz_store.cc:276] 0709 13:03:33.590184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.598939 (+  8755us) service_pool.cc:224] Handling call
0709 13:03:35.086641 (+1487702us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b54000e2ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:35.086769 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1858889088}
I20260709 13:03:35.088055  7167 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.088832  1664 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:35.089869  7247 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.092346  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.093348  7287 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.054658  7235 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.059813  1586 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:35.094578  7247 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:35.096235  7247 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.096696  7235 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:35.096813  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.098188  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.098470  7247 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.098780  7158 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.075855  1665 rpcz_store.cc:276] 0709 13:03:33.750452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.801030 (+ 50578us) service_pool.cc:224] Handling call
0709 13:03:34.999041 (+1198011us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b540013e2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:34.999187 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1028387712}
I20260709 13:03:35.072049  7103 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.084384  1590 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:35.101181  1591 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:35.051065  7255 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.040933  7317 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:35.106153  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.109515  1593 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:35.050587  7270 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.516s	user 0.007s	sys 0.006s
I20260709 13:03:35.115903  1662 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:35.116994  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.117053  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.118481  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.121131  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.121755  1660 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.033167  7330 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.088332  7240 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.149885  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.151147  1593 raft_consensus.cc:2410] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
W20260709 13:03:35.147948  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 247353us
I20260709 13:03:35.153388  1592 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:03:35.125262  7295 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.590s	user 0.008s	sys 0.005s
I20260709 13:03:35.058693  7035 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.166440  1662 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:35.167567  2312 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:35.162689  1665 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:35.169370  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.170917  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.173588  1593 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:35.175041  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 237197us
I20260709 13:03:35.176105  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:35.174721  1586 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:35.182545  6956 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.177639  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.183112  6956 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.184067  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.184686  1592 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:35.178188  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:35.186159  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.186417  6956 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:35.186985  1592 raft_consensus.cc:2368] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:35.187937  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.188539  1592 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:35.151211  1667 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:35.190919  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.191555  1741 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:35.178498  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.194820  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.196431  7158 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:35.197012  7158 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.197381  7158 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:35.198182  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.198645  7172 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.200755  7172 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.203917  1665 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:35.206149  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.206837  1668 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:35.176954  7240 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.208235  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.212141  2313 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:35.216183  1593 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:35.115151  7239 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:35.220202  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 239254us
I20260709 13:03:35.221484  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.222060  1668 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:03:35.222312  1741 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:35.225490  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.174947  7103 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.230403  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.231432  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.235469  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.133366  7270 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.226342  2317 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
W20260709 13:03:35.237972  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.238309  1662 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:35.227821  1592 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:35.153779  7288 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.242743  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:35.237936  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 219061us
I20260709 13:03:35.240671  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.233918  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.248767  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.243484  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:35.250887  2310 raft_consensus.cc:606] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:35.251283  2310 raft_consensus.cc:614] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:35.243760  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.239568  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.252242  2313 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:35.252916  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.255899  1662 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [15.8-17.9]
I20260709 13:03:35.228214  6956 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.259443  2318 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:03:35.262605  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.269613  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 914) took 1943 ms. Trace:
I20260709 13:03:35.270089  1666 rpcz_store.cc:276] 0709 13:03:33.326447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.326676 (+   229us) service_pool.cc:224] Handling call
0709 13:03:33.345936 (+ 19260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:34.459108 (+1113172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.269257 (+810149us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:35.269538 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2389278464}
I20260709 13:03:35.244904  7233 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 13:03:35.239595  7349 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:35.279060  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.288270  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.300781  2317 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:35.270960  7158 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.294718  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.302184  1664 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:35.288738  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.289223  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.283496  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.249619  7298 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:35.309544  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.310210  1592 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:35.311393  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.295140  2316 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:35.304059  1741 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
W20260709 13:03:35.312839  2316 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1032) took 1799 ms (client timeout 1558 ms). Trace:
I20260709 13:03:35.306785  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.313696  1590 raft_consensus.cc:2376] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15. Re-sending same reply.
I20260709 13:03:35.315243  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:35.322249  1590 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:35.261891  7100 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:35.257169  7353 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.228428  7283 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.328192  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.330861  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.336005  1662 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:35.336162  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:35.313390  2316 rpcz_store.cc:269] 0709 13:03:33.531615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:33.557018 (+ 25403us) service_pool.cc:224] Handling call
0709 13:03:34.030412 (+473394us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54005aec20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0e18 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:35.002061 (+971649us) spinlock_profiling.cc:243] Waited 77.5 ms on lock 0x7b54005aec20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf5b06f 00007fcdedfd3f60 00007fcdedfd57de 00007fcdedfe127c 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2
0709 13:03:35.312756 (+310695us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547759232}
I20260709 13:03:35.228293  7317 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.342453  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.311831  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:35.280925  7035 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.332763  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.282235  7239 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.290289  7330 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:35.341538  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:35.277463  7336 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.199s	user 0.003s	sys 0.000s
I20260709 13:03:35.256954  7266 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.344897  1668 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:35.343127  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:35.347704  1662 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:35.349210  2313 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:35.350646  1587 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:35.353188  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.331966  7235 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:35.355720  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.358556  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.363099  1742 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:35.340783  7233 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.360235  1588 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:35.335126  7064 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:35.286679  6956 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.359853  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.364401  1660 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:35.364934  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.365403  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.349920  7318 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:35.374122  1742 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:35.370702  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:35.375247  1742 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:35.378832  1590 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:35.379916  1741 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:35.380649  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 145966us
I20260709 13:03:35.345954  7287 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:03:35.308748  7349 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.376462  1742 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.385648  1740 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 13:03:35.389026  1740 pending_rounds.cc:85] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 12
I20260709 13:03:35.389150  1742 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.389303  1740 pending_rounds.cc:107] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 13:03:35.389729  1740 raft_consensus.cc:2889] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:35.390662  1737 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:35.374579  7292 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:35.380882  7255 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.409298  2313 raft_consensus.cc:606] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:35.366043  7235 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:35.398031  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 145211us
I20260709 13:03:35.407452  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.408881  1742 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:03:35.409832  2313 raft_consensus.cc:614] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:35.374816  7318 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:35.412477  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:35.374151  7064 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.417776  1708 leader_election.cc:400] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:35.418267  1708 leader_election.cc:403] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 13:03:35.419534  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.420209  1660 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:35.420691  7137 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:35.421134  7137 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:35.422039  7137 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.423358  7137 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:35.426132  7158 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.430258  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.409400  7266 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.410888  7247 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.433857  7270 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.435588  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:35.437834  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.432026  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.427395  6956 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:35.416226  7318 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:35.433391  7361 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:35.411551  7250 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.447751  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.467886  1667 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:35.433271  7298 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:35.436321  7356 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 13:03:35.428685  7292 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:35.449923  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.450911  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.495620  1666 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:35.453922  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.451984  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:35.496264  1664 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:35.496441  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44554us
I20260709 13:03:35.458014  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.460763  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.469856  7278 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:35.554019  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 93431.7us
I20260709 13:03:35.475083  1665 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:35.476974  1662 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 13:03:35.545236  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.558843  1667 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:35.494392  7292 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:35.478981  6956 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:03:35.546941  7153 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:35.495797  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:35.553052  6485 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:35.553122  7282 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:35.554075  7288 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:35.554648  7278 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.456152  7262 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:35.561677  7352 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:35.562706  7064 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.563457  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.570328  1742 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 12. Current term is 13. Ops: []
W20260709 13:03:35.573947  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78277.2us
I20260709 13:03:35.575218  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.599596  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.571491  7292 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:35.607241  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43858.4us
I20260709 13:03:35.605131  1663 raft_consensus.cc:2376] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15. Re-sending same reply.
I20260709 13:03:35.638756  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.609339  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:35.634948  7352 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.646945  7349 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.649775  7282 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:35.652671  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:35.651405  7352 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.660056  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.594353  7158 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:35.661927  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:35.667565  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.668668  1591 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:35.670929  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:35.672521  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:35.678078  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:35.705885  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 96649.3us
I20260709 13:03:35.594575  6934 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:35.709870  1591 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:03:35.636574  7153 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:35.710564  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:35.641247  7262 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:35.645116  7235 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:35.723140  1664 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
W20260709 13:03:35.725184  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:35.745594  1585 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:35.746412  7153 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.608417  7288 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.751873  7288 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.753131  7262 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.755018  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1038) took 1266 ms. Trace:
I20260709 13:03:35.755532  2318 rpcz_store.cc:276] 0709 13:03:34.488370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.489289 (+   919us) service_pool.cc:224] Handling call
0709 13:03:34.515410 (+ 26121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.259853 (+744443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.754687 (+494834us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:35.754946 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1710650752}
I20260709 13:03:35.707429  7158 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.766848  7158 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.767721  7103 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:35.770963  7278 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.724038  7259 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 13:03:35.776616  7270 raft_consensus.cc:993] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:03:35.727787  6956 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:35.719954  7282 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:03:35.720049  7352 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:35.824239  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:35.825263  1591 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:35.786897  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62070.7us
I20260709 13:03:35.796999  1584 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:03:35.715448  7349 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:35.849552  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:35.718995  7064 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:35.718480  7283 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:03:35.777686  7244 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.842s	user 0.010s	sys 0.006s
W20260709 13:03:35.788936  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58121.5us
I20260709 13:03:35.863341  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:35.865258  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:35.855499  1586 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.557s	user 0.019s	sys 0.003s
I20260709 13:03:35.874392  1586 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:35.779757  7282 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:35.716209  6934 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:35.777181  7270 raft_consensus.cc:1081] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:03:35.858928  7349 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.819481  7240 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.904953  1664 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:35.838243  6956 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:35.866035  7250 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:35.912135  7011 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:35.861763  7295 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:35.915899  1668 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 13:03:35.969686  1668 pending_rounds.cc:85] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:35.903484  7282 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:35.780370  7290 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.123s
I20260709 13:03:35.777904  7279 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.127s
I20260709 13:03:35.901055  7244 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:35.981141  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 347767us
I20260709 13:03:35.897271  7039 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:35.900398  7248 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:35.898077  7137 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:35.967686  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:35.970080  1668 pending_rounds.cc:107] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:03:35.860304  7064 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:35.971390  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.780296  7352 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:35.838737  7265 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.069s
I20260709 13:03:35.920898  6934 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.001283  7250 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:36.001547  7355 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.001924  7250 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.002370  7355 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:35.926498  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:36.006678  1665 raft_consensus.cc:2376] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13. Re-sending same reply.
I20260709 13:03:35.996321  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:35.923368  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:35.999197  1668 raft_consensus.cc:2889] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:36.008517  1664 raft_consensus.cc:2368] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:36.009477  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:36.010519  1664 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:35.963512  7035 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:36.013401  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:36.014249  1667 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:35.917706  7349 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:36.007798  1741 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:36.016664  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.016908  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:36.018512  1665 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:35.976938  7259 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.006222  7361 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:35.960240  7240 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.000700  7064 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.008972  7279 consensus_queue.cc:1243] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:36.012025  7239 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.067487  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:36.037909  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.073618  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.075016  1740 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:36.080765  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:36.081740  1667 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:36.084010  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:36.033793  7235 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.007032  7248 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.098434  1659 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:36.095184  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:36.008657  7137 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.072062  7158 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.029721  7259 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.109066  1667 raft_consensus.cc:2376] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16. Re-sending same reply.
W20260709 13:03:36.105201  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 232508us
W20260709 13:03:36.112668  2316 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.016s	sys 0.008s
W20260709 13:03:36.113055  2316 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.016s	sys 0.008s
I20260709 13:03:36.113444  1586 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:46322
I20260709 13:03:36.114225  1586 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:36.114665  1586 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.087139  7239 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:36.126410  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 261992us
W20260709 13:03:36.122385  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 264696us
W20260709 13:03:36.125967  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98864.1us
I20260709 13:03:36.140961  1705 leader_election.cc:400] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12.
I20260709 13:03:36.122113  7283 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:36.148439  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 280947us
I20260709 13:03:36.152197  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.153218  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.099309  7282 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.104285  7158 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.123584  7235 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:03:36.142212  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.103545  6956 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.159706  1586 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.175835  1586 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.179791  7259 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:36.183416  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:36.187762  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.172418  7240 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.176419  7361 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:36.197064  7344 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:36.209060  1740 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:36.198992  7290 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:36.212719  7290 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.213166  7290 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:36.211076  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.211200  7137 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.209110  7248 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.192101  7282 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.196158  7039 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.197793  7259 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:36.204201  7361 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.192310  7265 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 13:03:36.200968  7153 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:36.205930  1664 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:36.206195  7344 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.352826  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.381310  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.391513  1740 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:36.386343  1741 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:36.394526  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:36.396373  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:36.350989  7361 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:36.401429  2317 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 13:03:36.231235  7259 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:36.410924  1587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.022s	sys 0.070s
I20260709 13:03:36.403430  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:36.411489  1587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.822s	user 0.022s	sys 0.070s
I20260709 13:03:36.402587  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:36.406736  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:36.409559  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 256395us
I20260709 13:03:36.412175  2318 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 13:03:36.395792  7318 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.918s	user 0.008s	sys 0.009s
I20260709 13:03:36.383486  7344 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.412524  2318 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:36.396291  7248 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.351925  7153 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.412858  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:36.417402  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 229906us
I20260709 13:03:36.419909  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3003) took 1248 ms. Trace:
I20260709 13:03:36.396260  7240 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:36.420431  1587 rpcz_store.cc:276] 0709 13:03:35.171190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.363885 (+192695us) service_pool.cc:224] Handling call
0709 13:03:35.377396 (+ 13511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:35.377655 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:35.589326 (+211671us) log.cc:844] Serialized 14 byte log entry
0709 13:03:36.419561 (+830235us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:36.419620 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:36.419666 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.419734 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.419852 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9068,"prepare.queue_time_us":33622,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":1005408,"threads_started":2}
I20260709 13:03:36.405860  7381 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:36.408831  7382 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.347977  7039 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.422086  2318 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:36.419560  7367 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.761s	user 0.006s	sys 0.002s
I20260709 13:03:36.430605  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:36.414928  7352 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.431334  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.413620  7259 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.436131  2318 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.416095  7158 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.440809  1734 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:36.423512  7153 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.424748  7137 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.424902  7235 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:36.354176  7283 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.418974  7318 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.456563  1735 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 13:03:36.462744  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.460316  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:36.463995  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.468726  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.470259  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.426553  7248 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.463544  7235 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:36.490703  7377 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:36.494892  7039 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:36.479512  6485 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.888s	user 0.014s	sys 0.000s
I20260709 13:03:36.498538  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:36.498584  1734 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:36.431860  7240 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.543690  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.541326  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.466002  7352 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:36.544631  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:36.511787  7292 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.838s	user 0.011s	sys 0.001s
I20260709 13:03:36.522456  7355 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.504873  7380 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:03:36.496949  7385 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:36.471751  7158 raft_consensus.cc:2764] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:03:36.465129  7137 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.526774  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2178) took 1166 ms. Trace:
I20260709 13:03:36.539753  7262 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:36.527949  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2873) took 1304 ms. Trace:
I20260709 13:03:36.529966  7389 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:36.522866  6934 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.004s
I20260709 13:03:36.542995  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.543412  7326 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:36.611933  7326 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:36.605027  7262 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:36.613031  7326 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.614362  7326 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:36.614362  7262 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.615670  7262 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:36.615796  1664 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:03:36.617413  7389 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.618712  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.619954  1593 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:36.620715  7361 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.621004  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.544492  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:36.547890  7247 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:36.630143  7247 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.630582  7247 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.553193  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.632468  7318 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.632679  1585 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:36.633898  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:36.551481  7248 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.635048  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.635707  1585 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:36.547184  7340 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.467s	user 0.005s	sys 0.000s
I20260709 13:03:36.566596  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:59800 (request call id 25) took 1222 ms. Trace:
I20260709 13:03:36.437688  7382 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.636695  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:36.657166  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:36.584873  1663 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:03:36.576309  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78028.8us
I20260709 13:03:36.659461  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.660372  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:36.662122  1590 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:36.589383  2318 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:36.590473  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.520875  7259 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:36.554461  7386 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 13:03:36.528787  7035 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:36.600716  1662 rpcz_store.cc:276] 0709 13:03:35.360821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.362981 (+  2160us) service_pool.cc:224] Handling call
0709 13:03:35.366741 (+  3760us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.558178 (+191437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.526449 (+968271us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400117e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:36.526700 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252861696}
I20260709 13:03:36.527230  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 926) took 1323 ms. Trace:
I20260709 13:03:36.487831  7378 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.106s
W20260709 13:03:36.601675  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 131499us
I20260709 13:03:36.671271  1661 rpcz_store.cc:276] 0709 13:03:35.203734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.205417 (+  1683us) service_pool.cc:224] Handling call
0709 13:03:35.205725 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:35.209156 (+  3431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:35.209192 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:35.209234 (+    42us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:35.209268 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.526999 (+1317731us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b54002f1f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:36.527060 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.527174 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7426944}
I20260709 13:03:36.675591  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:36.583122  7287 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.584399  7255 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:36.586508  7244 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:36.540818  7388 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.605115  1737 rpcz_store.cc:276] 0709 13:03:35.223863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.352869 (+129006us) service_pool.cc:224] Handling call
0709 13:03:36.527779 (+1174910us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:36.527889 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84330880}
W20260709 13:03:36.606835  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 186185us
I20260709 13:03:36.608016  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 528) took 1052 ms. Trace:
I20260709 13:03:36.596508  7385 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:36.854260  7385 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:36.854954  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:36.856308  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:36.840356  2316 rpcz_store.cc:276] 0709 13:03:35.555584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.555759 (+   175us) service_pool.cc:224] Handling call
0709 13:03:35.563610 (+  7851us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:35.563826 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:36.113429 (+549603us) log.cc:844] Serialized 14 byte log entry
0709 13:03:36.606901 (+493472us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:36.606969 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:36.607699 (+   730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.607770 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.607952 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27500,"prepare.queue_time_us":611516,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":6499,"raft.queue_time_us":47466,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7746,"thread_start_us":1042156,"threads_started":3,"wal-append.queue_time_us":326644}
I20260709 13:03:36.857491  7388 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.860345  7244 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:36.821385  7290 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.608s	user 0.014s	sys 0.001s
I20260709 13:03:36.862511  7290 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.611418  7137 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.542227  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.867906  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:36.622123  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 2915) took 1666 ms (client timeout 2221 ms). Trace:
I20260709 13:03:36.592404  7292 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.626063  7158 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.627777  6934 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.638139  1593 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:36.638365  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 525) took 1449 ms. Trace:
I20260709 13:03:36.636574  2313 rpcz_store.cc:276] 0709 13:03:35.343848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.343978 (+   130us) service_pool.cc:224] Handling call
0709 13:03:36.566303 (+1222325us) spinlock_profiling.cc:243] Waited 56 ms on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfd7853 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:36.566477 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128745728}
I20260709 13:03:36.570132  7235 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:36.641480  7278 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:03:36.641949  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.09791e+06us
I20260709 13:03:36.644771  1585 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:37.033872  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.034441  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:37.035110  1585 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:03:36.648375  7064 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.555s	user 0.013s	sys 0.000s
I20260709 13:03:36.592116  7153 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:36.652771  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:36.544615  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.657537  1659 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.673905  7361 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.046136  7355 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:37.033294  7249 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:36.673583  7318 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:37.047158  7064 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.804252  7344 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.047866  7249 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.049793  7344 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:36.664985  7330 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.000s	user 0.005s	sys 0.000s
W20260709 13:03:37.050520  7330 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.406s	user 0.007s	sys 0.004s
W20260709 13:03:36.810360  7349 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.005s
I20260709 13:03:36.675422  7035 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:36.807093  7240 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:36.811362  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 135956us
I20260709 13:03:37.065876  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:37.067289  1632 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:36.809739  7287 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:36.813395  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:36.814556  7248 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:36.817054  7365 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.093s	user 0.006s	sys 0.000s
I20260709 13:03:36.818190  7389 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.818393  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3008) took 1638 ms. Trace:
I20260709 13:03:37.060945  7408 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.086465  1659 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:36.835031  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:36.828351  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2228) took 2122 ms. Trace:
I20260709 13:03:36.841029  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:36.973819  7207 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.997458  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2880) took 1032 ms. Trace:
I20260709 13:03:36.983829  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:36.982553  7372 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 1.327s	user 0.003s	sys 0.002s
I20260709 13:03:37.007061  7239 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:36.868618  1588 rpcz_store.cc:269] 0709 13:03:34.955832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.956907 (+  1075us) service_pool.cc:224] Handling call
0709 13:03:36.621919 (+1665012us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:36.622064 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748059520}
I20260709 13:03:37.098573  1664 rpcz_store.cc:276] 0709 13:03:35.965416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.616703 (+651287us) service_pool.cc:224] Handling call
0709 13:03:36.617046 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:36.658651 (+ 41605us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:36.658799 (+   148us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:36.686406 (+ 27607us) log.cc:844] Serialized 26 byte log entry
0709 13:03:36.994329 (+307923us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:36.994376 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.994477 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:36.997142 (+  2665us) raft_consensus.cc:1685] finished
0709 13:03:36.997165 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.997364 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41053,"raft.queue_time_us":41192,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":15569,"thread_start_us":184266,"threads_started":1,"wal-append.queue_time_us":184768}
I20260709 13:03:37.008358  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2872) took 1078 ms. Trace:
I20260709 13:03:37.008677  1668 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:37.100668  1667 rpcz_store.cc:276] 0709 13:03:35.930067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.083369 (+153302us) service_pool.cc:224] Handling call
0709 13:03:37.008148 (+924779us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:37.008296 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55796608}
W20260709 13:03:37.009034  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 145967us
I20260709 13:03:36.964615  7283 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:36.964500  7395 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:36.961846  7397 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:36.974321  7290 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:36.823555  7247 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:36.991637  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:36.977339  7385 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.010763  1740 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:37.138024  7235 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:37.147773  7235 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.016000  7388 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:36.971715  1586 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:36.969822  1741 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:37.019598  7244 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.026504  2311 rpcz_store.cc:276] 0709 13:03:35.188549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.188693 (+   144us) service_pool.cc:224] Handling call
0709 13:03:35.739471 (+550778us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.740002 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:35.740044 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:35.740751 (+   707us) log.cc:844] Serialized 26 byte log entry
0709 13:03:35.958765 (+218014us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:35.958947 (+   182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.959537 (+   590us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:36.210605 (+251068us) raft_consensus.cc:1685] finished
0709 13:03:36.210637 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.637824 (+427187us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54006004d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:36.638288 (+   464us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4012,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"spinlock_wait_cycles":1265771776,"thread_start_us":216289,"threads_started":1,"wal-append.queue_time_us":385739}
I20260709 13:03:37.029601  1660 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:37.029691  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:37.034660  1662 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:36.647640  7382 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.035020  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:37.036022  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.166306  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:36.676018  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 263289us
I20260709 13:03:37.168289  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:37.036454  7158 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.046861  7355 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:36.809396  7259 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:36.723258  7250 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.052728  1742 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:37.054893  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:37.205448  1590 raft_consensus.cc:2376] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14. Re-sending same reply.
I20260709 13:03:37.013245  7292 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.207034  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:37.207722  1666 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:37.067992  7349 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.069439  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.083767  1592 rpcz_store.cc:276] 0709 13:03:35.180422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.573585 (+393163us) service_pool.cc:224] Handling call
0709 13:03:35.581116 (+  7531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:35.769999 (+188883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:35.770036 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:35.770086 (+    50us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:35.770121 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:36.818033 (+1047912us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400036bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:36.818115 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.818311 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":433862528}
I20260709 13:03:36.675422  7381 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:37.038848  7266 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.033489  7278 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:37.122117  1584 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:37.122802  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.216531  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.217465  1734 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:37.124115  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.136304  7249 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.137961  7239 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:37.140306  7399 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.180204  7378 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:37.269313  2318 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:37.140551  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.141232  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.141880  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.090500  1739 rpcz_store.cc:276] 0709 13:03:34.705616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:34.705816 (+   200us) service_pool.cc:224] Handling call
0709 13:03:34.706210 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:34.722346 (+ 16136us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:34.722384 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:34.723081 (+   697us) log.cc:844] Serialized 26 byte log entry
0709 13:03:34.908317 (+185236us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:34.908364 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:34.908440 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:36.546277 (+1637837us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:36.827795 (+281518us) raft_consensus.cc:1685] finished
0709 13:03:36.827831 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:36.828216 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22960,"raft.queue_time_us":17942,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":7510,"spinlock_wait_cycles":30490496,"thread_start_us":174133,"threads_started":1,"wal-append.queue_time_us":327125}
I20260709 13:03:37.014219  1589 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:03:37.141872  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:37.272460  1659 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:37.151161  2316 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 13:03:37.153326  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 161783us
I20260709 13:03:37.155515  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2875) took 1212 ms. Trace:
I20260709 13:03:37.274976  1668 rpcz_store.cc:276] 0709 13:03:35.943367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.586300 (+642933us) service_pool.cc:224] Handling call
0709 13:03:37.008302 (+422002us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.100737 (+ 92435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.135213 (+ 34476us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400117950. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:37.155448 (+ 20235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939405056}
I20260709 13:03:37.162604  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.276876  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:37.085736  7326 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.170593  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:37.118000  7389 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.300529  1667 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:37.098402  7064 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.145146  7385 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.166522  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.098860  7396 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:37.170622  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:37.326436  7235 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.180022  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.327522  7100 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.327736  7039 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:37.074621  7406 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:37.328255  7100 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.329295  7406 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:37.084661  7240 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 13:03:37.195319  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:37.331707  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 136532us
I20260709 13:03:37.160383  7344 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:37.214219  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 144893us
I20260709 13:03:37.334544  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:37.213474  7250 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.099022  7248 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.131451  7137 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:03:37.178639  7282 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:37.277189  2317 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:03:37.302361  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 130) took 12361 ms (client timeout 15000 ms). Trace:
I20260709 13:03:37.121889  7207 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:37.308240  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 141830us
I20260709 13:03:37.203557  7290 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 13:03:37.353080  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.354977  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.270144  6485 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:37.323015  7388 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:37.336391  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:37.358667  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2871) took 2142 ms. Trace:
I20260709 13:03:37.300014  7270 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.221944  1665 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
W20260709 13:03:37.326920  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 156381us
I20260709 13:03:37.366310  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.338363  1663 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 13:03:37.327553  1741 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:37.340343  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:37.303550  7378 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:37.157640  7397 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.359263  1738 rpcz_store.cc:276] 0709 13:03:35.216127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.372288 (+156161us) service_pool.cc:224] Handling call
0709 13:03:35.372646 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:35.379262 (+  6616us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:35.379294 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:35.386503 (+  7209us) log.cc:844] Serialized 26 byte log entry
0709 13:03:35.436584 (+ 50081us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:35.469593 (+ 33009us) log.cc:844] Serialized 14 byte log entry
0709 13:03:35.487392 (+ 17799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:35.487521 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:36.816412 (+1328891us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:37.346919 (+530507us) raft_consensus.cc:1685] finished
0709 13:03:37.358396 (+ 11477us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:37.358583 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17397,"prepare.queue_time_us":32773,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":18058,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":17347,"spinlock_wait_cycles":73742720,"thread_start_us":25534,"threads_started":1,"wal-append.queue_time_us":205020}
I20260709 13:03:37.326484  7396 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:37.383757  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:37.351142  1486 rpcz_store.cc:269] 0709 13:03:24.902442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:24.902649 (+   207us) service_pool.cc:224] Handling call
0709 13:03:37.262886 (+12360237us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:33.156578 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:33.200333 (+ 43755us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:33.213877 (+ 13544us) write_op.cc:435] Acquired schema lock
0709 13:03:33.215012 (+  1135us) tablet.cc:662] Decoding operations
0709 13:03:33.358028 (+143016us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:33.358132 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:33.358162 (+    30us) tablet.cc:685] Acquiring locks for 14 operations
0709 13:03:33.359524 (+  1362us) tablet.cc:701] Row locks acquired
0709 13:03:33.359576 (+    52us) write_op.cc:260] PREPARE: finished
0709 13:03:33.359784 (+   208us) write_op.cc:270] Start()
0709 13:03:33.359969 (+   185us) write_op.cc:276] Timestamp: P: 1783602213359705 usec, L: 0
0709 13:03:33.360014 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 13:03:33.360908 (+   894us) log.cc:844] Serialized 5575 byte log entry
0709 13:03:33.821721 (+460813us) op_driver.cc:464] REPLICATION: finished
0709 13:03:34.386823 (+565102us) write_op.cc:301] APPLY: starting
0709 13:03:34.410131 (+ 23308us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:34.709308 (+299177us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:34.709494 (+   186us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:35.501608 (+792114us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:35.548479 (+ 46871us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 13:03:35.548643 (+   164us) write_op.cc:312] APPLY: finished
0709 13:03:35.549700 (+  1057us) log.cc:844] Serialized 129 byte log entry
0709 13:03:35.664863 (+115163us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:35.710655 (+ 45792us) write_op.cc:454] Released schema lock
0709 13:03:35.728129 (+ 17474us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:35.762572 (+ 34443us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:35.762617 (+    45us) write_op.cc:454] Released schema lock
0709 13:03:36.981805 (+1219188us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
Metrics: {"mutex_wait_us":491991,"reported_tablets":26,"spinlock_wait_cycles":1792,"child_traces":[["op",{"apply.queue_time_us":518879,"apply.run_cpu_time_us":55429,"apply.run_wall_time_us":1409410,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62675,"num_ops":14,"prepare.queue_time_us":201807,"prepare.run_cpu_time_us":31200,"prepare.run_wall_time_us":339692,"raft.queue_time_us":330335,"raft.run_cpu_time_us":160036,"raft.run_wall_time_us":380715,"replication_time_us":461492,"spinlock_wait_cycles":249469312,"thread_start_us":913832,"threads_started":5,"wal-append.queue_time_us":208702,"wal-append.run_cpu_time_us":128650,"wal-append.run_wall_time_us":1329108}]]}
I20260709 13:03:37.386409  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:37.387072  1741 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.346395  7240 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:37.349941  7235 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:37.134030  7283 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.180330  7244 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.122s
I20260709 13:03:37.118279  7408 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.147650  7395 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.263749  7239 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:37.456986  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2242) took 1489 ms. Trace:
I20260709 13:03:37.306571  7399 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.267019  7266 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.341578  1659 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:37.369765  7385 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:37.428426  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:37.394282  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.415361  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.415892  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.431815  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:37.455006  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 100154us
I20260709 13:03:37.484236  2312 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.485472  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:37.343175  7389 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.457470  1742 rpcz_store.cc:276] 0709 13:03:35.967222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.967377 (+   155us) service_pool.cc:224] Handling call
0709 13:03:37.456746 (+1489369us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400107cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:37.456916 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431418240}
I20260709 13:03:37.337149  7326 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.391918  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.465428  7235 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:37.384598  7270 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.395254  7137 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:37.394819  7153 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.462291  7239 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.470886  7349 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:37.482142  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53903.2us
I20260709 13:03:37.373593  7378 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:37.492506  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.607422  1659 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:37.502378  2317 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:37.482990  1733 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:03:37.389879  6485 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:37.466708  7283 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.353159  7250 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.613279  2313 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:37.481468  6934 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.467257  7244 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:37.396039  7240 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:37.433279  7396 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:37.596271  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.626709  1668 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 9.
I20260709 13:03:37.468158  7207 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.414984  7282 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.110s
I20260709 13:03:37.389485  7361 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.159s
I20260709 13:03:37.499536  7406 raft_consensus.cc:2764] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 13:03:37.615396  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 308039us
I20260709 13:03:37.436894  7317 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.161s
I20260709 13:03:37.499249  7385 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:37.619841  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 584891us
I20260709 13:03:37.457815  7279 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.147s
W20260709 13:03:37.629422  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:37.597088  7137 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:37.696291  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:37.383855  7278 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.671808  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:37.680727  2312 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:37.690029  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.620200  7244 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:37.707306  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 221942us
I20260709 13:03:37.625530  7240 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:37.697239  1668 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.481274  7381 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.720075  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:37.721202  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49541.8us
I20260709 13:03:37.670471  7207 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.689814  7385 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:37.673820  7409 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:37.643477  1741 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:37.700588  7137 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:37.734016  1735 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.009s	sys 0.008s
I20260709 13:03:37.733814  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:37.735096  1659 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:37.704161  7247 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:37.730757  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.023s	sys 0.036s
W20260709 13:03:37.746315  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.023s	sys 0.036s
W20260709 13:03:37.746850  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 622894us
W20260709 13:03:37.734412  1735 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.009s	sys 0.008s
I20260709 13:03:37.638015  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:37.718672  7355 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.758126  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:37.764010  1708 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:37.638697  7344 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:37.636566  7396 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:37.783109  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 145101us
I20260709 13:03:37.743875  7278 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:37.730177  7415 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:37.753440  7158 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.803721  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:37.740604  7039 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:37.782578  7409 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:37.782671  7247 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.735811  7207 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.817195  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:37.792913  1666 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.000s
I20260709 13:03:37.820582  7039 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:37.821242  7411 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 13:03:37.822024  7412 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:03:37.822243  7250 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.822300  7432 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.009s
W20260709 13:03:37.834077  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 112749us
I20260709 13:03:37.822901  7290 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:37.826231  1666 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:37.804323  7421 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.809610  7248 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.741340  6485 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.812049  7282 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:37.839309  1667 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:03:37.814741  7420 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:37.840672  7397 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:37.823263  7266 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.814885  7158 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:37.826189  7415 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:37.829347  7259 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:37.760214  7235 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.827740  7247 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.835067  7249 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:37.810139  7244 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.837565  7432 consensus_queue.cc:1243] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 13:03:37.915746  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.841251  1659 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:37.874825  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:37.894264  1668 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:37.905112  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.905387  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:37.857832  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2881) took 1436 ms. Trace:
I20260709 13:03:37.975989  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.833827  7153 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.096s
I20260709 13:03:37.984776  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:37.878139  7396 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.861545  7207 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.884853  7395 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.008728  7207 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:37.988179  7434 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:38.010788  7137 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.999934  7262 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.012850  7262 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.861970  7266 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260709 13:03:37.857898  7282 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.870455  7415 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.014900  7266 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.015043  7415 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:38.015672  7282 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.016283  7282 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.016775  7282 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.858992  7399 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:37.870102  7344 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:37.852231  7421 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:37.986066  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 133893us
I20260709 13:03:37.988549  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:38.021930  1668 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:37.987118  1666 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:37.989673  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.015441  7266 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:37.923736  7416 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:37.858039  7389 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.986311  1741 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:37.987567  1663 rpcz_store.cc:276] 0709 13:03:36.421670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.659181 (+237511us) service_pool.cc:224] Handling call
0709 13:03:37.092873 (+433692us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.372795 (+279922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.856955 (+484160us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54002f2e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:37.857578 (+   623us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54002f2e50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:37.857759 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1559474944}
I20260709 13:03:37.987844  1740 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:37.858634  7420 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:37.884874  7326 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:37.859302  7397 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:38.024595  1659 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:37.884510  7408 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:38.030122  7426 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
W20260709 13:03:38.030812  7035 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.946s	user 0.007s	sys 0.007s
I20260709 13:03:38.033293  7282 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.039583  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.054701  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:38.090170  1663 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:38.092343  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52873.8us
I20260709 13:03:38.097575  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.100579  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:38.101593  1740 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:38.061590  1741 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:38.072512  2317 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:38.041090  7409 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:37.852169  7290 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.078274  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:37.985178  7417 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.110s
I20260709 13:03:38.046886  7421 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.080694  7385 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.082305  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:38.147959  1659 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:38.045475  7416 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.106312  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:38.009418  7207 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.145526  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.122640  1668 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:38.139652  1666 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:38.152329  1735 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:38.154533  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:38.134516  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.155506  2316 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:38.111174  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.142903  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:38.020394  7239 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.154759  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.156741  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:38.142127  7378 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.509s	user 0.002s	sys 0.011s
I20260709 13:03:38.160152  2312 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:03:38.160387  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 335702us
I20260709 13:03:38.140805  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:38.162760  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51860us
I20260709 13:03:38.157096  1741 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:38.162621  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.164191  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:38.141933  7408 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:38.110466  7428 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:38.168722  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 404821us
I20260709 13:03:38.168587  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:38.225308  2280 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.572s (SENT) [suppressed 20 similar messages]
W20260709 13:03:38.240056  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75951.4us
W20260709 13:03:38.247083  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 74778.6us
I20260709 13:03:38.250802  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:38.251775  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89247.5us
I20260709 13:03:38.252808  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:38.256217  2309 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 13:03:38.259928  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.252902  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.207608  7409 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:38.266786  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:38.321740  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.322695  1659 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:38.248864  7240 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.515s	user 0.012s	sys 0.000s
I20260709 13:03:38.266887  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:38.318287  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 68160.5us
I20260709 13:03:38.094611  7035 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.315851  7249 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:38.320518  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60704.3us
I20260709 13:03:38.151988  7416 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:38.316216  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63396.1us
I20260709 13:03:38.325090  2311 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:38.315798  7250 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 13:03:38.328276  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 103040us
I20260709 13:03:38.246755  7434 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:38.251955  7408 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:38.257284  7421 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:38.387257  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:38.387871  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.257586  7326 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.325333  7248 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.457315  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 905) took 1345 ms. Trace:
I20260709 13:03:38.391077  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.324322  7407 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:38.269074  7418 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.318008  7396 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:38.317184  7287 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.477787  1588 rpcz_store.cc:276] 0709 13:03:37.111858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.112176 (+   318us) service_pool.cc:224] Handling call
0709 13:03:37.112490 (+   314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.438768 (+326278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.438802 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.462152 (+ 23350us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.756030 (+293878us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:37.756070 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.824401 (+ 68331us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54003b10d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:37.824470 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.366279 (+541809us) raft_consensus.cc:1685] finished
0709 13:03:38.366316 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.457211 (+ 90895us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1873,"raft.queue_time_us":154741,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":11656,"spinlock_wait_cycles":746835712,"thread_start_us":290178,"threads_started":1,"wal-append.queue_time_us":419817}
I20260709 13:03:38.479981  1659 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:38.197449  7378 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.326445  7409 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:38.470504  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83717.4us
I20260709 13:03:38.462944  7428 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:38.315912  7158 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.158s
I20260709 13:03:38.466787  7408 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:38.476538  7419 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 13:03:38.494186  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.477169  7248 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.472493  7250 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:38.556582  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.558075  1659 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:38.471936  7434 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:38.364506  7240 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.518189  7158 consensus_queue.cc:1243] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 16.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 13:03:38.526235  7423 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.369s	user 0.000s	sys 0.000s
I20260709 13:03:38.533819  7393 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.488782  7153 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
W20260709 13:03:38.568735  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1045) took 1446 ms (client timeout 1630 ms). Trace:
I20260709 13:03:38.537672  7430 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:38.539083  7431 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:38.540704  7439 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 13:03:38.328320  7424 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.165s
I20260709 13:03:38.544148  7385 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:38.546182  7443 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
W20260709 13:03:38.547245  7438 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:03:38.495811  7418 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:38.554195  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60351.1us
I20260709 13:03:38.556171  7248 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:38.500233  7409 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:38.473254  7421 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.507680  7428 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.483560  7326 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.567323  1659 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:03:38.551649  7247 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.162047  7420 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.603363  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:38.569679  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 182309us
W20260709 13:03:38.569275  2318 rpcz_store.cc:269] 0709 13:03:37.122371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.122515 (+   144us) service_pool.cc:224] Handling call
0709 13:03:38.568494 (+1445979us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b54007233d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:38.568656 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119763712}
W20260709 13:03:38.570503  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.555187  7396 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:38.573796  1664 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:38.552994  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 179379us
W20260709 13:03:38.581426  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.008s	sys 0.009s
I20260709 13:03:38.582988  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2884) took 1557 ms. Trace:
I20260709 13:03:38.581746  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.585798  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2895) took 1158 ms. Trace:
I20260709 13:03:38.597008  1659 raft_consensus.cc:606] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:38.631284  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.017s	sys 0.011s
W20260709 13:03:38.631489  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:38.631958  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 175232us
W20260709 13:03:38.632426  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:38.633466  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.634627  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:38.573287  7393 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:38.644385  7278 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.838s	user 0.013s	sys 0.001s
I20260709 13:03:38.640149  1659 raft_consensus.cc:614] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:38.596262  7428 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:38.650620  1739 rpcz_store.cc:276] 0709 13:03:37.427799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.427952 (+   153us) service_pool.cc:224] Handling call
0709 13:03:37.435466 (+  7514us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.435982 (+   516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.436041 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:37.436713 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 13:03:37.726976 (+290263us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:37.727038 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:37.727140 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.585581 (+858441us) raft_consensus.cc:1685] finished
0709 13:03:38.585606 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.585733 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":184512,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":11397,"thread_start_us":289808,"threads_started":1,"wal-append.queue_time_us":636048}
W20260709 13:03:38.650632  1740 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.535s	user 0.013s	sys 0.001s
I20260709 13:03:38.670006  1740 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:38.651414  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69761.5us
I20260709 13:03:38.661489  7442 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:38.661602  7445 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.648130  7424 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.649798  7452 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:38.631912  7250 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.665830  7381 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.591482  7290 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.707154  7399 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 13:03:38.710326  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.711148  1586 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:38.712308  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.712950  1586 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:38.712652  7417 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:38.714401  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:38.704643  7424 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.719282  7421 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.723152  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 2892) took 1359 ms. Trace:
I20260709 13:03:38.723927  1737 rpcz_store.cc:276] 0709 13:03:37.363790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.366011 (+  2221us) service_pool.cc:224] Handling call
0709 13:03:38.722769 (+1356758us) spinlock_profiling.cc:243] Waited 952 ms on lock 0x7b5400404554. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:38.723048 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2189501312}
I20260709 13:03:38.726095  7318 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:38.726327  7064 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:38.726857  7318 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.729175  7318 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.676975  7385 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.677448  7011 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:03:38.678021  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 940) took 2023 ms (client timeout 1572 ms). Trace:
I20260709 13:03:38.678905  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.679713  7270 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.685415  7453 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.680502  7437 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:38.761162  7100 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.762035  7437 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.764828  7437 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.681062  7235 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.765791  7235 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.768003  7418 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.681996  6934 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.682328  1736 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:38.768869  6934 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.769419  7326 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.769809  7317 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:38.770015  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.770713  1736 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:38.770740  7317 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.771487  6934 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.683169  1734 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:38.683678  7444 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:38.774366  7317 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.773929  7444 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.688138  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:38.776777  7444 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.688512  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.688797  1742 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:38.573407  7431 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.695876  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:38.697206  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:52744 (request call id 14) took 1030 ms. Trace:
I20260709 13:03:38.626515  7239 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.639377  7397 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.699297  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.700040  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.782464  1593 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:38.782115  7397 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.783720  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.784371  1593 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:38.639106  7207 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.701889  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.703114  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.789584  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.791776  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.792620  7378 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.792919  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.703972  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.717392  7424 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.701120  7445 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.781540  1741 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:38.704989  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.795780  1587 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:38.798744  1590 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:38.799429  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:38.800016  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.580976  7249 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.800694  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:38.746031  7415 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.718s	user 0.010s	sys 0.004s
I20260709 13:03:38.709878  7381 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.707650  7452 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.726866  7064 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:38.731407  7266 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.818249  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2887) took 1791 ms. Trace:
I20260709 13:03:38.731596  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.820721  1737 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:38.738835  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:38.671315  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:38.650555  1661 rpcz_store.cc:276] 0709 13:03:37.025854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.027358 (+  1504us) service_pool.cc:224] Handling call
0709 13:03:37.027753 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.060323 (+ 32570us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:37.206565 (+146242us) log.cc:844] Serialized 14 byte log entry
0709 13:03:38.247870 (+1041305us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.248602 (+   732us) log.cc:844] Serialized 26 byte log entry
0709 13:03:38.248815 (+   213us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:38.248853 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:38.248944 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:38.582757 (+333813us) raft_consensus.cc:1685] finished
0709 13:03:38.582781 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.582920 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7012,"prepare.queue_time_us":371085,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":15812,"raft.queue_time_us":108881,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":14293,"thread_start_us":1070538,"threads_started":4,"wal-append.queue_time_us":521470}
W20260709 13:03:38.746575  1665 rpcz_store.cc:269] 0709 13:03:36.654376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.656819 (+  2443us) service_pool.cc:224] Handling call
0709 13:03:38.677826 (+2021007us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54002f1a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:38.677948 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1296125824}
I20260709 13:03:38.685796  7153 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:38.648074  7353 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:38.668401  7035 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:38.747102  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68283.4us
I20260709 13:03:38.687618  7393 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.747973  7011 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:38.750977  7355 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:38.751506  7385 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.752907  7137 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.758437  7451 raft_consensus.cc:2764] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 13:03:38.768153  7235 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.774147  7409 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:38.684473  7247 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:38.780503  7431 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.702751  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.787125  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:38.773329  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.843022  7153 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.795267  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:38.704387  1584 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:38.803668  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:38.803241  7248 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.806950  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.776955  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:39.059442  1738 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:38.786687  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.060082  1587 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 13:03:38.815398  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 212140us
I20260709 13:03:39.060753  7248 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.061379  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.061789  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:39.062373  1587 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:03:39.062757  1587 pending_rounds.cc:85] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 7
I20260709 13:03:39.062968  1587 pending_rounds.cc:107] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:03:39.063282  1587 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:39.063439  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2190) took 1569 ms. Trace:
I20260709 13:03:38.785290  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:38.816187  1659 rpcz_store.cc:276] 0709 13:03:37.631448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.563725 (+932277us) service_pool.cc:224] Handling call
0709 13:03:38.652441 (+ 88716us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:38.824076  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.060799  1592 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:39.064507  1661 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:38.707368  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.821748  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:38.818701  1660 rpcz_store.cc:276] 0709 13:03:37.026745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.181160 (+154415us) service_pool.cc:224] Handling call
0709 13:03:37.202469 (+ 21309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.260387 (+ 57918us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:37.260606 (+   219us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:37.260675 (+    69us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:37.260719 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:38.817970 (+1557251us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:38.818028 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:38.818183 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":132650240}
W20260709 13:03:38.835515  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 164310us
I20260709 13:03:39.066663  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:38.836062  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 129552us
I20260709 13:03:38.837945  1590 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:38.794221  7240 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:38.668776  7396 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:39.077674  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 229831us
I20260709 13:03:39.079615  1736 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:39.058291  1668 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.083607  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.083683  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.084360  7430 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:39.085052  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.084595  1593 raft_consensus.cc:2376] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13. Re-sending same reply.
I20260709 13:03:39.086045  7396 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:39.086791  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.086854  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.087647  1660 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:39.088340  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.088606  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.089021  1593 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:38.841298  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:38.840106  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.089883  7441 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:39.090320  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 250323us
I20260709 13:03:39.090520  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.092485  7153 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:38.844111  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105461us
I20260709 13:03:39.094475  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.094796  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.095180  1741 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:39.096321  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.096714  1741 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:03:38.785465  1735 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 5. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 13:03:38.878767  1585 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:38.844997  7011 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.097803  7393 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.053161  7381 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.878386  7247 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.041558  7431 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:39.042966  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.043668  7262 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.044452  2309 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:39.124276  7393 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.047188  7452 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:38.843808  7353 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.048038  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2191) took 1552 ms. Trace:
I20260709 13:03:39.169852  1667 rpcz_store.cc:276] 0709 13:03:37.495910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.496069 (+   159us) service_pool.cc:224] Handling call
0709 13:03:37.838973 (+342904us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:37.858356 (+ 19383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.047798 (+1189442us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b540013f6d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:39.047964 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":787561344}
I20260709 13:03:39.050631  7407 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:39.050956  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:39.053807  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 168839us
I20260709 13:03:39.054232  7244 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.055744  7385 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:38.796236  7453 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:38.843320  7355 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.183662  7355 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:38.878414  7064 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.186764  7064 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.063822  1664 rpcz_store.cc:276] 0709 13:03:37.494081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.494922 (+   841us) service_pool.cc:224] Handling call
0709 13:03:37.495363 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:38.668320 (+1172957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.063194 (+394874us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:39.063377 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2479758208}
I20260709 13:03:39.190403  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.191145  1664 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:39.192305  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.193087  7452 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.068131  1591 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:39.194556  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.195182  1591 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.068691  1734 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:39.085160  7430 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:39.085460  7395 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.075176  1590 pending_rounds.cc:85] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:03:39.085994  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.217149  1590 pending_rounds.cc:107] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 13:03:39.084416  1659 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:39.218547  1590 raft_consensus.cc:2889] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:39.219080  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:38.852645  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.221803  1665 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:39.222004  2311 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:39.220520  1659 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:39.091578  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.840830  1586 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:39.224476  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.225030  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.225615  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:38.839792  1742 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:03:39.125311  1737 raft_consensus.cc:2410] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 13:03:39.123066  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.122973  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2933) took 2578 ms. Trace:
I20260709 13:03:39.126562  2316 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:39.089262  1660 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:39.127223  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.108068  1736 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 13:03:39.100224  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:39.098521  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:39.048077  7416 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.576s	user 0.014s	sys 0.028s
I20260709 13:03:39.128381  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.090440  7441 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.139478  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1050) took 1352 ms. Trace:
W20260709 13:03:39.144457  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.045722  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.226969  1584 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:39.159228  1668 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:39.123729  7240 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:39.076815  7457 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 13:03:39.129657  7408 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.585s	user 0.015s	sys 0.022s
I20260709 13:03:39.237323  1739 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:39.165820  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.123318  7282 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:39.069193  7235 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.077294  7266 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.255718  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.138226  7247 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:39.174558  1593 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:39.130618  7248 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.171944  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.167565  7393 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.207890  1591 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:39.213138  1585 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:38.833159  7415 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.225929  1586 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:39.227494  1589 rpcz_store.cc:276] 0709 13:03:36.544907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.545027 (+   120us) service_pool.cc:224] Handling call
0709 13:03:36.545331 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:37.271970 (+726639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.122659 (+1850689us) spinlock_profiling.cc:243] Waited 468 ms on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:39.122892 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1077341952}
I20260709 13:03:39.225811  1659 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:39.181663  7407 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:39.302759  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.180917  7409 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.235412  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.306926  1587 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
W20260709 13:03:39.230926  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102653us
I20260709 13:03:39.242744  7153 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:39.163789  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.231230  2317 rpcz_store.cc:276] 0709 13:03:37.787138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:37.787380 (+   242us) service_pool.cc:224] Handling call
0709 13:03:37.809123 (+ 21743us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:38.103417 (+294294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.139183 (+1035766us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b540059e7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:39.139403 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":605254912}
I20260709 13:03:39.238831  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:39.240540  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 142072us
I20260709 13:03:39.241698  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.237180  7279 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.073s
I20260709 13:03:39.314913  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.193740  7452 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:39.245337  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.247341  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:39.324105  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 76881.6us
W20260709 13:03:39.250418  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 158952us
I20260709 13:03:39.256959  1660 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:39.266381  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.326498  1668 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:39.326277  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.269218  1667 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:39.223421  6485 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:39.273841  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.223209  7385 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.256173  7361 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 13:03:39.284557  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.310025  7381 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.284147  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.347815  1586 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:39.347673  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:39.286341  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 159231us
I20260709 13:03:39.350013  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.301698  1665 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:39.196573  7445 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.307740  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.231595  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.244665  7457 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:39.309863  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.246639  7035 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:39.168129  7397 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.321982  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.160137  7431 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.321095  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.328233  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.284791  7424 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:39.238722  7395 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.332453  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.335351  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.378839  1660 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.335892  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.274809  7235 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.242533  7416 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.273824  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.419009  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.420759  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.246418  7408 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.344206  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:39.282810  7248 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.317523  7420 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.360859  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.428544  1668 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:39.363296  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.365721  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.366716  2314 raft_consensus.cc:2368] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:03:39.368109  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.431149  2309 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.434207  7235 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:39.434914  7462 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 13:03:39.368268  7457 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:39.437541   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:03:39.438208  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.439690  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.440590   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:39.444584  7441 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.446300  7453 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.271951  7266 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.448758  7266 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.348980  1589 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:39.450295  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.451881  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.451994  7248 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.366880  7035 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.453282  7035 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:39.453948  7415 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.454829  7467 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:03:39.371783  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43701.9us
I20260709 13:03:39.456295  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.379594  2315 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:39.283218  7282 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:39.333115  7039 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:39.338223  7392 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:39.337194  7409 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.355624  7442 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:39.469918  1588 raft_consensus.cc:2368] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
W20260709 13:03:39.471160  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107978us
I20260709 13:03:39.425624  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.445288  7441 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:39.453179  1591 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:39.371217  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.331192  7344 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:39.441073  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.474114  1663 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.474315  1660 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:39.467739  1659 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.476245  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.477335  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.478617  1668 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:39.463411  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:39.461395  7428 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.761s	user 0.014s	sys 0.004s
I20260709 13:03:39.491312  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.367231  7381 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.492882  1660 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:39.342831  6485 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.499825  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.466554  7207 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:39.502334  7282 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.502897  6485 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.466787  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.472368  1634 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.503731  1584 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:39.464733  7266 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.504918  7266 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.505323  7266 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.507620  7408 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.470427  7235 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 13:03:39.508117  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:39.520166  1664 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 11.
W20260709 13:03:39.520565  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48301us
I20260709 13:03:39.354790  7385 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.472527  1585 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:39.484288  7344 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:39.363618  7430 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.524971  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.467839  7453 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:39.539660  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.472062  7392 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:39.539472  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.481407  7441 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:39.564044  1659 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:39.482532  7409 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.558398  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2194) took 2060 ms. Trace:
I20260709 13:03:39.559713  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:39.566452  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 237285us
I20260709 13:03:39.566465  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2193) took 2069 ms. Trace:
I20260709 13:03:39.567250  1660 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:39.565788  1665 rpcz_store.cc:276] 0709 13:03:37.497533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.521003 (+2023470us) service_pool.cc:224] Handling call
0709 13:03:39.557661 (+ 36658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.557921 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.557959 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.558044 (+    85us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:39.558086 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.558160 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.558307 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.567359  1662 rpcz_store.cc:276] 0709 13:03:37.497211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.580615 (+1083404us) service_pool.cc:224] Handling call
0709 13:03:38.584182 (+  3567us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:38.584610 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:38.584646 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:38.585175 (+   529us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.547379 (+962204us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18003b9b60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1
0709 13:03:39.554394 (+  7015us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:39.554510 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.554661 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:39.554746 (+    85us) raft_consensus.cc:1685] finished
0709 13:03:39.554760 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.566398 (+ 11638us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1016,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":8428,"spinlock_wait_cycles":3662080,"thread_start_us":193721,"threads_started":1,"wal-append.queue_time_us":215041}
I20260709 13:03:39.577698  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2201) took 2072 ms. Trace:
I20260709 13:03:39.578387  1659 rpcz_store.cc:276] 0709 13:03:37.505283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.563302 (+2058019us) service_pool.cc:224] Handling call
0709 13:03:39.563781 (+   479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.577197 (+ 13416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.577591 (+   394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.518476  7428 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.583681  7428 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.587131  1706 leader_election.cc:400] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:03:39.476409  7039 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.587713  1706 leader_election.cc:403] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:39.590294  1666 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.592211  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:39.594444  7395 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.595242  7353 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.602051  7039 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.518275  7442 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:39.603473  7385 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.540210  7349 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.518813  7283 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 13:03:39.604436  7349 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.605016  7349 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.605095  7283 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:39.605603  7283 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:39.604166  7385 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.607095  7247 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.606820  7385 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.604168  7442 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.609742  7415 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.611683  7355 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.587291  7416 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.615486  7431 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.615762  7011 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.618839  7457 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.620028  2313 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:03:39.621140  7035 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.568918  7441 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.537762  7397 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 13:03:39.622215  1662 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:39.565640  7235 consensus_queue.cc:1243] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 15.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:03:39.622774  7381 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.625658  7247 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.627403  7385 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:39.627605  7407 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.627916  7344 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.629010  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.629698  1660 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [15.8-17.9]
I20260709 13:03:39.675796  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2203) took 2169 ms. Trace:
I20260709 13:03:39.527683  7248 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:39.631223  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 323612us
I20260709 13:03:39.566365  7137 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:03:39.627602  7396 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.541s	user 0.006s	sys 0.006s
I20260709 13:03:39.552407  7250 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:39.566548  7409 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.711135  1586 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:39.562714  7392 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.676630  1660 rpcz_store.cc:276] 0709 13:03:37.505884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.577432 (+2071548us) service_pool.cc:224] Handling call
0709 13:03:39.629192 (+ 51760us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:39.675345 (+ 46153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.675717 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.714432  1660 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:39.715127  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2205) took 2208 ms. Trace:
I20260709 13:03:39.715550  1660 rpcz_store.cc:276] 0709 13:03:37.506494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.713557 (+2207063us) service_pool.cc:224] Handling call
0709 13:03:39.713953 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.714859 (+   906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.715052 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:39.718071  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:39.718477  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.572103  7445 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:39.672390  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.527971  7207 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:39.723327  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.683576  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.723683  7445 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.606521  7283 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.728230  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.729614  1738 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:03:39.701462  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:39.564916  7239 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:39.724860  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:39.717240  1660 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:39.724702  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.698390  7039 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:03:39.741572  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 397498us
I20260709 13:03:39.682005  7457 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.736765  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:39.746907  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 946) took 1149 ms. Trace:
I20260709 13:03:39.748051  1666 rpcz_store.cc:276] 0709 13:03:38.598036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.688210 (+ 90174us) service_pool.cc:224] Handling call
0709 13:03:39.746493 (+1058283us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54002f1a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:39.746809 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867535104}
W20260709 13:03:39.751971  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 425803us
W20260709 13:03:39.756115  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83835.5us
I20260709 13:03:39.699185  7137 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:39.739048  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.762074  1634 leader_election.cc:400] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:39.713791  7279 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.757902  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:39.724839  7445 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.738343  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2206) took 2231 ms. Trace:
I20260709 13:03:39.762472  1634 leader_election.cc:403] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:03:39.724634  7250 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.012s
I20260709 13:03:39.753119  2283 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:39.727612  7344 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.777343  1660 rpcz_store.cc:276] 0709 13:03:37.506768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.716500 (+2209732us) service_pool.cc:224] Handling call
0709 13:03:39.716895 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.737884 (+ 20989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.738253 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.729573  7248 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:39.741875  7451 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:39.779464  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.803817  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.805859  7428 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:39.806730  7428 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:39.761685  7467 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:39.808916  7467 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.810936  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2895) took 2214 ms. Trace:
I20260709 13:03:39.811419  1666 rpcz_store.cc:276] 0709 13:03:37.596131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.784091 (+2187960us) service_pool.cc:224] Handling call
0709 13:03:39.810267 (+ 26176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.810564 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.810594 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.810640 (+    46us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:39.810681 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.810741 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.810877 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.813789  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 941) took 2065 ms. Trace:
I20260709 13:03:39.813989  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.814230  1666 rpcz_store.cc:276] 0709 13:03:37.748282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.812629 (+2064347us) service_pool.cc:224] Handling call
0709 13:03:39.813097 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.813358 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.813386 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.813432 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:03:39.813471 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.813564 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.813716 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.780578  1632 leader_election.cc:400] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
W20260709 13:03:39.786237  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 43744.3us
I20260709 13:03:39.818337  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:39.819504  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80598.6us
I20260709 13:03:39.820665  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.820984  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.702481  7064 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 13:03:39.825273  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.826169  7064 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:39.826452  1588 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:03:39.827296  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69881.9us
I20260709 13:03:39.828387  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:39.830497  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.831653  1588 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:39.832088  7470 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:39.832437  7361 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:39.832597  7470 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.832969  7361 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.833123  7470 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.833389  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.833715  7361 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:39.834470  1588 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:39.836977  7407 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.741168  7262 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:39.837834  7262 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:39.838441  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:39.765761  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2192) took 2268 ms (client timeout 2810 ms). Trace:
I20260709 13:03:39.741585  7039 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:39.750432  7399 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:39.800139  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:39.843444  1591 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:39.759822  7468 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:39.839461  1663 rpcz_store.cc:269] 0709 13:03:37.496902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:38.112634 (+615732us) service_pool.cc:224] Handling call
0709 13:03:39.765407 (+1652773us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540013e7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:39.765663 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3022668032}
I20260709 13:03:39.713263  7396 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.840719  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.816380  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 942) took 2059 ms. Trace:
I20260709 13:03:39.762739  7137 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:39.723862  7207 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.850657  1666 rpcz_store.cc:276] 0709 13:03:37.756946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.815483 (+2058537us) service_pool.cc:224] Handling call
0709 13:03:39.815826 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.816043 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.816071 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.816126 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:39.816162 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.816217 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:39.816323 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:39.782495  7287 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:39.790084  7247 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15.
I20260709 13:03:39.781968  7153 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:39.852695  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 313307us
I20260709 13:03:39.854465  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.858942  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.787669  7480 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:39.871901  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:39.872822  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.876876  1742 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:03:39.751163  7393 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.869647  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.883865  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.885176  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.881188  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.886935  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.846306  7399 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.847234  7248 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:39.840303  7475 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:39.790477  7279 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.901958  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.859809  7385 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:03:39.929203  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 465986us
I20260709 13:03:39.930250  2312 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
I20260709 13:03:39.903396  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.903365  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:39.927824  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:39.932587  1663 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.932387  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:39.934441  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 141210us
I20260709 13:03:39.935382  7035 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:39.935709  7478 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:39.936506  7478 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:39.930516  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:39.838804  7262 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:39.859956  7287 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.940276  7287 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:39.942619  7396 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.942518  7349 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.927006  7247 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:39.946208  7247 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.946669  7247 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.948050  7266 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:39.950258  1735 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:39.886309  7407 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:39.951747  7407 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:39.952164  7407 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:39.952731  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 187951us
I20260709 13:03:39.883302  7355 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:39.888248  7244 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.674s	user 0.013s	sys 0.002s
I20260709 13:03:39.892234  7468 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:39.956393  7468 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:39.956528  1661 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:39.956940  7468 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:39.957845  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2902) took 1283 ms. Trace:
I20260709 13:03:39.958491  1661 rpcz_store.cc:276] 0709 13:03:38.674238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.084751 (+410513us) service_pool.cc:224] Handling call
0709 13:03:39.957659 (+872908us) spinlock_profiling.cc:243] Waited 829 ms on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:39.957784 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1906496768}
I20260709 13:03:39.959865  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.960734  1661 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.962653  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.963332  1666 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:39.967665  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:39.968709  1668 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.895610  7399 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:39.936743  7035 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:39.984342  1742 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:39.985378  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:39.991722  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.992782  1734 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:39.994712  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.994706  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:39.995723  1667 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
W20260709 13:03:39.997362  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 179132us
I20260709 13:03:40.003851  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.016220  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:40.017108  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.018327  1668 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:40.018260  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.020421  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.021180  1668 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:40.022243  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.022979  1668 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:40.024334  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.026014  1593 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:40.025579  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.028337  2316 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:40.028448  1668 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:40.029799  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.030637  1668 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:40.019402  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.033932  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.034169  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.035600  1666 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:39.906188  7475 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:40.032261  1663 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:40.041693  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:39.989023  7235 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:40.034941  1735 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:40.042402  1733 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:40.051365  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.043880  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.052536  1734 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:40.045778  1736 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:40.053960  1736 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.056674  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.047225  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.048252  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.063678  2315 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:40.013334  7415 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:03:40.068382  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.069828  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.079537  1735 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:40.080188  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.080657  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.064630  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:40.082577  2309 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 13:03:40.070108  2317 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:03:40.072288  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.016040  7385 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:40.074542  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:40.097199  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.055598  1736 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:40.098068  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 269915us
I20260709 13:03:40.098996  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:40.078653  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.086228  1663 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:40.087013  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.100215  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.100531  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 948) took 1492 ms. Trace:
I20260709 13:03:40.100636  2311 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:40.073405  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.041373  7349 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:39.961272  7248 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.048682  7474 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:40.099144  1736 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:40.100921  1663 rpcz_store.cc:276] 0709 13:03:38.607744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.054007 (+1446263us) service_pool.cc:224] Handling call
0709 13:03:40.085780 (+ 31773us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.100235 (+ 14455us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.100471 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.082659  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.119259  1735 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:03:40.069376  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 947) took 1466 ms. Trace:
I20260709 13:03:40.014031  7035 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.104831  1661 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:40.086238  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.047755  7467 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:40.130085  7467 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:40.131897  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2204) took 2625 ms. Trace:
I20260709 13:03:40.132347  1662 rpcz_store.cc:276] 0709 13:03:37.506192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.612076 (+2105884us) service_pool.cc:224] Handling call
0709 13:03:39.612535 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.658702 (+ 46167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.131587 (+472885us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b54003c8a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:40.131820 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21576064}
I20260709 13:03:40.107877  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.134021  1733 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:40.134344  1662 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:03:40.135056  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2213) took 1459 ms. Trace:
I20260709 13:03:40.027918  7064 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.135372  1662 rpcz_store.cc:276] 0709 13:03:38.675393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.133710 (+1458317us) service_pool.cc:224] Handling call
0709 13:03:40.134118 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:40.134798 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.135001 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.136518  7064 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.137477  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2218) took 1460 ms. Trace:
I20260709 13:03:40.137925  1662 rpcz_store.cc:276] 0709 13:03:38.676712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.136303 (+1459591us) service_pool.cc:224] Handling call
0709 13:03:40.136609 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.136800 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:40.136826 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:40.136870 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:40.137182 (+   312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.137253 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.137430 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.042361  7475 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:39.927857  7039 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.140842  7039 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:40.114533  7486 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:03:40.155189  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.156078  1740 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:40.018275  7392 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.155278  2311 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:40.121436  1666 rpcz_store.cc:276] 0709 13:03:38.603274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.068071 (+1464797us) service_pool.cc:224] Handling call
0709 13:03:40.068416 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.068633 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:40.068662 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:40.068846 (+   184us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:40.068924 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.069141 (+   217us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.069317 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:40.104242  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49222.3us
I20260709 13:03:40.127193  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:40.160148  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:03:40.163389  1663 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:40.164286  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2212) took 1489 ms. Trace:
I20260709 13:03:40.164940  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:40.165688  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:40.128825  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.164614  1663 rpcz_store.cc:276] 0709 13:03:38.675139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.145437 (+1470298us) service_pool.cc:224] Handling call
0709 13:03:40.162970 (+ 17533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.163948 (+   978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.164161 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.139832  1662 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:40.147557  1661 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:40.167840  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2901) took 1494 ms. Trace:
I20260709 13:03:40.168323  1661 rpcz_store.cc:276] 0709 13:03:38.673924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.147025 (+1473101us) service_pool.cc:224] Handling call
0709 13:03:40.147367 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.167517 (+ 20150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.167766 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.135552  1733 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:40.128161  7262 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:40.171119  1666 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [11.10-16.12]
I20260709 13:03:40.173259  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2219) took 1496 ms. Trace:
I20260709 13:03:40.173708  1662 rpcz_store.cc:276] 0709 13:03:38.677284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.139163 (+1461879us) service_pool.cc:224] Handling call
0709 13:03:40.139628 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.172350 (+ 32722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.173198 (+   848us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:40.130556  7467 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.176429  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.171967  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.119724  7244 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.120666  7415 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:40.186782  1742 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:40.191185  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.195837  1666 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:40.196374  1663 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 13:03:40.206367  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:40.176795  7479 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:40.166074  7207 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.211347  7207 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.212134  7355 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:40.212443  7287 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.190323  7494 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:40.213624  7494 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:40.122671  7474 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.214056  7494 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:40.215813  1487 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 5 to 8, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:40.135490  7441 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.219983  7441 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.220852  7452 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.210520  7479 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:40.222093  7479 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.222191  7392 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.225919  1666 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:40.225576  7445 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.191123  7262 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 13:03:40.078390  7153 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.238080  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:40.209986  1742 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:03:40.230556  1663 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:40.232291  1736 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:40.232492  1740 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:40.248883  7407 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.254267  2316 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:40.193763  7396 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:40.239665  7452 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:03:40.267582  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 469526us
I20260709 13:03:40.239499  7355 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:40.285089  1740 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 13:03:40.284895  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88896.3us
I20260709 13:03:40.286769  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.287479  7500 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:40.288228  7064 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:40.289458  7262 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:40.290426  7262 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.259012  7385 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.292088  7262 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.293550  7035 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.196895  7244 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.296165  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:40.261878  7452 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:40.310123  7473 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.269s	user 0.001s	sys 0.000s
I20260709 13:03:40.264425  7393 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.323141  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.325449  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:40.327224  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:40.333278  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.337872  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:40.279438  7407 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.293426  7262 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.313632  1487 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 5 to 6, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:40.298784  7355 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:40.317189  7452 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:40.302091  7266 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.332449  7483 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:40.341142  7500 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:40.340061  7430 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:40.366055  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.374301  1662 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:40.296411  7396 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 13:03:40.382262  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.382925  2309 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:40.385747  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 542) took 1309 ms. Trace:
I20260709 13:03:40.351418  7471 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:40.386709  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 284378us
I20260709 13:03:40.394040  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.341295  7035 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:03:40.406611  7287 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.408766  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.409614  1734 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:40.388012  7430 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:40.411269  7430 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.418576  1734 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 13:03:40.373445  7445 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.386328  2310 rpcz_store.cc:276] 0709 13:03:39.076447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.076607 (+   160us) service_pool.cc:224] Handling call
0709 13:03:39.083519 (+  6912us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.099912 (+ 16393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.099955 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:39.100466 (+   511us) log.cc:844] Serialized 26 byte log entry
0709 13:03:39.335823 (+235357us) spinlock_profiling.cc:243] Waited 986 us on lock 0x7b18001a1ee0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:39.780909 (+445086us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:39.780968 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.781128 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.377103 (+595975us) raft_consensus.cc:1685] finished
0709 13:03:40.377130 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.385649 (+  8519us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49449,"raft.queue_time_us":16834,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":2268416,"thread_start_us":444242,"threads_started":1,"wal-append.queue_time_us":869886}
I20260709 13:03:40.389273  7500 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.439750  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:40.440320  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 954) took 1349 ms (client timeout 1648 ms). Trace:
W20260709 13:03:40.440809  1664 rpcz_store.cc:269] 0709 13:03:39.084215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.192042 (+107827us) service_pool.cc:224] Handling call
0709 13:03:40.433290 (+1241248us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:40.433416 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":752474880}
W20260709 13:03:40.449849  7432 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 13:03:40.454305   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:03:40.429100  7470 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.005s
I20260709 13:03:40.456753  2310 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:40.461774  7500 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:40.462447  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68513.6us
I20260709 13:03:40.464731  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.465965  7287 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:40.466586  7287 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:40.467720  7287 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.469826  7207 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.470434  7505 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:40.470938  7505 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:40.471630  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.469905  7491 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 13:03:40.471370  7505 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:40.476164  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2197) took 2977 ms. Trace:
I20260709 13:03:40.476621  1667 rpcz_store.cc:276] 0709 13:03:37.503860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.511286 (+2007426us) service_pool.cc:224] Handling call
0709 13:03:39.511633 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.996188 (+484555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.475880 (+479692us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:40.476096 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112895360}
W20260709 13:03:40.487740  7137 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.001s
I20260709 13:03:40.488125  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:40.488345  7137 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.489825  7259 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:40.490839  7259 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:40.490491  1487 catalog_manager.cc:5697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:40.491824  7207 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:40.492214  7207 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.493225  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3052) took 1165 ms. Trace:
I20260709 13:03:40.492947  1487 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:40.493145  7207 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.491379  7259 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.493736  1586 rpcz_store.cc:276] 0709 13:03:39.327828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.365327 (+ 37499us) service_pool.cc:224] Handling call
0709 13:03:40.492987 (+1127660us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54001fa254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:40.493148 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645389568}
I20260709 13:03:40.494539  1487 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 5, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 5 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:40.495936  7207 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:40.496440  7355 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.497893  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1052) took 1432 ms. Trace:
I20260709 13:03:40.498417  2313 rpcz_store.cc:276] 0709 13:03:39.065453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.065636 (+   183us) service_pool.cc:224] Handling call
0709 13:03:39.074323 (+  8687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.620544 (+546221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.497435 (+876891us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b5400573ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:40.497807 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1254417280}
I20260709 13:03:40.494865  7259 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:40.500946  7409 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.766s	user 0.010s	sys 0.003s
I20260709 13:03:40.504024  1661 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:03:40.510946  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:40.512472  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.558091  1659 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:40.520691  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3044) took 1807 ms. Trace:
I20260709 13:03:40.561201  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2220) took 1246 ms. Trace:
I20260709 13:03:40.526432  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.562832  1665 rpcz_store.cc:276] 0709 13:03:39.314860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.142946 (+828086us) service_pool.cc:224] Handling call
0709 13:03:40.143314 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.143717 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:40.331930 (+188213us) log.cc:844] Serialized 14 byte log entry
0709 13:03:40.560696 (+228766us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:40.560768 (+    72us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:40.560826 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.560912 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.561123 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":253726,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":660,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":20816,"thread_start_us":376232,"threads_started":2}
I20260709 13:03:40.533326  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:40.568647  1591 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:40.541528  1585 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:40.543879  1588 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:40.570374  1588 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:40.571223  1588 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.572132  1588 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:40.428577  7471 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:40.582331  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 71721us
I20260709 13:03:40.579533  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.560827  1590 rpcz_store.cc:276] 0709 13:03:38.713137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.068419 (+355282us) service_pool.cc:224] Handling call
0709 13:03:39.074933 (+  6514us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:39.232786 (+157853us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.232838 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:39.233942 (+  1104us) log.cc:844] Serialized 52 byte log entry
0709 13:03:39.670276 (+436334us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:39.699701 (+ 29425us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.700192 (+   491us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.701915 (+  1723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.702004 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:40.463427 (+761423us) spinlock_profiling.cc:243] Waited 8.91 ms on lock 0x7b18004037e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:40.485667 (+ 22240us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b18004037e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1
0709 13:03:40.486271 (+   604us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b18004037e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:40.520321 (+ 34050us) raft_consensus.cc:1685] finished
0709 13:03:40.520356 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.520600 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4294,"prepare.queue_time_us":88043,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":10684,"raft.queue_time_us":10716,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":29837,"spinlock_wait_cycles":97548032,"thread_start_us":364698,"threads_started":1,"wal-append.queue_time_us":459096}
I20260709 13:03:40.599740  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:40.599927  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:40.599898  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:40.600378  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.610695  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:40.612604  7432 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:40.585035  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.613271  7432 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.614886  7153 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:03:40.614492  7432 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.618144  7415 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:03:40.545034  7361 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.711s	user 0.009s	sys 0.004s
I20260709 13:03:40.626636  1734 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:40.443522  7468 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.652087  7501 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:40.652598  7501 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.654109  7501 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.654757  7468 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.629415  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.657938  1667 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:40.658783  1588 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 13:03:40.660326  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:40.660509  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.615576  7153 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:40.659319  1588 pending_rounds.cc:85] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 9
I20260709 13:03:40.597786  7244 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.604018  7483 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.598318  7409 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.503885  7470 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.641665  7475 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.645581  7451 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:40.447124  7399 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.651800  7011 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.663337  1588 pending_rounds.cc:107] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 13:03:40.604094  7064 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:40.601241  7250 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:40.603675  7428 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:40.688108  1659 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:40.712827  1662 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:40.691475  1588 raft_consensus.cc:2889] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:40.690292  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116207us
I20260709 13:03:40.651906  7137 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.665967  7153 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:40.713171  1738 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:40.672755  7468 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:40.717046  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:40.719589  1588 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:40.719801  1667 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:40.692878  7039 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.552s	user 0.005s	sys 0.006s
W20260709 13:03:40.705911  7247 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.759s	user 0.014s	sys 0.000s
I20260709 13:03:40.645694  7239 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.062s
W20260709 13:03:40.721911  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108071us
I20260709 13:03:40.725062  1741 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:40.691355  7451 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:40.708098  7467 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.004s
I20260709 13:03:40.718451  7428 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.735119  7428 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.735783  7137 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:40.708230  7064 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:40.723549  7509 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.002s	sys 0.000s
I20260709 13:03:40.656937  7361 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.741760  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.664705  7479 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.749179  1588 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:03:40.750516  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:40.751396  7494 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.005s
I20260709 13:03:40.682346  7483 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.756223  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:40.724797  7248 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 13:03:40.757885  1736 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:40.744010  7064 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.656186  1487 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 5 to 6, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:40.760641  1660 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:40.773029  7445 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:40.746737  7471 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.773844  7445 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:40.743005  7262 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 13:03:40.778808  7262 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:40.740633  7247 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.753176  7494 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.786047  7262 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:40.787569  7262 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.758669  7468 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:40.746457  7259 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 13:03:40.742169  7467 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.785768  7445 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:40.798470  1668 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:40.797894  7468 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:40.803081  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.799587  7505 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.808564  1667 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:40.761116  7249 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
W20260709 13:03:40.811558  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.818336  7399 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.819311  7467 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.817866  7468 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.821183  7479 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.825611  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.826074  7474 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.826551  7452 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.827605  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.830750  7475 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.834388  1487 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 8, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 8 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:03:40.835559  7505 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.839828  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.829118  7452 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.840082  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.843073  1734 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:40.811249  1659 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:40.843750  1667 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:03:40.853731  7287 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:40.855187  7287 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:40.855587  7287 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:40.856741  7361 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:40.872769  2312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 540) took 1827 ms (client timeout 2233 ms). Trace:
I20260709 13:03:40.858944  7247 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.861788  1659 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:40.863891  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:40.876209  1585 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:40.861814  7153 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:40.877099  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2905) took 1047 ms. Trace:
I20260709 13:03:40.877622  1661 rpcz_store.cc:276] 0709 13:03:39.829934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.169406 (+339472us) service_pool.cc:224] Handling call
0709 13:03:40.169884 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:40.575152 (+405268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.876774 (+301622us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54001a6dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:40.877032 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767761792}
I20260709 13:03:40.869201  7494 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.879074  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:40.879170  7153 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:40.881765  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.870482  7480 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:40.882512  7480 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:40.882627  2314 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:40.857657  7287 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:40.866627  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 918) took 1561 ms. Trace:
I20260709 13:03:40.871418  7470 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:40.873457  2312 rpcz_store.cc:269] 0709 13:03:39.045210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.045369 (+   159us) service_pool.cc:224] Handling call
0709 13:03:40.872543 (+1827174us) spinlock_profiling.cc:243] Waited 698 ms on lock 0x7b54002e8454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:40.872698 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1605314816}
I20260709 13:03:40.886529  7287 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:40.886062  1584 rpcz_store.cc:276] 0709 13:03:39.305552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.507293 (+201741us) service_pool.cc:224] Handling call
0709 13:03:39.792298 (+285005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.792506 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.792529 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.792570 (+    41us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:03:39.792597 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:39.792649 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:40.866347 (+1073698us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54001c4650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:40.866533 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":654781696}
I20260709 13:03:40.871814  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 969) took 1151 ms. Trace:
I20260709 13:03:40.890528  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.894223  7279 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.870999  7483 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:40.890525  1737 rpcz_store.cc:276] 0709 13:03:39.720148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.736407 (+ 16259us) service_pool.cc:224] Handling call
0709 13:03:40.871592 (+1135185us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b540012d9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:40.871742 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818417408}
I20260709 13:03:40.896795  7250 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:40.897446  7250 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.898123  1487 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:40.900131  1487 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:40.903800  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.904258  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.906400  2313 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:40.908087  7409 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:40.909370  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.910125  1661 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:40.801792  7262 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:40.912187  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 910) took 1777 ms. Trace:
I20260709 13:03:40.919229  2309 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 13:03:40.917106  1584 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 13:03:40.912648  1593 rpcz_store.cc:276] 0709 13:03:39.135148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.502244 (+367096us) service_pool.cc:224] Handling call
0709 13:03:39.503173 (+   929us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.026476 (+523303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:40.911858 (+885382us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b540042e354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:40.912113 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1201968512}
I20260709 13:03:40.923275  1661 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:40.927824  1661 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:40.928503  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:40.929003  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.921036  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.930831  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:40.923964  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:40.942430  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:40.942718  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:40.943348  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.948099  1584 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:40.948479  2309 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
I20260709 13:03:40.947211  1734 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:03:40.980711  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:40.984097  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:40.984433  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:40.984314  1735 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:40.986259  1740 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:40.987910  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:40.985487  1734 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
W20260709 13:03:40.992679  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64337.4us
W20260709 13:03:40.993857  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:40.998736  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:40.986281  1737 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
W20260709 13:03:41.010263  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.013351  7452 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:41.013265  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89447.6us
I20260709 13:03:41.014640  7452 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.016734  7239 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:41.016968  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:41.018460  7467 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:41.018812  1632 leader_election.cc:400] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
I20260709 13:03:41.019250  1632 leader_election.cc:403] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 13:03:41.017740  7452 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.020270  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:41.021028  7467 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
I20260709 13:03:41.021255  7239 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:41.015239  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:41.026202  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:41.026960  7505 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:41.027652  2312 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:41.029680  1585 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:41.026247  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:41.035449  1737 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:41.028280  7505 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.006480  7153 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:41.043453  1740 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:41.033478  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:41.043823  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:41.049290  1740 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.049660  1659 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:41.044571  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 541) took 1998 ms. Trace:
I20260709 13:03:41.050631  2318 rpcz_store.cc:276] 0709 13:03:39.046605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.074058 (+ 27453us) service_pool.cc:224] Handling call
0709 13:03:39.447898 (+373840us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b5800195c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:03:39.448136 (+   238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:39.501797 (+ 53661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:39.501833 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:39.501877 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:39.501905 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.044320 (+1542415us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b540061dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:41.044377 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.044511 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126386816}
I20260709 13:03:41.052878  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:41.053032  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:41.010644  7396 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:41.054152  1664 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:41.054524  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:41.031039  7483 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:41.050331  1740 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.000355  7247 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 13:03:41.069027  1740 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.040345  7239 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.046262  7035 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:41.087092  7501 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:41.100167  1735 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:03:41.102789  7247 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.103530  7430 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:41.103856  7247 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.104574  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 862) took 1071 ms. Trace:
I20260709 13:03:41.104390  7064 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.105783  1739 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
I20260709 13:03:41.047719  7471 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.106294  1487 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:41.107149  7471 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.105060  2316 rpcz_store.cc:276] 0709 13:03:40.033187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.033888 (+   701us) service_pool.cc:224] Handling call
0709 13:03:41.104214 (+1070326us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:41.104496 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503945984}
I20260709 13:03:41.064435  7011 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:41.102912  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:41.115450  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2964) took 1326 ms. Trace:
I20260709 13:03:41.116103  7507 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:41.115996  1587 rpcz_store.cc:276] 0709 13:03:39.788731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.788881 (+   150us) service_pool.cc:224] Handling call
0709 13:03:39.789271 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:41.114970 (+1325699us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:41.115013 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:41.115075 (+    62us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:41.115120 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.115207 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.115370 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1325175}
W20260709 13:03:41.118834  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.119359  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.119626  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.119899  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.120158  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.120754  7507 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 13:03:41.121932  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3043) took 2409 ms. Trace:
I20260709 13:03:41.122511  7397 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:41.123266  7397 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.122468  1592 rpcz_store.cc:276] 0709 13:03:38.712834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.064796 (+351962us) service_pool.cc:224] Handling call
0709 13:03:39.065194 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:39.065697 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:39.084983 (+ 19286us) log.cc:844] Serialized 14 byte log entry
0709 13:03:39.504266 (+419283us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b18000a3e40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:40.109239 (+604973us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b18000a3e40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:03:41.118333 (+1009094us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:41.118959 (+   626us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.119160 (+   201us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:41.119201 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.119278 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.121692 (+  2414us) raft_consensus.cc:1685] finished
0709 13:03:41.121714 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.121866 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":850634,"prepare.queue_time_us":19781,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":861662,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"spinlock_wait_cycles":37335552,"thread_start_us":158043,"threads_started":1,"wal-append.queue_time_us":1266963}
I20260709 13:03:41.122017  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:41.126456  7381 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:03:41.127928  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.129163  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2898) took 2460 ms. Trace:
I20260709 13:03:41.130259  1668 rpcz_store.cc:276] 0709 13:03:38.668700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.064392 (+1395692us) service_pool.cc:224] Handling call
0709 13:03:40.065258 (+   866us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.801596 (+736338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.128803 (+327207us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:41.129060 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1685810432}
I20260709 13:03:41.131112  1733 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:41.133316  1557 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
I20260709 13:03:41.129946  7239 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.126921  7381 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:41.136216  7381 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:41.139457  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.142575  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2274) took 1290 ms. Trace:
I20260709 13:03:41.146721  1738 rpcz_store.cc:276] 0709 13:03:39.852555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.858487 (+  5932us) service_pool.cc:224] Handling call
0709 13:03:41.142141 (+1283654us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b5400435154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:41.142484 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959162112}
I20260709 13:03:41.133337  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:41.142737  1742 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:41.148483  1659 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:41.143599  7416 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 13:03:41.152932  7397 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.046483  7399 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.090s
I20260709 13:03:41.157227  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:41.155165  1557 leader_election.cc:403] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:41.197600  1584 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:41.188735  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:41.193324  1668 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:41.200857  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:41.153623  7528 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:03:41.197104  7385 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:41.137455  7239 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.108785  1487 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:41.240221  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.205066  7478 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:41.204898  7471 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.148022  7467 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.083s
W20260709 13:03:41.273488  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 219152us
I20260709 13:03:41.197696  7520 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.205725  7011 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:41.307242  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80338.9us
I20260709 13:03:41.265868  7476 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:41.313195  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:41.260953  7235 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.231190  7483 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.313258  1740 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 13:03:41.334421  1740 pending_rounds.cc:85] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 7
I20260709 13:03:41.334741  1740 pending_rounds.cc:107] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 13:03:41.335139  1740 raft_consensus.cc:2889] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:41.328447  1659 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:03:41.329519  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.280521  7451 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.528s	user 0.006s	sys 0.006s
I20260709 13:03:41.223301  6485 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
I20260709 13:03:41.311793  7478 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:41.355997  7500 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.796s	user 0.019s	sys 0.028s
I20260709 13:03:41.210784  7355 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.127s
W20260709 13:03:41.365561  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.333436  7476 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.257720  7505 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.376024  1487 catalog_manager.cc:5697] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 1 to 7, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:41.365084  7247 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.322235  7480 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.425148  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 343102us
W20260709 13:03:41.369938  7500 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.020s	sys 0.029s
I20260709 13:03:41.342312  7235 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:41.412809  7468 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.025s	sys 0.072s
I20260709 13:03:41.349023  7451 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.346202  7483 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.358887  7470 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 13:03:41.376641  7491 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 13:03:41.426139  7287 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.005s
I20260709 13:03:41.360759  7137 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 13:03:41.358443  7487 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.114s
I20260709 13:03:41.331264  7520 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.423832  7064 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:41.460295  7468 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.027s	sys 0.072s
I20260709 13:03:41.429792  7035 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.366793  7250 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.376262  7494 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.090s
W20260709 13:03:41.467659  7262 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.557s	user 0.013s	sys 0.000s
I20260709 13:03:41.466362  7458 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:41.487421  7458 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.488652  7458 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.490181  7534 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:03:41.490427  7458 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.491197  7381 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.479740  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.470600  7445 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.631s	user 0.010s	sys 0.003s
I20260709 13:03:41.437588  7452 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 13:03:41.483034  7287 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 13:03:41.498057  2310 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:41.435113  7011 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.476389  7478 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.497273  7445 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.475210  7476 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.510542  7476 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.493429  7480 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:41.511032  7483 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.499961  7520 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:41.519475  7250 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.496907  7381 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:41.516836  7445 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:41.500555  7235 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:41.527952  7451 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.499078  7533 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:41.529546  7533 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.534865  7533 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.540234  7533 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:41.419502  1487 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:03:41.593951  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:41.598516  1661 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:41.599932  1734 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:41.600620  1734 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.598659  1665 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:41.611322  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2896) took 4000 ms. Trace:
I20260709 13:03:41.612000  1660 rpcz_store.cc:276] 0709 13:03:37.610589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.791117 (+2180528us) service_pool.cc:224] Handling call
0709 13:03:39.792073 (+   956us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.761945 (+969872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.610527 (+848582us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:41.611012 (+   485us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2227221888}
I20260709 13:03:41.611017  1734 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.640185  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:41.642755  7467 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.643663  7468 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:41.644482  7467 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.645053  7468 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.654381  7476 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:41.655983  7476 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.496237  7035 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.662189  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:41.663055  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:41.665118  2318 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:41.665938  1740 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:41.666393  1740 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.665154  1738 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:03:41.668126  1740 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:41.624416  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 286589us
I20260709 13:03:41.695218  7239 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:03:41.695971  7239 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:41.674295  1740 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:41.678121  7409 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:41.616076  1487 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:03:41.680822  7524 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.183s	user 0.003s	sys 0.000s
I20260709 13:03:41.685370  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:41.688704  7468 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.690112  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:41.695863  1734 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:41.654304  7467 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.709033  7409 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:41.720305  7541 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 13:03:41.722489  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:41.722183  7239 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.723402  7542 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 13:03:41.723922  7239 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:41.726002  7240 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:41.726408  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 961) took 1223 ms (client timeout 1617 ms). Trace:
I20260709 13:03:41.726742  7240 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:41.726914  1662 rpcz_store.cc:269] 0709 13:03:40.502805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.518036 (+ 15231us) service_pool.cc:224] Handling call
0709 13:03:41.725879 (+1207843us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:41.726177 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240618880}
I20260709 13:03:41.730909  7476 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:41.731637  7539 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.731196  7478 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.732506  7539 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.733044  7478 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.735256  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:41.736443  1738 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:41.739033  7409 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.741585  7478 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:41.741989  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.743309  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:41.743597  7467 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:03:41.728895  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.744607  1733 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:41.744773  7467 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.747850  7468 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.747711  1665 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:41.728376  7240 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.748802  7548 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.125s	user 0.002s	sys 0.000s
I20260709 13:03:41.750566  1487 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 4 to 8, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 8 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:41.715214  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 345435us
I20260709 13:03:41.716547  7540 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:41.776355  7507 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.778370  7539 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:41.783324  7483 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:41.784102  7283 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.785533  7467 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:41.783425  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.770048  7540 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.790905  7461 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:41.800652  7430 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:03:41.801422  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.803118  7501 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:41.805126  7494 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:41.805631  7494 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.809587  7494 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:41.811061  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.88057e+06us
I20260709 13:03:41.811349  7494 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
W20260709 13:03:41.812891  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:41.816253  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.817279  7407 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:41.818982  1584 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:03:41.818307  7407 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.819406  1584 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:41.809840  7468 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.812629  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:41.822789  1584 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:41.817456  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.811676  7540 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.827266  7468 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.827412  7540 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:41.828405  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:41.828596  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:41.828894  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:41.830273  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:41.830716  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:41.831254  7480 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:41.832757  7480 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:41.833148  7480 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:41.839936  7441 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:41.840346  1734 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 13:03:41.844185  1734 pending_rounds.cc:85] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:03:41.835211  7266 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:41.830154  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:41.846490  1591 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:41.846632  1592 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:41.836819  7416 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:41.838116  1584 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:41.840485  7441 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:41.841603  7451 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.841447  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:41.839782  7480 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.842037  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:41.806665  7461 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:41.842491  7457 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.843667  7344 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:41.843824  1593 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:41.834990  1665 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:41.777236  7507 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:41.834089  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 706559us
I20260709 13:03:41.849546  7477 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:41.845346  7266 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.856751  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2983) took 1020 ms. Trace:
I20260709 13:03:41.850471  1734 pending_rounds.cc:107] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 13:03:41.870393  1734 raft_consensus.cc:2889] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:41.858672  7441 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.874135  7461 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.874948  1661 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:41.863273  7344 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:41.875667  1659 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
W20260709 13:03:41.887785  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 75456.7us
I20260709 13:03:41.866811  7477 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:41.897320  7483 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.898976  7416 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:41.911680  7483 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.912545  7416 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:41.913148  7416 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:41.899492  7480 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:41.915153  7483 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.915781  7266 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:41.903438  7472 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:41.904270  7396 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:41.911675  7485 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
I20260709 13:03:41.922190  7451 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.926690  7344 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.929417  7477 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.931300  7477 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.906123  7461 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:41.906522  7457 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:41.882836  7507 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.943969  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2276) took 2073 ms. Trace:
W20260709 13:03:41.944427  7515 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 13:03:41.944911  7451 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:41.945210  7472 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:41.948788  1738 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:03:41.952768  7543 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.383s	user 0.001s	sys 0.004s
I20260709 13:03:41.953459  7479 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:41.869268  1590 rpcz_store.cc:276] 0709 13:03:40.833885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.834087 (+   202us) service_pool.cc:224] Handling call
0709 13:03:40.834526 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:41.126564 (+292038us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:41.346333 (+219769us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.845611 (+499278us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:41.846398 (+   787us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.846563 (+   165us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:41.846627 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.846720 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.849873 (+  3153us) raft_consensus.cc:1685] finished
0709 13:03:41.849902 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:41.854541 (+  4639us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1214452,"prepare.queue_time_us":335845,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":518630,"raft.run_cpu_time_us":411,"raft.run_wall_time_us":3233,"thread_start_us":495353,"threads_started":2,"wal-append.queue_time_us":276540}
W20260709 13:03:41.956677  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 93332.8us
W20260709 13:03:41.962312  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59675.8us
I20260709 13:03:41.918135  1487 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:03:41.966017  7536 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.474s	user 0.005s	sys 0.000s
I20260709 13:03:41.968288  7457 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:41.969970  7248 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:41.970455  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:41.971158  7344 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:41.971968  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:41.971884  7532 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:41.919574  7396 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:41.973879  7407 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:41.976588  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.980916  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:41.976289  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:41.985559  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:41.986917  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:41.980928  1741 rpcz_store.cc:276] 0709 13:03:39.865237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:39.866433 (+  1196us) service_pool.cc:224] Handling call
0709 13:03:39.892345 (+ 25912us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:40.729037 (+836692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.934113 (+1205076us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:41.938956 (+  4843us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909005056}
W20260709 13:03:41.986569  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105849us
I20260709 13:03:41.987146  7472 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:42.004498  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.011394  1592 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:42.015816  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.018934  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.020237  7407 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:42.020714  7407 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:42.021188  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.022099  7407 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.023931  7407 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.024681  7396 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.026403  7451 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:42.026621  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.026962  7451 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.027381  7451 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.028115  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.028805  1593 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:42.035219  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 85132.4us
I20260709 13:03:42.025929  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.021201  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:42.044836  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.045729  2316 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:42.046285  1734 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:42.046114  7457 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:42.047511  1735 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 13:03:42.045882  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.056602  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:42.057797  2312 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:42.058449  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.059717  7415 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:42.061090  7415 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:42.062081  7415 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:42.064702  7489 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:42.065308  7489 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:42.066380  7489 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.067993  7489 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:42.069012  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.069823  1590 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:42.071900  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2982) took 1277 ms. Trace:
I20260709 13:03:42.072331  1588 rpcz_store.cc:276] 0709 13:03:40.794197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.794383 (+   186us) service_pool.cc:224] Handling call
0709 13:03:40.794759 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:41.124150 (+329391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:41.124190 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:41.124833 (+   643us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.356581 (+231748us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:41.356682 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.356789 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:41.955276 (+598487us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:42.071328 (+116052us) raft_consensus.cc:1685] finished
0709 13:03:42.071352 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:42.071694 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":330990,"raft.queue_time_us":330337,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"spinlock_wait_cycles":33810816,"thread_start_us":231251,"threads_started":1,"wal-append.queue_time_us":337908}
I20260709 13:03:42.081640  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.083860  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.096596  1664 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:42.096222  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:42.100514  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63550.9us
I20260709 13:03:42.086220  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.105589  7472 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.114574  7472 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:42.114836  1738 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:03:42.116981  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:42.119508  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:42.016669  7532 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.122038  1737 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:42.123883  1593 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:42.129959  1632 leader_election.cc:400] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14.
W20260709 13:03:42.131238  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47514.3us
I20260709 13:03:42.121488  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.039479  7501 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 13:03:42.151762  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.165087  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.154101  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.154125  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.155846  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.147693  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.121847  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:42.167737  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46352.8us
I20260709 13:03:42.169480  1592 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:03:42.169776  1590 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 6.
I20260709 13:03:42.148118  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.149019  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.175489  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.123294  7415 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.182571  1739 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 13:03:42.183053  1739 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 10
I20260709 13:03:42.183312  1739 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 13:03:42.183552  1739 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:42.171885  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:42.185993  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 87131.7us
I20260709 13:03:42.188313  7550 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:03:42.194025  7381 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:42.196732  7457 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:42.199347  1734 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260709 13:03:42.200524  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 122529us
I20260709 13:03:42.202710  7457 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:42.220714  7457 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:42.221397  7457 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:42.227759  1592 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:03:42.235715  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 2974) took 1705 ms. Trace:
I20260709 13:03:42.237131  7532 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.209792  1742 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:42.206297  7557 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 13:03:42.121968  7485 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.092s
I20260709 13:03:42.155267  7430 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:42.214033  1664 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:42.215304  1660 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:42.189175  7550 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:42.217351  7556 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.174s	user 0.002s	sys 0.000s
I20260709 13:03:42.218218  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:42.219928  7501 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.235605  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:42.131493  7505 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:42.240228  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.266367  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.269589  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.270717  1591 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:42.175102  7240 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:03:42.274201  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94875.9us
W20260709 13:03:42.274590  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.178656  7409 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:42.260825  7540 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:42.279976  1659 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:42.236246  1586 rpcz_store.cc:276] 0709 13:03:40.530060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:40.530268 (+   208us) service_pool.cc:224] Handling call
0709 13:03:40.530678 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:41.125784 (+595106us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:41.355077 (+229293us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.804858 (+449781us) spinlock_profiling.cc:243] Waited 8.89 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:41.812051 (+  7193us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:41.813115 (+  1064us) log.cc:844] Serialized 26 byte log entry
0709 13:03:41.813407 (+   292us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:41.814056 (+   649us) log.cc:844] Serialized 14 byte log entry
0709 13:03:41.814291 (+   235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:41.814454 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:42.235264 (+420810us) raft_consensus.cc:1685] finished
0709 13:03:42.235290 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:42.235466 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":817562,"prepare.queue_time_us":310964,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":118,"raft.queue_time_us":875141,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":430,"spinlock_wait_cycles":20439424,"thread_start_us":461973,"threads_started":2,"wal-append.queue_time_us":139978}
I20260709 13:03:42.289441  7550 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 11.12, Last appended by leader: 12, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:42.291235  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57653.2us
I20260709 13:03:42.276965  7240 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:42.280543  7239 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.556s	user 0.016s	sys 0.002s
I20260709 13:03:42.327067  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:42.327606  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.329296  7240 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.281301  7505 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:42.287613  7540 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.288146  7532 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:42.332322  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:42.269558  7416 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.272465  7381 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.049s
W20260709 13:03:42.340878  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.342214  1590 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:42.344123  1741 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:42.337147  1585 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:42.356119  1590 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:42.292920  7479 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 13:03:42.367241  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.301379  7501 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.332407  7507 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:42.352421  7532 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:42.352062  7540 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.316494  7344 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.046s
W20260709 13:03:42.375141  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 245530us
I20260709 13:03:42.336874  7477 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:42.382908  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147431us
W20260709 13:03:42.377599  7494 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.560s	user 0.016s	sys 0.002s
I20260709 13:03:42.389008  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:42.410482  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81297.9us
I20260709 13:03:42.411908  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:42.415026  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:42.416278  1659 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:42.396965  7409 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.418367  7409 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:42.422282  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.430507  1664 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 13:03:42.433308  1590 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.369405  7239 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:42.434628  7239 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.434983  7250 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:42.435205  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:42.438167  7248 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:42.438632  7561 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:03:42.429953  7533 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:42.434644  1590 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:42.382609  7532 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:42.440882  7409 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:42.441051  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167777us
I20260709 13:03:42.447507  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:42.450294  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.450572  7485 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:42.440156  7557 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:03:42.451917  1665 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:03:42.450996  7560 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 13:03:42.454917  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:42.445937  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.683s (SENT)
I20260709 13:03:42.433068  7479 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:03:42.485873  1660 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:42.490309  7239 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.499121  7035 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:42.504194  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.510886  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.513985  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 63952.4us
I20260709 13:03:42.515007  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:42.516086  7441 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.578s	user 0.008s	sys 0.005s
I20260709 13:03:42.516525  7441 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.517805  1587 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:42.520836  7505 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.522034  7501 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:42.523954  7416 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.517254  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:42.545038  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.546551  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.567227  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.568039  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.482128  7480 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:42.601547  1659 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 13:03:42.605906  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 274110us
W20260709 13:03:42.584406  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112818us
W20260709 13:03:42.571308  7451 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.543s	user 0.002s	sys 0.012s
W20260709 13:03:42.614374  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.614446  1590 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
W20260709 13:03:42.622596  7569 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 13:03:42.618031  1590 pending_rounds.cc:85] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:42.623641  1590 pending_rounds.cc:107] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 16.9
W20260709 13:03:42.624325  7563 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.164s	user 0.000s	sys 0.004s
I20260709 13:03:42.625255  1590 raft_consensus.cc:2889] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:42.632043  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.658872  7489 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.663590  7441 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.610184  7468 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:42.625147  7501 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.673120  1660 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:42.609642  7239 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
W20260709 13:03:42.687772  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 170630us
I20260709 13:03:42.688771  7035 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:42.690691  7494 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 13:03:42.693204  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 293816us
W20260709 13:03:42.694947  7570 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:03:42.696090  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:42.697743  7416 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:42.699074  7416 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:42.699784  7416 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:42.700188  7416 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:42.704938  7461 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:42.705808  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:42.706648  1590 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:42.710369  7344 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:42.717101  1590 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:03:42.718016  1590 pending_rounds.cc:85] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:03:42.718477  1590 pending_rounds.cc:107] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:03:42.718979  1590 raft_consensus.cc:2889] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:42.722118  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:42.725966  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.726794  7035 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:42.726948  7501 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.727902  7407 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.728560  7407 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.635258  7451 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.730773  1588 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:42.731981  7451 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.732955  7507 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:42.669194  7480 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:42.733734  7407 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.664206  7489 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.737378  7035 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:42.740464  7489 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:42.693387  7573 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 13:03:42.741789  7507 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.746232  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:42.747802  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.706s (SENT)
I20260709 13:03:42.760149  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:42.766594  7501 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:42.767643  7416 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:42.782634  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 413706us
I20260709 13:03:42.771725  7461 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.774794  7407 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.780431  7240 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:42.770848  7457 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.549s	user 0.008s	sys 0.005s
W20260709 13:03:42.791811  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:42.788666  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.822404  1660 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:03:42.819022  7480 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:42.826345  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80333.3us
I20260709 13:03:42.841363  1661 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:03:42.842097  1664 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 13:03:42.821326  7441 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:03:42.846884  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 99178.3us
I20260709 13:03:42.814392  7248 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:42.849179  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89216.4us
I20260709 13:03:42.809330  7461 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.820973  7381 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 13:03:42.817987  7457 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.843300  7501 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:42.858662  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.844580  7494 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 13:03:42.872651  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 416658us
I20260709 13:03:42.866374  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:42.883777  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.884138  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 495709us
I20260709 13:03:42.839226  7480 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:42.882834  2316 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:42.840307  7407 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.890309  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:42.892338  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:42.884893  1708 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.713s (SENT)
I20260709 13:03:42.874248  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:42.898010  1593 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:42.891263  2309 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:42.897450  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:42.906740  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:42.908066  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 447178us
I20260709 13:03:42.910046  1668 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:42.910434  1661 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 13:03:42.911015  7240 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.914042  1661 pending_rounds.cc:85] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 13
I20260709 13:03:42.918684  1661 pending_rounds.cc:107] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 13:03:42.919037  1661 raft_consensus.cc:2889] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:42.935875  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:42.942858  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47929us
I20260709 13:03:42.919037  7494 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:42.931948  7430 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:42.952124  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:42.954630  7430 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:42.955579  7430 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:42.955648  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:42.956632  1663 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:42.957619  7240 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:42.961998  1737 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:42.962913  7480 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.970314  7250 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:42.970499  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3001) took 1003 ms. Trace:
I20260709 13:03:42.974488  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.975783  7407 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:42.975939  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.976678  7507 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:42.979487  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.980085  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.979609  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.982355  1589 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:42.991958  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:42.992929  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 128624us
I20260709 13:03:42.937768  7489 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:42.973963  1592 rpcz_store.cc:276] 0709 13:03:41.966706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.228985 (+262279us) service_pool.cc:224] Handling call
0709 13:03:42.229364 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.280680 (+ 51316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.280717 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:42.328965 (+ 48248us) log.cc:844] Serialized 26 byte log entry
0709 13:03:42.789265 (+460300us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:42.789320 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:42.789397 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:42.969477 (+180080us) raft_consensus.cc:1685] finished
0709 13:03:42.970139 (+   662us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:42.970405 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":417115,"threads_started":1,"wal-append.queue_time_us":581014}
I20260709 13:03:42.998234  7239 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:42.997809  7248 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:42.999585  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.000243  7550 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.000633  1589 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:43.001473  7396 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:43.003887  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.014909  1591 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 10: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:43.013888  1589 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:43.003957  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.021960  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:42.974148  7416 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:43.022578  1668 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:43.023962  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.020547  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:43.025803  1587 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:42.994045  7485 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:43.028040  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102822us
I20260709 13:03:43.016185  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.005051  7489 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:43.030752  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.756s (SENT)
I20260709 13:03:43.028719  1584 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:42.926446  7532 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.032773  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.034160  1667 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:43.030238  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.037918  1591 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:43.039700  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:43.056560  1660 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:43.057852  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.049901  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.034261  1589 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.062393  1589 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:43.025626  7580 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:43.040932  7489 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:43.081920  1593 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:03:43.076292  1586 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:43.089413  1584 pending_rounds.cc:85] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:03:43.089998  1584 pending_rounds.cc:107] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 13:03:43.090313  1584 raft_consensus.cc:2889] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:43.056928  7507 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:43.069691  7540 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.003s
I20260709 13:03:43.031653  7239 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.099959  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.103415  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.032459  7550 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.064868  7582 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:43.105690  1592 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:43.108419  7584 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:43.113291  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:43.112090  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.116482  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.123867  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.097735  7457 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:43.129345  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 270785us
I20260709 13:03:43.129011  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:43.132532  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.133977  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.136950  7590 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:43.138749  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116272us
I20260709 13:03:43.106325  7461 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:43.115974  7584 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.135493  1664 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:03:43.163195  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103191us
W20260709 13:03:43.163203  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 104045us
W20260709 13:03:43.174953  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.180998  1667 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 13:03:43.183630  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.187575  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.105049  7540 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.191054  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:43.192440  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 158151us
I20260709 13:03:43.194021  1661 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:43.193725  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:43.183705  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:43.200345  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:43.146118  7584 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:43.215443  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.219167  1664 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 13:03:43.223291  1661 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:03:43.225167  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.233353  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:43.235574  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55799.6us
I20260709 13:03:43.236022  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:43.235607  2318 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:43.219803  7494 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:03:43.259150  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.251184  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:43.220029  7461 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:43.209331  7595 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:43.266870  7576 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:43.269095  2316 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:43.270707  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.238456  7479 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.280412  1592 raft_consensus.cc:2368] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
I20260709 13:03:43.272861  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:43.285578  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.236981  7505 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.287453  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:43.290113  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93914.1us
W20260709 13:03:43.298192  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47475.5us
I20260709 13:03:43.302207  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:43.308243  1740 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:43.279915  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.317822  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.268488  7461 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:43.268678  7595 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.289418  7485 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:43.318692  1592 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:03:43.327243  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 133997us
I20260709 13:03:43.253993  7478 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 13:03:43.330377  1586 raft_consensus.cc:1217] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Deduplicated request from leader. Original: 10.6->[11.7-13.8]   Dedup: 13.8->[]
I20260709 13:03:43.330727  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:43.333066  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.336812  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:43.277890  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 67934.9us
W20260709 13:03:43.343469  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:43.344705  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.343641  1586 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [11.7-13.8]
I20260709 13:03:43.331058  7595 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:43.342586  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.348095  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.348284  1667 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:43.364996  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.322775  7461 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:43.344329  7396 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.384171  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:43.328809  7485 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:43.317958  7480 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 13:03:43.300539  7479 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:43.422122  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.385066  1737 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:43.303638  7505 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.421587  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:43.450434  1591 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:43.451826  7583 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
W20260709 13:03:43.452690  7608 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 13:03:43.442516  7550 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:43.457851  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.458539  1592 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:43.458772  7550 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:43.459748  7611 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:43.460198  7611 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.460551  7611 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:43.462344  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.463836  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:43.464757  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:43.466002  7585 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
W20260709 13:03:43.467299  7599 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 13:03:43.471881  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.473178  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:43.474094  1739 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:43.475822  1739 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:03:43.477778  7396 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.480654  7396 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:43.481664  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58055.6us
W20260709 13:03:43.481986  7605 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
W20260709 13:03:43.467442  7588 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 13:03:43.485364  1736 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260709 13:03:43.488472  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.501207  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:43.501598  1593 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:43.502938  1593 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 13:03:43.507719  7441 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:43.508157  1593 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:43.508250  7441 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:43.508740  7441 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:43.482007  7609 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.509629  1593 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 13:03:43.518044  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:43.375751  7035 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.520573  1666 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [16.13-17.14]
I20260709 13:03:43.511924  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:43.390470  7533 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.122s
I20260709 13:03:43.524016  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:43.659559  1737 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:43.549451  1738 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 8. (term mismatch)
I20260709 13:03:43.661345  1738 pending_rounds.cc:85] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 7
I20260709 13:03:43.661926  1738 pending_rounds.cc:107] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 13:03:43.662274  1738 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:43.655186  1584 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:43.664539  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.665534  1584 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:43.656654  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 354570us
W20260709 13:03:43.547842  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 246927us
W20260709 13:03:43.680150  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 168344us
I20260709 13:03:43.680217  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:43.520110  7501 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.652s	user 0.003s	sys 0.009s
W20260709 13:03:43.690869  1589 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.628s	user 0.008s	sys 0.006s
I20260709 13:03:43.509690  7532 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:43.515862  7248 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.447567  7485 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:43.515691  7468 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:03:43.622974  7240 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.665s	user 0.011s	sys 0.004s
I20260709 13:03:43.606417  7606 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:43.706017  1589 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:03:43.509436  7584 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:43.723616  7583 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.725309  7578 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:43.725641  7627 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.726105  7491 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.304s
I20260709 13:03:43.705600  7501 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:43.729195  1661 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 13:03:43.727362  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2948) took 1244 ms. Trace:
W20260709 13:03:43.734719  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67367us
I20260709 13:03:43.737401  1585 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:43.728397  1593 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:43.522055  7609 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.730459  7468 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.770093  1589 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 13:03:43.779145  1589 pending_rounds.cc:85] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:43.779496  1589 pending_rounds.cc:107] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 13:03:43.779796  1589 raft_consensus.cc:2889] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:43.550885  7457 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.220s
I20260709 13:03:43.738287  7240 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:43.771965  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.734230  7607 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.785861  1663 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:43.764817  1662 rpcz_store.cc:276] 0709 13:03:42.482781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.482951 (+   170us) service_pool.cc:224] Handling call
0709 13:03:42.487379 (+  4428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.582911 (+ 95532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.582968 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:42.596811 (+ 13843us) log.cc:844] Serialized 26 byte log entry
0709 13:03:42.916497 (+319686us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:42.916556 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:42.916650 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:43.727105 (+810455us) raft_consensus.cc:1685] finished
0709 13:03:43.727133 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:43.727297 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53236,"raft.queue_time_us":213826,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":319190,"threads_started":1,"wal-append.queue_time_us":491307}
W20260709 13:03:43.765972  7430 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.810s	user 0.016s	sys 0.000s
I20260709 13:03:43.788404  1734 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:43.739584  7578 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:43.734157  7381 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 13:03:43.689919  7035 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:43.797534  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 117441us
I20260709 13:03:43.786481  7468 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:43.804003  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:43.811451  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:43.803233  7416 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.774s	user 0.012s	sys 0.002s
I20260709 13:03:43.812031  1593 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:03:43.742329  7248 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:43.745584  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 436376us
I20260709 13:03:43.829457  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:43.797288  7578 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:43.834769  1585 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:43.839498  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:43.840106  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.800590  7581 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.846005  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.850023  1587 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:43.803927  7415 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:43.853328  1589 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:43.784232  7574 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 13:03:43.860366  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:43.763404  7627 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:43.784076  7583 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:43.776935  7606 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.836066  7478 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:43.859126  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.877264  1593 raft_consensus.cc:2368] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16.
I20260709 13:03:43.878851  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:43.860495  7489 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.764s	user 0.008s	sys 0.006s
I20260709 13:03:43.798672  7430 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.846918  7628 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:43.864612  7582 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.726s	user 0.008s	sys 0.005s
I20260709 13:03:43.836520  7479 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:43.798462  7607 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.907055  1666 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:03:43.887137  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 383599us
W20260709 13:03:43.887888  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 126692us
I20260709 13:03:43.882174  7478 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:03:43.913997  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:43.869961  7533 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 13:03:43.910951  1593 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:43.939888  1589 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:43.927500  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:43.947031  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:43.864451  7415 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:43.950521  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.870222  7581 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:43.908095  7582 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:43.953431  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:43.955588  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.888638  7489 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:43.911659  7505 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:43.962756  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:43.926107  7478 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:44.013875  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 148892us
W20260709 13:03:44.019907  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57241.1us
I20260709 13:03:44.018570  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:44.016290  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:43.924549  7631 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.084s
W20260709 13:03:43.966557  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 201402us
I20260709 13:03:43.953998  7597 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 13:03:43.925518  7625 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 13:03:44.065642  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.014043  7628 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:44.023737  7640 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.024560  7584 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:44.068339  7580 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.981s	user 0.009s	sys 0.005s
W20260709 13:03:44.067468  7611 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.607s	user 0.009s	sys 0.004s
W20260709 13:03:44.086663  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.681s (SENT) [suppressed 22 similar messages]
I20260709 13:03:44.062433  7609 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.136204  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:44.134519  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47900.9us
W20260709 13:03:44.139259  1592 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.680s	user 0.014s	sys 0.000s
W20260709 13:03:44.139804  1741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.030s	sys 0.054s
I20260709 13:03:44.139896  1592 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:44.140277  1741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.030s	sys 0.055s
I20260709 13:03:44.143481  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:44.087595  7035 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:03:44.143555  7643 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.168886  7643 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:44.157804  7624 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.237s	user 0.001s	sys 0.000s
W20260709 13:03:44.171291  7250 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.200s	user 0.009s	sys 0.005s
I20260709 13:03:44.171932  7250 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:44.143481  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.172067  7643 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:44.173416  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.174099  1589 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:44.174731  7627 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:44.175100  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:44.175640  1589 raft_consensus.cc:2376] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18. Re-sending same reply.
I20260709 13:03:44.177028  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.179821  1589 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:44.162164  1585 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:44.182389  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:44.149566  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 15 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:44.184689  1585 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:44.184775  1664 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:44.182323  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.189728  2312 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:44.164393  1667 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:44.138285  7580 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:44.183955  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 225349us
I20260709 13:03:44.231436  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:44.136260  7480 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.178387  7489 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:44.242110  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:44.258471  7527 negotiation.cc:336] Failed RPC negotiation. Trace:
0709 13:03:40.748610 (+     0us) reactor.cc:730] Submitting negotiation task for client connection to 127.0.107.193:41267 (local address 127.0.0.1:45540)
0709 13:03:41.142793 (+394183us) negotiation.cc:107] Waiting for socket to connect
0709 13:03:41.142903 (+   110us) client_negotiation.cc:175] Beginning negotiation
0709 13:03:41.426530 (+283627us) client_negotiation.cc:262] Sending NEGOTIATE NegotiatePB request
0709 13:03:44.158838 (+2732308us) negotiation.cc:326] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.0.107.193:41267: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":387360,"thread_start_us":239829,"threads_started":1}
I20260709 13:03:44.259512  7606 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.139257  7611 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.233296  7248 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.270236  7415 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:44.271523  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3059) took 1262 ms. Trace:
I20260709 13:03:44.258570  7381 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.273181  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2239) took 1010 ms. Trace:
I20260709 13:03:44.273243  7540 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:44.275816  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.096256  7640 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.238365  7430 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.278865  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.147938  7629 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.282218  7607 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.282518  7239 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:44.285182  7589 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.093s	user 0.000s	sys 0.002s
W20260709 13:03:44.285873  7573 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.789s	user 0.002s	sys 0.001s
I20260709 13:03:44.287289  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:44.150307  7642 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:03:44.193061  1737 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:44.292366  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:44.149096  7649 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 13:03:44.326930  7638 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 13:03:44.327526  7644 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:44.327767  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.328521  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 987) took 1136 ms. Trace:
I20260709 13:03:44.329730  7636 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:44.328896  1668 rpcz_store.cc:276] 0709 13:03:43.191669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.191905 (+   236us) service_pool.cc:224] Handling call
0709 13:03:43.214878 (+ 22973us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:43.404739 (+189861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:43.404800 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:43.518129 (+113329us) log.cc:844] Serialized 26 byte log entry
0709 13:03:43.867342 (+349213us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:43.867407 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.867508 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:44.328252 (+460744us) raft_consensus.cc:1685] finished
0709 13:03:44.328277 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:44.328443 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47069,"raft.queue_time_us":153383,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":76457,"thread_start_us":328694,"threads_started":2,"wal-append.queue_time_us":364311}
I20260709 13:03:44.331089  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:44.332340  7600 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.922s	user 0.001s	sys 0.002s
I20260709 13:03:44.333036  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.333714  1659 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:44.333968  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1073) took 1478 ms. Trace:
I20260709 13:03:44.334841  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.336261  1659 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:44.336745  7581 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.313295  7628 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.337827  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.338193  7501 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.338443  1659 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:44.339454  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:44.338966  7501 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.340021  1659 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:44.329172  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.342289  7640 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:44.343116  7457 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:44.343909  7457 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:44.346309  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.346740  7611 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.347246  7457 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:44.347575  7649 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:44.348233  7649 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:44.348628  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.334334  2310 rpcz_store.cc:276] 0709 13:03:42.855362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.858499 (+  3137us) service_pool.cc:224] Handling call
0709 13:03:42.882353 (+ 23854us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.888581 (+  6228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.888628 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:42.903572 (+ 14944us) log.cc:844] Serialized 26 byte log entry
0709 13:03:43.024285 (+120713us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b1800487ae0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:43.220534 (+196249us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:43.220623 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.220726 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:44.286232 (+1065506us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:44.333604 (+ 47372us) raft_consensus.cc:1685] finished
0709 13:03:44.333626 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:44.333763 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":26759,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":10774,"spinlock_wait_cycles":40507648,"thread_start_us":194312,"threads_started":1,"wal-append.queue_time_us":393391}
I20260709 13:03:44.349862  7457 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:03:44.350381  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.351354  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.352147  7457 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 13:03:44.353348  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.361647  7629 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.362696  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2947) took 1895 ms. Trace:
I20260709 13:03:44.363174  1665 rpcz_store.cc:276] 0709 13:03:42.467754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.467981 (+   227us) service_pool.cc:224] Handling call
0709 13:03:42.481756 (+ 13775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.653050 (+171294us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.653100 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:42.662704 (+  9604us) log.cc:844] Serialized 26 byte log entry
0709 13:03:42.925473 (+262769us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:42.928263 (+  2790us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:42.928538 (+   275us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:44.284353 (+1355815us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:44.362354 (+ 78001us) raft_consensus.cc:1685] finished
0709 13:03:44.362379 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:44.362621 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49710,"raft.queue_time_us":140956,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":35957120,"thread_start_us":261804,"threads_started":1,"wal-append.queue_time_us":435868}
I20260709 13:03:44.365053  7629 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:44.173834  7461 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.676s	user 0.013s	sys 0.002s
I20260709 13:03:44.234674  7625 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:44.309895  1586 rpcz_store.cc:276] 0709 13:03:43.009185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.142341 (+133156us) service_pool.cc:224] Handling call
0709 13:03:43.330239 (+187898us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:43.347237 (+ 16998us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.261153 (+913916us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400393fd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:44.271297 (+ 10144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339578240}
I20260709 13:03:44.149015  7650 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:44.311574  1663 rpcz_store.cc:276] 0709 13:03:43.268762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.285160 (+ 16398us) service_pool.cc:224] Handling call
0709 13:03:44.263314 (+978154us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:44.273085 (+  9771us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147884288}
I20260709 13:03:44.315433  1660 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:44.302068  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.319798  1591 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:44.319061  7381 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.379184  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:44.379966  1586 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.369303  7457 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:44.370606  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3069) took 1138 ms. Trace:
W20260709 13:03:44.371059  7595 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.002s	user 0.012s	sys 0.001s
I20260709 13:03:44.371399  7636 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 13:03:44.353215  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.825s (SENT)
I20260709 13:03:44.375780  1593 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
W20260709 13:03:44.321899  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.378556  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.392761  1593 pending_rounds.cc:85] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:44.393101  1593 pending_rounds.cc:107] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:03:44.393339  1593 raft_consensus.cc:2889] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:44.378878  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.378527  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.365381  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.390988  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.394392  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:44.395605  7630 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:44.396152  7630 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.396605  7630 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.393356  1659 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:44.390780  1587 rpcz_store.cc:276] 0709 13:03:43.232091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.335113 (+103022us) service_pool.cc:224] Handling call
0709 13:03:44.370349 (+1035236us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54001def54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:44.370537 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170752256}
I20260709 13:03:44.401360  1665 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:03:44.403894  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.410195  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.411304  1585 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20260709 13:03:44.408396  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:44.418984  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.417475  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.423441  1587 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:44.376812  7625 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:44.425400  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.427189  1589 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:44.390553  7457 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.436326  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:44.437033  1734 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:44.403097  7595 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.442485  1739 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 13:03:44.444736  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.445914  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.446156  7625 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.447371  7625 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.448130  1589 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 12. Preceding OpId from leader: term: 12 index: 13. (index mismatch)
I20260709 13:03:44.448943  7505 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:44.449555  7505 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:44.450191  7599 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.496s	user 0.003s	sys 0.000s
I20260709 13:03:44.450610  7505 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.452076  7595 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.454267  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:44.455282  2316 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:44.454265  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:44.436970  7494 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:44.457038  7494 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.457757  7574 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.458422  7574 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.460304  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3022) took 1295 ms. Trace:
I20260709 13:03:44.461845  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:44.462216  7574 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.462532  2316 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:44.463644  7381 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.464219  1742 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
I20260709 13:03:44.464531  1742 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 13:03:44.464752  1742 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:44.466059  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:44.466600  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:44.467573  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.468400  1739 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:03:44.468762  7507 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.351s	user 0.012s	sys 0.003s
I20260709 13:03:44.471109  7416 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:44.471547  7416 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:44.460704  1584 rpcz_store.cc:276] 0709 13:03:43.165235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.718564 (+553329us) service_pool.cc:224] Handling call
0709 13:03:43.738938 (+ 20374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:43.739316 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:43.739348 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:43.740016 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 13:03:43.949576 (+209560us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:43.949830 (+   254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.949937 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:44.460073 (+510136us) raft_consensus.cc:1685] finished
0709 13:03:44.460096 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:44.460239 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118070,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":6522,"thread_start_us":122966,"threads_started":1,"wal-append.queue_time_us":536556}
I20260709 13:03:44.310801  7480 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.430099  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.429316  1587 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:44.422356  7629 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:44.422571  7035 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:03:44.418553  7578 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.563s	user 0.012s	sys 0.001s
I20260709 13:03:44.483203  2318 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:44.432685  7634 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.409s	user 0.001s	sys 0.007s
I20260709 13:03:44.446661  1738 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
W20260709 13:03:44.426597  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73419.3us
I20260709 13:03:44.430188  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:44.484241  1708 leader_election.cc:336] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.874s (SENT)
I20260709 13:03:44.484793  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:44.481686  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.431144  7596 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:44.389747  7461 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.502388  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:44.503064  1584 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:44.476197  7457 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:44.476012  7507 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:03:44.485153  7485 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.748s	user 0.012s	sys 0.001s
I20260709 13:03:44.516718  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:44.529769  2312 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:44.476974  7381 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.476781  7636 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:44.485917  7532 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.001s	sys 0.015s
I20260709 13:03:44.530313  1665 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:03:44.531008  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:44.532233  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48048.5us
W20260709 13:03:44.538309  1708 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.098s (SENT)
I20260709 13:03:44.534484  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2246) took 1071 ms. Trace:
I20260709 13:03:44.538841  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:44.538997  1667 rpcz_store.cc:276] 0709 13:03:43.463045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.463548 (+   503us) service_pool.cc:224] Handling call
0709 13:03:44.534287 (+1070739us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b540030af54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:44.534416 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220490112}
I20260709 13:03:44.491225  7596 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.484797  7409 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.472533  7416 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.501381  7035 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.508596  7644 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:44.548575  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:44.501057  7629 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.501571  7578 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.557790  7578 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.562047  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.548040  7035 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:44.508390  7626 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:44.543558  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:44.571251  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.540023  7596 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:44.551477  7409 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.536729  7636 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.528101  7248 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.540962  7381 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.554055  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:44.555367  7611 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:44.543187  7601 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.572078  1740 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
W20260709 13:03:44.576789  7441 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.066s	user 0.010s	sys 0.006s
I20260709 13:03:44.549438  1736 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:44.570044  1591 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
W20260709 13:03:44.598834  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44874.4us
I20260709 13:03:44.524933  7640 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:44.608256  1586 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
W20260709 13:03:44.608925  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 235758us
I20260709 13:03:44.610546  1734 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:44.612356  7480 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.612813  7381 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.613131  7640 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:44.593250  7248 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.602021  7607 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:44.615272  7619 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:44.615540  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.663159  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.618871  1587 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:44.577348  7628 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:44.532570  7485 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.612422  7461 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.646970  1584 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 6.
W20260709 13:03:44.657553  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:44.620810  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.018s	sys 0.009s
I20260709 13:03:44.601826  7507 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:44.621812  7653 raft_consensus.cc:2764] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:03:44.625943  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:44.686501  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 210563us
I20260709 13:03:44.700948  1586 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:44.686942  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:44.697522  1585 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:44.685034  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.674456  7381 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:44.696380  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.018s	sys 0.009s
I20260709 13:03:44.710767  1592 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:44.710953  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:44.714217  1584 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:44.712993  1665 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:44.618336  7441 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.593464  7626 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.731022  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.733580  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.694231  7614 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:44.734292  1736 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:03:44.732788  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:44.660629  7595 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:03:44.617937  7601 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.678870  7606 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 13:03:44.701113  7416 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.749995  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:44.703414  7480 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.757983  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.776075  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.766294  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:44.780027  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.780743  1736 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:44.698947  7644 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 13:03:44.771850  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:44.696903  7607 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.767526  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.777246  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:44.792176  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 176732us
I20260709 13:03:44.794099  1734 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:44.711654  7507 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:44.751703  7614 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.742127  7663 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.826902  1586 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:03:44.831571  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81848us
I20260709 13:03:44.753991  7631 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:44.756743  7578 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:44.756520  7619 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:44.729588  7381 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.847930  2311 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
W20260709 13:03:44.848086  1478 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 318816us
I20260709 13:03:44.854017  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:44.861169  7648 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.796231  7629 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:44.864995  1584 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:03:44.847919  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:44.833210  7614 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:44.869114  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 113001us
I20260709 13:03:44.874781  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.798300  7480 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:44.919787  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52146us
I20260709 13:03:44.849412  7660 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:44.859666  7578 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:03:44.880066  7478 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.819s	user 0.006s	sys 0.006s
I20260709 13:03:44.935931  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:44.861475  7681 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:44.942940  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 256077us
I20260709 13:03:44.945088  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:44.841789  7663 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.962798  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:44.963078  7662 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:44.963608  2317 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:44.963809  7581 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.964439  7581 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.965523  7637 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:03:44.966408  7694 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:44.967129  7694 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.932067  7660 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:44.970455  7660 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
W20260709 13:03:44.963934  7667 negotiation.cc:336] Failed RPC negotiation. Trace:
0709 13:03:44.285928 (+     0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.0.107.193:41267)
0709 13:03:44.568334 (+282406us) server_negotiation.cc:207] Beginning negotiation
0709 13:03:44.568378 (+    44us) server_negotiation.cc:400] Waiting for connection header
0709 13:03:44.568465 (+    87us) server_negotiation.cc:408] Connection header received
0709 13:03:44.568833 (+   368us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0709 13:03:44.568867 (+    34us) server_negotiation.cc:462] Received NEGOTIATE request from client
0709 13:03:44.569264 (+   397us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0709 13:03:44.570066 (+   802us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45540: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":215948,"thread_start_us":94237,"threads_started":1}
I20260709 13:03:44.972483  7381 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.973062  7381 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:44.973657  7507 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:44.980561  7625 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.533s	user 0.005s	sys 0.005s
I20260709 13:03:44.981360  7480 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:44.981856  7666 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 13:03:44.954903  7630 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.002s
I20260709 13:03:44.989603  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
W20260709 13:03:44.991438  2316 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.528s	user 0.013s	sys 0.000s
I20260709 13:03:44.956544  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:44.992235  2316 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:44.881253  7648 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:44.932049  7636 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:44.993911  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2963) took 1804 ms. Trace:
I20260709 13:03:45.000135  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.001379  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.003587  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:44.885267  7494 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:44.999863  1742 rpcz_store.cc:276] 0709 13:03:43.189468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.189712 (+   244us) service_pool.cc:224] Handling call
0709 13:03:44.453714 (+1264002us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:03:44.463987 (+ 10273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.465315 (+  1328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.465353 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.465891 (+   538us) log.cc:844] Serialized 26 byte log entry
0709 13:03:44.728533 (+262642us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:44.728573 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.728653 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:44.983244 (+254591us) raft_consensus.cc:1685] finished
0709 13:03:44.983280 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:44.993829 (+ 10549us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8309120,"thread_start_us":261067,"threads_started":1,"wal-append.queue_time_us":411036}
I20260709 13:03:44.941461  7606 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:44.965549  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:44.961174  7618 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:45.054001  1734 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:45.057583  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:45.061971  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:45.064431  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108262us
W20260709 13:03:45.068117  1668 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.524s	user 0.012s	sys 0.001s
I20260709 13:03:44.946379  7478 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.074860  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.086544  1584 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:03:45.076619  7649 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.727s	user 0.012s	sys 0.003s
W20260709 13:03:45.059927  7468 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.210s	user 0.012s	sys 0.003s
I20260709 13:03:45.087911  1668 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:44.987128  7409 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.021597  7485 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.117744  7626 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.061339  7248 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.003898  7381 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.118898  7640 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:45.121083  7381 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.003017  7630 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.004827  7629 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 13:03:45.019757  7489 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 13:03:45.093039  7619 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:45.107977  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 233442us
I20260709 13:03:45.084667  7625 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:03:45.112694  2310 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:45.115172  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.124043  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:45.125319  7593 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.257s	user 0.048s	sys 0.033s
W20260709 13:03:45.177706  7593 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.452s	user 0.049s	sys 0.033s
I20260709 13:03:45.094295  7644 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:45.178870  7644 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.179816  7648 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.088027  7606 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.181324  7637 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.106135  7649 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.181949  7637 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.183849  7606 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.184367  7637 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:45.184557  7649 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.126550  7601 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.185346  1665 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
W20260709 13:03:45.190119  7659 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.916s	user 0.007s	sys 0.000s
I20260709 13:03:45.108100  7468 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.159096  7636 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:45.194420  7468 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.194933  7636 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:45.132118  7494 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.092542  7578 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.195926  7636 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.143535  7703 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:45.197899  7703 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:45.136861  7671 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.192s	user 0.002s	sys 0.001s
I20260709 13:03:45.106236  1584 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:45.201395  1584 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:45.201730  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2954) took 1365 ms. Trace:
I20260709 13:03:45.139423  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:45.140728  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 201066us
I20260709 13:03:45.105257  7441 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.107807  7461 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:45.169914  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 168616us
I20260709 13:03:45.170251  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.007782  7662 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.134s
I20260709 13:03:45.218297  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:45.174942  7650 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.756s	user 0.011s	sys 0.002s
I20260709 13:03:45.175182  7478 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.176151  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.175894  7619 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.198406  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.235066  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:45.110716  7681 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:45.199942  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.238739  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.200611  7607 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.204190  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.240264  1737 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:45.202158  1666 rpcz_store.cc:276] 0709 13:03:43.836704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.836849 (+   145us) service_pool.cc:224] Handling call
0709 13:03:45.201529 (+1364680us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b540013e7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.201660 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105230080}
I20260709 13:03:45.206099  2317 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:45.215193  1736 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15.
I20260709 13:03:45.215193  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.242542  7673 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:45.243460  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:45.244611  7673 raft_consensus.cc:2764] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:03:45.185317  7625 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:45.245781  7625 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:45.217800  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2953) took 1382 ms. Trace:
I20260709 13:03:45.175062  7639 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Stepping down as leader of term 9
I20260709 13:03:45.231312  1734 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:45.248909  7639 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.248782  1661 rpcz_store.cc:276] 0709 13:03:43.835789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.835946 (+   157us) service_pool.cc:224] Handling call
0709 13:03:43.864215 (+ 28269us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:43.909331 (+ 45116us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:43.909374 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:43.996103 (+ 86729us) log.cc:844] Serialized 26 byte log entry
0709 13:03:44.096742 (+100639us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:44.096792 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.096882 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.189664 (+1092782us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b24003d5ab8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2f09d 00007fcdedb6110e 00007fcdedb7888c 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221
0709 13:03:45.207438 (+ 17774us) raft_consensus.cc:1685] finished
0709 13:03:45.207461 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.217715 (+ 10254us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15345,"raft.queue_time_us":34022,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":3419,"spinlock_wait_cycles":42347776,"thread_start_us":71928,"threads_started":1,"wal-append.queue_time_us":262001}
I20260709 13:03:45.250317  7639 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.233435  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.241068  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.256989  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:45.188915  7660 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.244742  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:45.236604  7494 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:45.261776  7494 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.264580  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.264855  7581 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.265430  1742 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:45.267351  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2248) took 1092 ms. Trace:
I20260709 13:03:45.274441  1668 rpcz_store.cc:276] 0709 13:03:44.174512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.330773 (+156261us) service_pool.cc:224] Handling call
0709 13:03:45.267139 (+936366us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54002535d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.267281 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":488587008}
I20260709 13:03:45.275722  1662 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [13.7-14.8]
W20260709 13:03:45.270961  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2244) took 1808 ms (client timeout 2147 ms). Trace:
I20260709 13:03:45.272409  1659 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
W20260709 13:03:45.278591  1664 rpcz_store.cc:269] 0709 13:03:43.462293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.464483 (+  2190us) service_pool.cc:224] Handling call
0709 13:03:45.270724 (+1806241us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.270888 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":780550400}
I20260709 13:03:45.252735  7650 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 13:03:45.284705  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:45.288645  7457 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.752s	user 0.012s	sys 0.002s
I20260709 13:03:45.320266  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:45.331736  7704 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:45.331449  7457 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.332599  7700 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:45.333407  7700 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:45.333244  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.334187  7672 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:45.335086  7592 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.071s	user 0.000s	sys 0.004s
I20260709 13:03:45.334820  7672 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.337222  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 941) took 2592 ms. Trace:
I20260709 13:03:45.259516  7644 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.236817  7461 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.352708  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.354140  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 951) took 1076 ms. Trace:
W20260709 13:03:45.354990  7035 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.789s	user 0.008s	sys 0.005s
I20260709 13:03:45.355378  7461 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:45.355953  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3089) took 1338 ms. Trace:
I20260709 13:03:45.355443  7035 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.356356  1592 rpcz_store.cc:276] 0709 13:03:44.017345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.140942 (+123597us) service_pool.cc:224] Handling call
0709 13:03:45.355704 (+1214762us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b5400144954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.355859 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1206222592}
I20260709 13:03:45.266579  7694 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.200985  7703 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:45.359614  7620 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.633s	user 0.003s	sys 0.000s
I20260709 13:03:45.341940  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.360340  7701 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.360999  7701 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.361783  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.362156  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2952) took 1526 ms. Trace:
W20260709 13:03:45.362323  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 117681us
I20260709 13:03:45.362677  1740 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:45.342571   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:45.363942  7644 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:45.364588  7644 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:45.366371  7698 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 13:03:45.367189  7644 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.367514  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1079) took 1136 ms. Trace:
I20260709 13:03:45.367954  2312 rpcz_store.cc:276] 0709 13:03:44.230977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.231114 (+   137us) service_pool.cc:224] Handling call
0709 13:03:45.367305 (+1136191us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400724054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.367426 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676620544}
W20260709 13:03:45.343246  7416 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.568s	user 0.013s	sys 0.000s
I20260709 13:03:45.343844  7441 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:45.330335  7629 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:45.369731  7629 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.370828  7629 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.373643  1663 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:45.374269  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.374894  1737 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:45.375538  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2958) took 1504 ms. Trace:
I20260709 13:03:45.375926  1663 rpcz_store.cc:276] 0709 13:03:43.871275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.396742 (+525467us) service_pool.cc:224] Handling call
0709 13:03:44.397007 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.374126 (+977119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.375285 (+  1159us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b5400012a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:45.375478 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2245560704}
I20260709 13:03:45.335391  7681 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.368052  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.269132  7657 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:45.347012  7618 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:45.265472  7494 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:45.354557  1591 rpcz_store.cc:276] 0709 13:03:44.277677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.378273 (+100596us) service_pool.cc:224] Handling call
0709 13:03:45.353950 (+975677us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.354077 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406672768}
I20260709 13:03:45.346975  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.362591  1662 rpcz_store.cc:276] 0709 13:03:43.835438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:43.835587 (+   149us) service_pool.cc:224] Handling call
0709 13:03:45.271841 (+1436254us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:45.276343 (+  4502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.361926 (+ 85583us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540013f1d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.362084 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3248839552}
I20260709 13:03:45.377619  1663 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:03:45.378572  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:45.344535  1742 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:45.353487  1586 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:45.394163  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2247) took 1222 ms. Trace:
I20260709 13:03:45.378377  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:45.352115  1588 rpcz_store.cc:276] 0709 13:03:42.745216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:42.745417 (+   201us) service_pool.cc:224] Handling call
0709 13:03:42.745892 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:42.746373 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:42.746407 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:42.746952 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 13:03:43.016361 (+269409us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:43.016423 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:43.016498 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.334130 (+2317632us) spinlock_profiling.cc:243] Waited 6.78 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:45.336686 (+  2556us) raft_consensus.cc:1685] finished
0709 13:03:45.336724 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.337094 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11452,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":106,"spinlock_wait_cycles":15595264,"thread_start_us":262533,"threads_started":1,"wal-append.queue_time_us":383789}
I20260709 13:03:45.380270  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3082) took 1545 ms. Trace:
I20260709 13:03:45.394590  1663 rpcz_store.cc:276] 0709 13:03:44.172218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.376887 (+1204669us) service_pool.cc:224] Handling call
0709 13:03:45.377331 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.393838 (+ 16507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.394098 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.397038  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.395233  1593 rpcz_store.cc:276] 0709 13:03:43.834354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.002515 (+168161us) service_pool.cc:224] Handling call
0709 13:03:44.375064 (+372549us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.403611 (+ 28547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.376562 (+972951us) spinlock_profiling.cc:243] Waited 372 ms on lock 0x7b54001def50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.380170 (+  3608us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":855823616}
I20260709 13:03:45.404564  1591 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:45.406270  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.271239  7614 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.405946  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:45.415326  1662 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:45.411466  1742 raft_consensus.cc:2368] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:03:45.413183  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.413408  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.413801  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.417284  1591 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:45.419601  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.420312  1591 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:45.420315  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 997) took 1111 ms. Trace:
I20260709 13:03:45.381806  7657 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.424022  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2249) took 1235 ms. Trace:
I20260709 13:03:45.424410  1662 rpcz_store.cc:276] 0709 13:03:44.188362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.393607 (+1205245us) service_pool.cc:224] Handling call
0709 13:03:45.415114 (+ 21507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.423755 (+  8641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.423962 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.423126  1663 rpcz_store.cc:276] 0709 13:03:44.308974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.403048 (+1094074us) service_pool.cc:224] Handling call
0709 13:03:45.419751 (+ 16703us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.419967 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.419994 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.420041 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:45.420073 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.420129 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.420251 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.383798  7618 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.414357  7639 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 9 FOLLOWER]: Advancing to term 10
I20260709 13:03:45.428948  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.429556  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.430860  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.409799  7407 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:45.437193  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 998) took 1117 ms. Trace:
I20260709 13:03:45.439183  1737 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:45.438954  1662 rpcz_store.cc:276] 0709 13:03:44.319493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.425202 (+1105709us) service_pool.cc:224] Handling call
0709 13:03:45.436681 (+ 11479us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.436870 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.436894 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.436937 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:45.436968 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.437018 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.437134 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.440701  1586 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:45.414491  7494 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:45.410039  7663 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:45.442412  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.443686  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.444419  1588 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:45.445462  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.446115  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:45.447379  7609 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:45.447817  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:45.448141  7609 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:45.449854  7609 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:03:45.450091  7657 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.450629  7662 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:45.450930  7494 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:45.433794  7596 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.841s	user 0.013s	sys 0.002s
I20260709 13:03:45.446942  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.452221  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.443337  1586 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:03:45.463303  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.452998  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.423645  7611 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:45.474895  1588 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 13:03:45.501940  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54194.1us
I20260709 13:03:45.507643  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.514398  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.459110  7494 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:45.459564  7407 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:45.526211  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3024) took 1058 ms. Trace:
I20260709 13:03:45.534754  1591 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:45.526726  1589 rpcz_store.cc:276] 0709 13:03:44.467756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.485794 (+ 18038us) service_pool.cc:224] Handling call
0709 13:03:44.524692 (+ 38898us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.719531 (+194839us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.719583 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.741573 (+ 21990us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.005908 (+264335us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:45.005998 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.006188 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.525991 (+519803us) raft_consensus.cc:1685] finished
0709 13:03:45.526014 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.526147 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8185,"raft.queue_time_us":67909,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":29604,"thread_start_us":312792,"threads_started":2,"wal-append.queue_time_us":681150}
I20260709 13:03:45.475788  7611 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:45.566169  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:45.578599  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.526499  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:45.574362  1741 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:45.591086  1588 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:45.582634  1557 leader_election.cc:336] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.634s (SENT)
W20260709 13:03:45.594132  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68023.4us
W20260709 13:03:45.633736  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.596324  1589 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 6. (term mismatch)
I20260709 13:03:45.647475  1589 pending_rounds.cc:85] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 5
I20260709 13:03:45.647914  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3036) took 1001 ms. Trace:
I20260709 13:03:45.655754  1588 rpcz_store.cc:276] 0709 13:03:44.646553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.525939 (+879386us) service_pool.cc:224] Handling call
0709 13:03:45.581576 (+ 55637us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.647601 (+ 66025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.647843 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:45.648569  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.647984  1589 pending_rounds.cc:107] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 13:03:45.659528  1589 raft_consensus.cc:2889] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:45.649974  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.661084  1733 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:45.666478  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.580506  7611 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:45.596247  7716 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 13:03:45.483664  7596 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.673452  1733 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:03:45.680270  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.681178  1588 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:45.619316  7657 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:45.685738  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.689752  1741 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 13:03:45.691551  1733 raft_consensus.cc:1217] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Deduplicated request from leader. Original: 13.7->[14.8-16.9]   Dedup: 16.9->[]
I20260709 13:03:45.700037  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.701437  1733 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 13:03:45.695055  1741 pending_rounds.cc:85] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 7
I20260709 13:03:45.704646  1741 pending_rounds.cc:107] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:03:45.704941  1741 raft_consensus.cc:2889] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:45.672371  7704 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 13:03:45.688344  1708 connection.cc:256] RPC call timeout handler was delayed by -1.1309856060000243s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2721s for a response
I20260709 13:03:45.707474  1733 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:45.700582  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1001) took 1215 ms. Trace:
I20260709 13:03:45.708869  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.710034  1589 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:45.708408  1662 rpcz_store.cc:276] 0709 13:03:44.485770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.455138 (+969368us) service_pool.cc:224] Handling call
0709 13:03:45.505735 (+ 50597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.505996 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:45.509737 (+  3741us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.700207 (+190470us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.700266 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:45.700307 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.700378 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.700512 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13,"prepare.queue_time_us":4636,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":36230,"raft.queue_time_us":5602,"thread_start_us":190048,"threads_started":1}
I20260709 13:03:45.679279  7660 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:45.729034  7712 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:03:45.707845  7407 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:45.730508  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 155232us
W20260709 13:03:45.732065  1557 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.685s (SENT)
I20260709 13:03:45.739665  1736 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:45.744998  1741 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:45.746344  7035 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:03:45.747807  1741 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:45.748695  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.816062  1591 raft_consensus.cc:2368] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
I20260709 13:03:45.817278  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:45.749055  7718 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 13:03:45.750061  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.756799  7461 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:45.757876  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.759670  7583 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.762254  7614 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.764142  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1003) took 1184 ms. Trace:
I20260709 13:03:45.766777  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 555) took 1071 ms. Trace:
I20260709 13:03:45.700976  7693 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:45.773633  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:45.774076  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 2955) took 1936 ms (client timeout 1579 ms). Trace:
I20260709 13:03:45.774598  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3028) took 1305 ms. Trace:
I20260709 13:03:45.774960  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.775323  7416 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:45.775462  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2960) took 1903 ms. Trace:
I20260709 13:03:45.783058  7661 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:45.785177  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2961) took 1894 ms. Trace:
I20260709 13:03:45.781384  7494 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.788651  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 994) took 1832 ms. Trace:
I20260709 13:03:45.847663  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:45.789706  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.791081  1584 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.792284  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:45.842664  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 84912.2us
I20260709 13:03:45.792965  7596 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:45.807588  1736 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 13:03:45.818096  1591 raft_consensus.cc:2376] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12. Re-sending same reply.
I20260709 13:03:45.848047  1668 rpcz_store.cc:276] 0709 13:03:43.956198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.275444 (+1319246us) service_pool.cc:224] Handling call
0709 13:03:45.275850 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.276263 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.276300 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.319770 (+ 43470us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.514809 (+195039us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:45.514857 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.514934 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.788394 (+273460us) raft_consensus.cc:1685] finished
0709 13:03:45.788416 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.788579 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70165,"raft.queue_time_us":543,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":10929,"thread_start_us":141153,"threads_started":1,"wal-append.queue_time_us":335447}
W20260709 13:03:45.848585  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.853225  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.820374  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.847733  1667 rpcz_store.cc:276] 0709 13:03:43.890883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.563478 (+672595us) service_pool.cc:224] Handling call
0709 13:03:44.563846 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.783306 (+219460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.783355 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:44.783417 (+    62us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:44.783464 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.784871 (+1001407us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:45.784926 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.785111 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":504092416}
I20260709 13:03:45.850726  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:45.724682  7595 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:45.817906  1737 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:45.853947  1591 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:03:45.849592  1587 rpcz_store.cc:276] 0709 13:03:44.477183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.481246 (+  4063us) service_pool.cc:224] Handling call
0709 13:03:45.774432 (+1293186us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54001def54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.774536 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":303337728}
I20260709 13:03:45.856555  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:45.847218  1660 rpcz_store.cc:269] 0709 13:03:43.837427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.275474 (+438047us) service_pool.cc:224] Handling call
0709 13:03:45.773862 (+1498388us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b540013d8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.774000 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5991808}
I20260709 13:03:45.580884  7618 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.826316  2283 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:45.812438  7704 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:45.843760  1659 rpcz_store.cc:276] 0709 13:03:43.872035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.398926 (+526891us) service_pool.cc:224] Handling call
0709 13:03:45.272222 (+873296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.279995 (+  7773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.775270 (+495275us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b54002535d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.775405 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2007540096}
I20260709 13:03:45.844766  7416 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.824774  7461 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.847491  7661 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:45.877739  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:45.859208  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.878577  1591 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:45.871070  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:45.879969  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.855509  7648 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:45.882140  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.853945  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:45.849522  2309 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:45.862167  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.863056  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:45.827418  1662 rpcz_store.cc:276] 0709 13:03:44.579487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.710668 (+1131181us) service_pool.cc:224] Handling call
0709 13:03:45.763569 (+ 52901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.763767 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.763795 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.763853 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:45.763888 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.763948 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:45.764073 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:45.864976  1584 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:45.907579  1592 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:03:45.825351  7631 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:45.871286  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:45.910041  2317 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:45.843184  2314 rpcz_store.cc:276] 0709 13:03:44.695006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.839688 (+144682us) service_pool.cc:224] Handling call
0709 13:03:45.766529 (+926841us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54006004d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:45.766709 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312988928}
I20260709 13:03:45.906909  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.848194  7707 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:45.909364  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.910041  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.917110  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:45.920630  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.721s (SENT)
I20260709 13:03:45.921283  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:45.923010  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69160.1us
I20260709 13:03:45.924472  1587 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:45.927042  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:45.923732  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:45.929148  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.878423  7491 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:45.944105  7491 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:45.945139  7491 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.945884  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.946770  1741 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 16 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 13:03:45.932421  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:45.952054  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:45.952648  1585 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
W20260709 13:03:45.932844  7719 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.228s	user 0.000s	sys 0.000s
I20260709 13:03:45.933133  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:45.877578  7694 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:45.934024  7707 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.934125  7461 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.935885  7661 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.869866  7650 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:03:45.991235  7461 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:45.991861  7650 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:45.992269  7650 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:45.992563  7707 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:45.993199  7650 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.993083  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:45.994565  7650 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:45.996671  7693 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:45.942178  7704 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.926011  7601 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:45.997851  7601 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:45.998397  7601 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:45.999792  7717 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 13:03:45.943140  7618 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:46.000769  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.932554  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.002228  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.980487  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.002452  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.003024  1741 raft_consensus.cc:2368] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:46.003723  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.004007  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.004886  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.981179  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:46.006333  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.006563  7457 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.007259  7457 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:46.007563  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.008302  7457 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:46.008787  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.011036  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.011417  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1080) took 1644 ms. Trace:
I20260709 13:03:46.011768  2310 rpcz_store.cc:276] 0709 13:03:44.366694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.366872 (+   178us) service_pool.cc:224] Handling call
0709 13:03:44.367312 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.175430 (+808118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.011178 (+835748us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:46.011357 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1713586560}
I20260709 13:03:46.012204  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.012713  1740 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:46.013659  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:45.959753  1584 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 13:03:46.014493  1738 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:46.014608  1584 pending_rounds.cc:85] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 6
I20260709 13:03:46.015501  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.018688  7582 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:46.014986  1584 pending_rounds.cc:107] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 13:03:46.020181  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1015) took 1578 ms. Trace:
I20260709 13:03:46.020607  1735 rpcz_store.cc:276] 0709 13:03:44.441838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.444947 (+  3109us) service_pool.cc:224] Handling call
0709 13:03:44.445230 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.557158 (+111928us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:44.939398 (+382240us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.255232 (+315834us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:45.255304 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:45.255355 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.019929 (+764574us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54001081d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:46.019985 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.020121 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15414,"prepare.queue_time_us":109533,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3659,"raft.queue_time_us":915,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":2524,"spinlock_wait_cycles":256825216,"thread_start_us":661596,"threads_started":2,"wal-append.queue_time_us":626542}
I20260709 13:03:46.020617  1584 raft_consensus.cc:2889] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:45.866873  7595 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.979450  7489 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:45.986188  7673 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:45.986975  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.987193  7657 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 13:03:45.987015  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:45.988387  7686 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.819s	user 0.007s	sys 0.000s
I20260709 13:03:45.988463  1708 leader_election.cc:400] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
W20260709 13:03:45.990207  7639 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.562s	user 0.012s	sys 0.000s
I20260709 13:03:45.990538  7609 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:45.994145  7653 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.025128  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3027) took 1556 ms. Trace:
I20260709 13:03:46.040365  1584 rpcz_store.cc:276] 0709 13:03:44.468944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.096800 (+627856us) service_pool.cc:224] Handling call
0709 13:03:45.097165 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.788065 (+690900us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b4c0027b400. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:03:46.024847 (+236782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.025066 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3157888}
I20260709 13:03:46.040570  7653 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:45.931731  7660 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:46.042176  7660 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:46.038911  1667 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:03:46.042809  7616 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:46.043090  1708 leader_election.cc:403] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 13:03:46.043255  7660 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.043358  7616 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:45.908847  7606 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:46.033989  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.051720  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:46.056139  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67849.3us
I20260709 13:03:46.064883  1736 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:46.042924  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:45.974867  2317 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:46.045172  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3023) took 1577 ms. Trace:
I20260709 13:03:46.023929  1735 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:46.066049  1590 rpcz_store.cc:276] 0709 13:03:44.467448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.478385 (+ 10937us) service_pool.cc:224] Handling call
0709 13:03:44.508875 (+ 30490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:44.509300 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:44.509338 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:44.528963 (+ 19625us) log.cc:844] Serialized 26 byte log entry
0709 13:03:44.804288 (+275325us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:44.804347 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:44.804458 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:45.987702 (+1183244us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:46.044771 (+ 57069us) raft_consensus.cc:1685] finished
0709 13:03:46.044795 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.045066 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":67879,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":10122,"spinlock_wait_cycles":38494976,"thread_start_us":233189,"threads_started":1,"wal-append.queue_time_us":571416}
I20260709 13:03:46.010061  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.003929  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:45.970314  1588 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:46.080507  1588 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:46.081606  7707 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:03:46.082103  7707 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:46.088083  7501 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.088207  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.096287  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.008869  7457 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.143602  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.144575  1667 raft_consensus.cc:2376] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15. Re-sending same reply.
W20260709 13:03:46.068775  7625 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.822s	user 0.015s	sys 0.000s
I20260709 13:03:46.133638  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:45.925297  7681 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.146469  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.147332  1592 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:46.019315  7582 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.154410  1591 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:46.181572  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:46.179457  1740 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:46.023571  7601 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.185424  7639 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:03:46.185963  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.186695  7416 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:46.187949  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 121586us
I20260709 13:03:46.026903  7672 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:46.191820  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.046150  7494 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:46.044834  7627 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
W20260709 13:03:46.201650  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.131747  7609 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:46.197319  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:45.946995  7631 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.089762  7619 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
I20260709 13:03:46.046846  7694 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.203365  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.215303  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.216187  1667 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:03:46.212992  7649 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.006s	sys 0.011s
W20260709 13:03:46.221752  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.173115  7611 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.238906  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:46.224634  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.227545  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.223835  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.231741  1668 raft_consensus.cc:2410] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 13:03:46.244027  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.210443  7672 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:46.192744  7681 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:46.251332  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48028.9us
W20260709 13:03:46.259486  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.007s	sys 0.012s
W20260709 13:03:46.260038  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.007s	sys 0.012s
I20260709 13:03:46.264554  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.265604  1662 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:46.248093  7649 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.006s	sys 0.011s
I20260709 13:03:46.188958  7625 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.249161  7430 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:46.248477  7239 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 13:03:46.249426  7673 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:46.275928  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.261226  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 2981) took 1817 ms. Trace:
I20260709 13:03:46.249804  7627 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:46.293469  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.301332  1667 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:46.250097  1666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.007s	sys 0.011s
W20260709 13:03:46.302266  1666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.008s	sys 0.012s
I20260709 13:03:46.303551  7430 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:46.304821  7430 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.255647  7595 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:46.292006  7629 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.043s	sys 0.008s
I20260709 13:03:46.299790  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.313257  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2962) took 2262 ms. Trace:
I20260709 13:03:46.258671  7672 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:46.291563  7662 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:46.328433  7595 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:46.331398  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:46.341527  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.309294  7507 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:46.360085  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.360891  1740 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:46.326386  1664 rpcz_store.cc:276] 0709 13:03:44.050663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.286874 (+1236211us) service_pool.cc:224] Handling call
0709 13:03:45.287358 (+   484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.323566 (+ 36208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.323620 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.324318 (+   698us) log.cc:844] Serialized 26 byte log entry
0709 13:03:45.694538 (+370220us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:45.694608 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:45.694723 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.301281 (+606558us) raft_consensus.cc:1685] finished
0709 13:03:46.301308 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.313163 (+ 11855us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35642,"raft.queue_time_us":79864,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":289419,"threads_started":1,"wal-append.queue_time_us":828125}
I20260709 13:03:46.362222  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.363126  1740 raft_consensus.cc:2368] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15.
I20260709 13:03:46.363013  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.363941  1664 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:46.364270  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.365018  1740 raft_consensus.cc:2376] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16. Re-sending same reply.
I20260709 13:03:46.294528  7639 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:46.366111  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.360643  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.375231  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:03:46.326866  7629 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.044s	sys 0.008s
I20260709 13:03:46.309719  7494 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:46.376672  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.385676  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.416491  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.417523  1733 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:46.327739  7609 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.357831  7507 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:46.379062  7673 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:46.378378  7639 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:46.437541  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.446985  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.447914  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:46.450685  1664 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:46.424784  7416 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:46.455024  7416 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:46.455286  7494 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:46.455525  7416 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:46.455883  7494 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.443187  1667 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:46.445253  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.467080  1733 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:46.468145  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.498178  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:46.511608  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:46.513717  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.516319  1667 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:46.459707  7457 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.520689  1667 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 9. Current term is 10. Ops: []
I20260709 13:03:46.465015  7582 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.451427  7639 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.507925  7601 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.510520  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:46.551355  1733 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:46.552852  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:46.525651  7636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.014s	sys 0.046s
W20260709 13:03:46.589735  7636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.014s	sys 0.047s
I20260709 13:03:46.590452  7616 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.590758  7636 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.566295  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2307) took 1389 ms. Trace:
I20260709 13:03:46.572314  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:46.591881  1738 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:46.591631  1737 rpcz_store.cc:276] 0709 13:03:45.177339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.451922 (+274583us) service_pool.cc:224] Handling call
0709 13:03:46.566047 (+1114125us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:46.566197 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":734044032}
I20260709 13:03:46.528321  7673 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:46.554453  7694 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.552552  7681 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:46.582350  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.023s	sys 0.069s
W20260709 13:03:46.600368  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.024s	sys 0.070s
I20260709 13:03:46.526170  7507 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:46.593334  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1004) took 1932 ms (client timeout 2291 ms). Trace:
W20260709 13:03:46.600994  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 775976us
W20260709 13:03:46.605708  1665 rpcz_store.cc:269] 0709 13:03:44.660775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.747205 (+ 86430us) service_pool.cc:224] Handling call
0709 13:03:46.593146 (+1845941us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:46.593267 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936407424}
W20260709 13:03:46.582311  7597 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.018s	sys 0.043s
W20260709 13:03:46.601600  1659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.024s	sys 0.044s
W20260709 13:03:46.604393  1632 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.011s	sys 0.016s
W20260709 13:03:46.608107  1659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.024s	sys 0.044s
I20260709 13:03:46.516492  7672 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.607237  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:46.608120  1632 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.011s	sys 0.016s
W20260709 13:03:46.611590  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 684408us
I20260709 13:03:46.585258  7625 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.614108  2283 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:46.604123  7693 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.013s	sys 0.014s
W20260709 13:03:46.611052  7597 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.018s	sys 0.044s
I20260709 13:03:46.607205  7681 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.591456  7636 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:46.639019  2316 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.012s	sys 0.032s
W20260709 13:03:46.643291  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.644178  1665 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: [15.8-15.8]
W20260709 13:03:46.622467  7693 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.013s	sys 0.015s
I20260709 13:03:46.647843  7693 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:46.650627  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.008s	sys 0.023s
I20260709 13:03:46.618992  7656 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:46.639880  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:46.640411  7681 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:46.651042  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.008s	sys 0.020s
W20260709 13:03:46.700680  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.008s	sys 0.021s
W20260709 13:03:46.701081  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 651075us
W20260709 13:03:46.650374  2281 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.010s	sys 0.018s
I20260709 13:03:46.637049  7578 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:46.701732  2281 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.010s	sys 0.018s
W20260709 13:03:46.702086  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 653355us
I20260709 13:03:46.653082  2315 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:46.627491  7507 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:03:46.626786  1587 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.013s	sys 0.021s
W20260709 13:03:46.658018  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 476321us
W20260709 13:03:46.642323  2316 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.013s	sys 0.032s
W20260709 13:03:46.661597  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.008s	sys 0.023s
W20260709 13:03:46.708331  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.662907  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:46.605800  7707 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:46.713418  1587 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.014s	sys 0.022s
I20260709 13:03:46.709148  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.720453  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.722432  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.703519  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:46.734063  1662 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:46.737360  1592 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:46.661847  7656 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:46.713171  7618 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.713s	user 0.013s	sys 0.000s
W20260709 13:03:46.705852  7704 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.032s	sys 0.030s
I20260709 13:03:46.742690  1557 leader_election.cc:400] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15.
I20260709 13:03:46.754616  1557 leader_election.cc:403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:03:46.774596  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:46.778257  1740 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:46.740758  7745 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:46.745798  7656 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:46.599028  7673 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:46.806862  7727 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.182s	user 0.000s	sys 0.000s
I20260709 13:03:46.807917  7673 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:46.757467  7704 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.032s	sys 0.030s
W20260709 13:03:46.764046  7731 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.210s	user 0.001s	sys 0.001s
I20260709 13:03:46.792295  1592 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:46.754050  7468 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.810509  7681 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.810395  7468 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.811131  7681 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:46.811857  7681 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:03:46.743502  7457 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 13:03:46.783089  7738 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 13:03:46.649339  7693 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:46.813488  7728 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
I20260709 13:03:46.814961  7738 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:46.815363  7738 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:46.781533  7618 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.816227  7738 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:46.816639  7618 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.812875  7681 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.818181  7681 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.818845  7618 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:46.819856  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 992) took 2911 ms. Trace:
I20260709 13:03:46.820253  1666 rpcz_store.cc:276] 0709 13:03:43.908329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.256087 (+1347758us) service_pool.cc:224] Handling call
0709 13:03:45.283360 (+ 27273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.323442 (+ 40082us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.323490 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.394420 (+ 70930us) log.cc:844] Serialized 26 byte log entry
0709 13:03:46.302726 (+908306us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:46.302802 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.302932 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:46.819599 (+516667us) raft_consensus.cc:1685] finished
0709 13:03:46.819627 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.819785 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53422,"raft.queue_time_us":47397,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":8774,"thread_start_us":904250,"threads_started":1,"wal-append.queue_time_us":1203138}
I20260709 13:03:46.794435  7749 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:46.822666  1666 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 13:03:46.823680  7606 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.656s	user 0.014s	sys 0.001s
I20260709 13:03:46.824136  1666 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:46.824625  7606 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:46.825706  1664 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:46.813484  7457 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15.
I20260709 13:03:46.744074  7748 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 13:03:46.830130  7457 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:46.829943  7738 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:46.830639  7457 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:46.831638  7457 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:46.833046  7457 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:46.835502  7733 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 13:03:46.839932  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.842561  7701 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.843101  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3034) took 2226 ms. Trace:
I20260709 13:03:46.843910  7611 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.843608  1586 rpcz_store.cc:276] 0709 13:03:44.616956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.500282 (+883326us) service_pool.cc:224] Handling call
0709 13:03:45.508661 (+  8379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:46.841768 (+1333107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:46.841805 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.841856 (+    51us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:46.841896 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.842766 (+   870us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:46.842827 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.842996 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3064360704}
I20260709 13:03:46.849571  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:46.851078  1586 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:46.852578  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:46.801699  7639 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:46.859145  1664 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:46.784260  7582 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:46.860185  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 316025us
I20260709 13:03:46.856164  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:46.943315  1586 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
W20260709 13:03:46.871073  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:46.854203  7614 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:46.921264  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3048) took 1538 ms. Trace:
I20260709 13:03:46.855053  7681 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:46.887594  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:46.945798  1584 rpcz_store.cc:276] 0709 13:03:45.382514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.041699 (+659185us) service_pool.cc:224] Handling call
0709 13:03:46.042050 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:46.042325 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:46.561929 (+519604us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.920695 (+358766us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.920794 (+    99us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:46.920862 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.920967 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.921177 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103051,"prepare.queue_time_us":743723,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":62150,"thread_start_us":691529,"threads_started":2}
W20260709 13:03:46.982095  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 115361us
W20260709 13:03:46.983366  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 279806us
I20260709 13:03:46.989425  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:46.299098  1739 rpcz_store.cc:276] 0709 13:03:44.444259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:44.471228 (+ 26969us) service_pool.cc:224] Handling call
0709 13:03:44.471723 (+   495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:44.471942 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:44.701890 (+229948us) log.cc:844] Serialized 14 byte log entry
0709 13:03:45.743186 (+1041296us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b18004c1860. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a 00007fcdedfe7a2a
0709 13:03:46.260648 (+517462us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.260730 (+    82us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:46.260796 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.260982 (+   186us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:46.261152 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64179,"prepare.queue_time_us":189564,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":2172,"raft.queue_time_us":2100,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":7292,"spinlock_wait_cycles":46963840,"thread_start_us":644210,"threads_started":2,"wal-append.queue_time_us":1603785}
I20260709 13:03:47.012466  7707 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:03:46.943177  7491 raft_consensus.cc:2764] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:03:47.006251  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.022099  1586 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:47.024571  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3039) took 1650 ms. Trace:
I20260709 13:03:46.983121  7636 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.033866  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:47.034732  1586 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:47.030829  1587 rpcz_store.cc:276] 0709 13:03:45.374250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.929328 (+555078us) service_pool.cc:224] Handling call
0709 13:03:45.980256 (+ 50928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.980513 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:03:46.714038 (+733525us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.024035 (+309997us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.024229 (+   194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:47.024274 (+    45us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:47.024319 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.024394 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.024516 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2802,"prepare.queue_time_us":801269,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":89,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":1375,"thread_start_us":941568,"threads_started":2}
I20260709 13:03:47.040093  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:46.983354  7614 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:47.050369  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 307646us
W20260709 13:03:47.050755  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 338682us
I20260709 13:03:47.053611  7578 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:47.054500  7597 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:47.054731  7578 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.055615  1587 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:47.058239  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 887) took 1070 ms. Trace:
I20260709 13:03:47.056077  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:47.069567  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:47.075525  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 300903us
I20260709 13:03:47.062088  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:47.041982  7491 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:47.077427  1632 leader_election.cc:400] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:47.077899  1632 leader_election.cc:403] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 13:03:47.079823  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.061933  2316 rpcz_store.cc:276] 0709 13:03:45.987651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.020960 (+ 33309us) service_pool.cc:224] Handling call
0709 13:03:46.021299 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:46.021536 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:46.707694 (+686158us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.056970 (+349276us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:47.057062 (+    92us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:47.057131 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.057973 (+   842us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.058160 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":740934,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"thread_start_us":1028245,"threads_started":2}
I20260709 13:03:47.083335  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:47.057897  7461 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.066s	user 0.015s	sys 0.001s
W20260709 13:03:47.095938  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.103669  1586 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:47.103672  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:47.057298  7596 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.060539  7614 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:47.078799  7416 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.623s	user 0.014s	sys 0.000s
I20260709 13:03:47.108808  1733 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:47.074893  7636 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:47.094992  7491 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.114400  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:47.092519  7704 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:47.107308  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.120796  1586 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:47.130033  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:47.109936  7611 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:47.132972  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:47.112397  7582 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:47.111771  7747 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:47.116443  7491 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.134886  7430 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:47.138233  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:47.134958  7636 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:47.159937  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.160305  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.160946  1738 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:03:47.160010  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.118381  7596 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.138756  7681 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:47.140504  7701 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:47.119822  7416 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.182430  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.182725  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.183205  1736 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
W20260709 13:03:47.185254  7751 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 13:03:47.205892  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.210162  7416 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.173877  7701 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.211540  7701 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:03:47.232964  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 232493us
I20260709 13:03:47.151895  7597 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:47.239065  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.171243  7700 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:47.237996  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:47.242830  7609 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:47.252054  7609 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.252441  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:47.242734  7639 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.253935  7639 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.255157  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:47.255649  7768 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:47.256110  7639 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.257081  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:47.256127  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 961) took 1484 ms. Trace:
I20260709 13:03:47.258901  7609 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.231815  7491 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:03:47.259702  7609 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:47.260183  7681 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:47.260370  7609 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.260982  7681 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:47.261373  7700 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:47.262530  7700 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.258718  1591 rpcz_store.cc:276] 0709 13:03:45.772028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.189911 (+417883us) service_pool.cc:224] Handling call
0709 13:03:46.190224 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:46.656907 (+466683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:46.656958 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:46.699468 (+ 42510us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.021690 (+322222us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:47.021814 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.021913 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.255870 (+233957us) raft_consensus.cc:1685] finished
0709 13:03:47.255895 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.256043 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":166835,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4138,"thread_start_us":779674,"threads_started":2,"wal-append.queue_time_us":451391}
I20260709 13:03:47.266237  7693 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 13:03:47.266312  7662 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.266888  7693 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.267148  7662 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:47.267949  7693 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:47.268469  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2974) took 1274 ms. Trace:
I20260709 13:03:47.268888  1665 rpcz_store.cc:276] 0709 13:03:45.993855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.645366 (+651511us) service_pool.cc:224] Handling call
0709 13:03:46.645891 (+   525us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:46.646183 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:46.646220 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:46.646953 (+   733us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.128275 (+481322us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:47.128331 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.129388 (+  1057us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.268206 (+138818us) raft_consensus.cc:1685] finished
0709 13:03:47.268231 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.268403 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":480855,"threads_started":1,"wal-append.queue_time_us":618921}
I20260709 13:03:47.271800  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1000) took 2862 ms. Trace:
I20260709 13:03:47.272241  1663 rpcz_store.cc:276] 0709 13:03:44.409543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.428868 (+1019325us) service_pool.cc:224] Handling call
0709 13:03:45.440901 (+ 12033us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.441281 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.441309 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.454881 (+ 13572us) log.cc:844] Serialized 26 byte log entry
0709 13:03:46.920471 (+1465590us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b1800452aa0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:47.149706 (+229235us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:47.149766 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.149865 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.271444 (+121579us) raft_consensus.cc:1685] finished
0709 13:03:47.271467 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.271734 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25659,"raft.queue_time_us":10891,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":18792,"spinlock_wait_cycles":6711040,"thread_start_us":228955,"threads_started":1,"wal-append.queue_time_us":1807793}
I20260709 13:03:47.274824  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3119) took 1561 ms. Trace:
I20260709 13:03:47.275224  1590 rpcz_store.cc:276] 0709 13:03:45.713593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.144370 (+430777us) service_pool.cc:224] Handling call
0709 13:03:46.185986 (+ 41616us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:46.658753 (+472767us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:46.658802 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:46.707185 (+ 48383us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.128463 (+421278us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:47.149547 (+ 21084us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.152058 (+  2511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.152142 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.274544 (+122402us) raft_consensus.cc:1685] finished
0709 13:03:47.274577 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.274764 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66338,"prepare.queue_time_us":21723,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":83306,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":22601,"thread_start_us":841902,"threads_started":2,"wal-append.queue_time_us":501571}
I20260709 13:03:47.245353  7597 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:47.277752  7703 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:47.277596  7597 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:47.278081  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 172116us
I20260709 13:03:47.280865  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.282295  1590 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 13:03:47.285833  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.285154  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.283514  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.276379  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1007) took 1263 ms. Trace:
I20260709 13:03:47.283283  7681 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.286118  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3126) took 1352 ms. Trace:
I20260709 13:03:47.287304  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.244791  7457 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.290939  1590 rpcz_store.cc:276] 0709 13:03:45.933309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.276954 (+1343645us) service_pool.cc:224] Handling call
0709 13:03:47.277328 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:47.285685 (+  8357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.285912 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:47.287575  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 185991us
W20260709 13:03:47.282788  1556 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.013s	sys 0.023s
I20260709 13:03:47.293797  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:47.302038  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.013s	sys 0.023s
I20260709 13:03:47.250303  7636 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.288484  1662 rpcz_store.cc:276] 0709 13:03:46.012917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.739982 (+727065us) service_pool.cc:224] Handling call
0709 13:03:46.754454 (+ 14472us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:46.766096 (+ 11642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:46.766149 (+    53us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:46.791114 (+ 24965us) log.cc:844] Serialized 78 byte log entry
0709 13:03:47.080087 (+288973us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:47.106124 (+ 26037us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.275702 (+169578us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b18003d2820. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:03:47.275924 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.276018 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.276058 (+    40us) raft_consensus.cc:1685] finished
0709 13:03:47.276157 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.276318 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15750,"prepare.queue_time_us":30594,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":55,"raft.queue_time_us":182817,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":13270,"spinlock_wait_cycles":4278144,"thread_start_us":224718,"threads_started":1,"wal-append.queue_time_us":396560}
I20260709 13:03:47.284932  7582 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.295358  7656 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.316363  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.326390  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.325423  1587 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:47.328123  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.328866  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:47.333549  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.335271  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:47.336357  1587 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:47.337383  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.338332  1587 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:47.339249  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.339975  1587 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:03:47.341014  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.294373  7681 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:47.346575  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.322773  7619 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:47.332167  7672 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:47.402022  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 155755us
I20260709 13:03:47.332238  7745 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:47.342396  7650 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:47.374197  7747 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:47.416209  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:47.373615  7611 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:47.277594  7662 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.325706  7582 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.347587  7749 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:47.410202  7507 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.691s	user 0.015s	sys 0.000s
I20260709 13:03:47.322481  7457 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.400748  7766 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:47.331776  7636 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.413115  7672 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.470784  1733 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:47.427989  7611 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.421522  7773 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.449328  7609 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.452346  7639 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:47.481802  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 71795us
I20260709 13:03:47.486302  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.494403  1587 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:47.448881  7777 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 13:03:47.448966  7749 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.487900  7609 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.455941  7766 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.491225  7597 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.523108  7609 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.527469  7763 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:47.497463  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:47.530298  2316 raft_consensus.cc:606] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:47.532415  7776 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 13:03:47.535382  7766 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.536983  7457 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.537786  7656 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:47.540212  7636 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.540848  7597 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.543135  7597 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.546523  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.547324  2316 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:03:47.550256  7789 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 13:03:47.552851  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3125) took 1619 ms. Trace:
I20260709 13:03:47.553390  1591 rpcz_store.cc:276] 0709 13:03:45.933013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.264549 (+1331536us) service_pool.cc:224] Handling call
0709 13:03:47.265064 (+   515us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:47.284166 (+ 19102us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:47.284203 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:47.299119 (+ 14916us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.474701 (+175582us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:47.488224 (+ 13523us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.496356 (+  8132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.496441 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.552541 (+ 56100us) raft_consensus.cc:1685] finished
0709 13:03:47.552574 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.552764 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5850,"prepare.queue_time_us":43841,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":59,"raft.queue_time_us":17805,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":10484,"thread_start_us":99146,"threads_started":1,"wal-append.queue_time_us":147611}
I20260709 13:03:47.563287  7611 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.492784  7773 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.484290  7764 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 13:03:47.570472  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3053) took 1620 ms. Trace:
I20260709 13:03:47.567298  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:47.485555  7639 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.574399  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.571022  1591 rpcz_store.cc:276] 0709 13:03:45.949628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.556059 (+1606431us) service_pool.cc:224] Handling call
0709 13:03:47.556562 (+   503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.569962 (+ 13400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:47.570012 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:47.570080 (+    68us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:47.570127 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.570211 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.570391 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.605289  1591 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:47.606279  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3054) took 1562 ms. Trace:
W20260709 13:03:47.579943  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 86606.2us
I20260709 13:03:47.606717  1591 rpcz_store.cc:276] 0709 13:03:46.043430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.604464 (+1561034us) service_pool.cc:224] Handling call
0709 13:03:47.604945 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.605892 (+   947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.606204 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.519606  7631 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.614156  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:47.572598  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81912.3us
I20260709 13:03:47.570885  7636 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.616984  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3124) took 1684 ms. Trace:
I20260709 13:03:47.621613  1593 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:03:47.604885  7700 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:47.619788  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 290527us
W20260709 13:03:47.629396  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.621094  1586 rpcz_store.cc:276] 0709 13:03:45.932603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.133365 (+1200762us) service_pool.cc:224] Handling call
0709 13:03:47.147576 (+ 14211us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:47.148099 (+   523us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:47.148138 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:47.185227 (+ 37089us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.481741 (+296514us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:47.616370 (+134629us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.616676 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.616750 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:47.616793 (+    43us) raft_consensus.cc:1685] finished
0709 13:03:47.616804 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.616926 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16218,"prepare.queue_time_us":44450,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":2337,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":8257,"thread_start_us":393777,"threads_started":2,"wal-append.queue_time_us":344711}
W20260709 13:03:47.634613  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 312901us
I20260709 13:03:47.630497  7700 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.646103  7707 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.646891  7707 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:47.649058  1556 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.147s (SENT)
I20260709 13:03:47.649582  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:47.659242  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:46298 (request call id 0) took 1036 ms. Trace:
I20260709 13:03:47.659787  1588 rpcz_store.cc:276] 0709 13:03:46.627964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.629343 (+1001379us) service_pool.cc:224] Handling call
0709 13:03:47.659135 (+ 29792us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.662595  7609 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:47.663419  7430 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:47.663934  7430 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:47.670188  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:47.676193  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.678802  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 966) took 1053 ms. Trace:
I20260709 13:03:47.683521  1588 rpcz_store.cc:276] 0709 13:03:46.625385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.660668 (+1035283us) service_pool.cc:224] Handling call
0709 13:03:47.661052 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.661289 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:47.661324 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:47.661372 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:47.661408 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.661470 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:47.678718 (+ 17248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:47.626435  7748 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.625012  7703 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.630052  7662 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:47.688349  7766 raft_consensus.cc:2764] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:03:47.630596  7704 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13.
I20260709 13:03:47.689630  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:47.687520  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:47.698719  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.699640  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "219129ac643e4650b611a85f565f5990"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:47.713848  2316 raft_consensus.cc:606] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:47.625160  7636 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.714293  2316 raft_consensus.cc:614] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:47.700139  2317 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:47.684724  7663 consensus_queue.cc:1175] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ff44a4643e01482cb6b6a79b099fa766" responder_term: 16 status { last_received { term: 15 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:03:47.703904  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.716392  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:47.685825  7778 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 13:03:47.704474  1478 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 316f3bf18ed94e32b238130a6bfedf1a is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 11 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:47.687135  7582 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.719722  2315 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:47.727210  7693 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.728268  7681 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.730067  7639 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:47.730592  7581 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:47.730911  7609 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:47.718480  1735 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:47.717247  7803 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:47.721350  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72282.6us
I20260709 13:03:47.754302  1584 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:47.735132  7662 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:03:47.702315  7748 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.707067  7703 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.735970  7581 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:47.740281  7700 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.737486  7609 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:47.794279  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:47.795246  1733 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:47.770900  7457 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:47.801005  1737 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:47.777298  7711 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.769423  7662 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:47.816458  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:47.783479  7581 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:03:47.823006  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.788196  7609 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:47.823988  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.777248  7693 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.826557  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:47.833716  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 115865us
I20260709 13:03:47.844442  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:47.853288  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:47.789141  7582 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.864189  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:47.855921  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:47.789373  7773 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.790320  7700 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:47.755216  7803 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.786975  7606 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:47.865687  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:47.866557  1735 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:47.823846  7662 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:47.895702  7803 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:47.827816  7792 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:47.864689  7611 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:47.826866  7653 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.814666  7614 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 13:03:47.877667  7582 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.907303  7631 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:47.918526  7792 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.826968  7711 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.901376  7606 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:47.978238  7806 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.006285  7819 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:47.980506  7807 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:48.007227  7807 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:48.007637  7807 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:47.982287  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.986145  7703 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:47.989288  7701 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.777s	user 0.010s	sys 0.005s
I20260709 13:03:48.012571  7701 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.013429  7636 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.015254  7701 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.991654  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:47.996695  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.019068  2312 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:47.976289  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.021767  2310 raft_consensus.cc:2410] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 13 because replica has last-logged OpId of term: 12 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 12.
I20260709 13:03:47.997040  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:48.032603  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:47.984666  1740 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 13:03:47.974225  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 104105us
W20260709 13:03:48.038771  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 173115us
I20260709 13:03:48.041018  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 967) took 1403 ms. Trace:
I20260709 13:03:48.041036  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:48.045534  1733 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:47.992807  7631 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:48.036192  7636 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:48.064141  7636 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:48.064677  7636 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:48.063858  7631 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.066486  7824 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.067088  7824 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.068240  7703 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:48.069011  7703 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.070618  7649 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:48.071200  7649 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:48.072352  1663 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:03:48.043124  1588 rpcz_store.cc:276] 0709 13:03:46.637707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.713276 (+1075569us) service_pool.cc:224] Handling call
0709 13:03:47.734518 (+ 21242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.039318 (+304800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.039353 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.039399 (+    46us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:48.039436 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.039657 (+   221us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.040949 (+  1292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38024,"raft.queue_time_us":134886,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":20946,"thread_start_us":304298,"threads_started":1}
I20260709 13:03:48.073196  1663 raft_consensus.cc:606] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:48.073885  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.075577  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.039752  7441 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:48.076571  1662 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:48.033706  7701 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.079032  7701 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:47.982148  7704 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:48.080768  7704 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:48.081216  7606 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:48.030445  7630 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:47.958153  7430 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.073885  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:48.076576  7441 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.083390  1588 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.084512  7430 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.081979  7704 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:47.982761  7802 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:03:48.075407  7774 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:48.098141  7774 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:48.102952  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.104341  1660 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 10.
I20260709 13:03:48.106696  1665 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:48.072117  7649 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.097628  1662 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:03:48.249087  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:47.978423  7681 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:47.981331  7792 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.344836  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.353153  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:48.354789  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.355625  1588 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:48.362915  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.367561  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.389541  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.390801  1662 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:03:48.352494  7768 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.022s	sys 0.025s
W20260709 13:03:48.383683  1586 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.048s	sys 0.114s
I20260709 13:03:48.211428  7704 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.251860  7810 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:48.398881  1586 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.049s	sys 0.115s
I20260709 13:03:48.402714  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.410187  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 965) took 1787 ms. Trace:
I20260709 13:03:48.338421  7773 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.410805  1586 rpcz_store.cc:276] 0709 13:03:46.622917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.631660 (+1008743us) service_pool.cc:224] Handling call
0709 13:03:47.667163 (+ 35503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.667411 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:03:47.667717 (+   306us) log.cc:844] Serialized 14 byte log entry
0709 13:03:48.408839 (+741122us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.409024 (+   185us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:48.409157 (+   133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.409707 (+   550us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.410094 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38890,"prepare.queue_time_us":1324,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":106795,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":14704,"thread_start_us":740329,"threads_started":2,"wal-append.queue_time_us":275269}
I20260709 13:03:48.421836  7773 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:48.434798  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 343756us
W20260709 13:03:48.439508  1559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.024s	sys 0.026s
I20260709 13:03:48.383605  7818 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:48.441215  1559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.025s	sys 0.027s
I20260709 13:03:48.444788  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.394191  7681 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:48.403900  7768 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.022s	sys 0.026s
I20260709 13:03:48.439363  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.455964  1588 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:48.449280  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.452648  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.406510  7796 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.459789  1586 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:48.461472  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:48.460999  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:48.463774  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 395446us
I20260709 13:03:48.464594  1588 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:48.099385  7774 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.463042  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.445176  7703 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.490206  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.496290  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.503098  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:48.501735  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.436363  7810 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.471808  7407 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:03:48.516673  7820 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:48.517302  7820 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:48.518674  7820 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.520615  7820 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.522063  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.525941  1586 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:03:48.525970  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.509828  1665 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:48.527549  1588 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:48.529013  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.529176  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.529963  1663 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:48.530467  1588 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:48.532742  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.457329  7681 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.538137  1662 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:03:48.540151  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:48.542317  7821 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:03:48.542692  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:48.543406  1586 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:03:48.543820  7806 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.540s	user 0.007s	sys 0.003s
I20260709 13:03:48.544277  7806 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:48.461148  7818 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.545143  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 968) took 1902 ms. Trace:
I20260709 13:03:48.546842  7818 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.532593  7824 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.550904  1586 rpcz_store.cc:276] 0709 13:03:46.643038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.544234 (+1901196us) service_pool.cc:224] Handling call
0709 13:03:48.544608 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.544799 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.544826 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.544873 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:48.544911 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.544966 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.545085 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.552978  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.533699  1588 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:48.505870  7774 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:48.556679  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:48.532200  7807 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.524s	user 0.007s	sys 0.005s
I20260709 13:03:48.561240  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.464851  7796 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:48.542407  7609 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.000s
I20260709 13:03:48.564744  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:48.558984  7822 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.513s	user 0.000s	sys 0.002s
I20260709 13:03:48.549968  7701 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.579174  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.581044  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.584566  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.586790  1586 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 13:03:48.588071  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.037s	sys 0.064s
W20260709 13:03:48.588675  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.037s	sys 0.064s
I20260709 13:03:48.588869  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 969) took 1934 ms. Trace:
W20260709 13:03:48.589193  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.589335  1586 rpcz_store.cc:276] 0709 13:03:46.654232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.567110 (+1912878us) service_pool.cc:224] Handling call
0709 13:03:48.586305 (+ 19195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.588556 (+  2251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.588807 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.574983  7630 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:03:48.582916  7581 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.719s	user 0.013s	sys 0.000s
W20260709 13:03:48.585745  7611 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.661s	user 0.008s	sys 0.004s
W20260709 13:03:48.609424  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 257410us
I20260709 13:03:48.611276  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.564787  7807 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.576495  7609 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.622902  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3055) took 2297 ms. Trace:
I20260709 13:03:48.623409  1591 rpcz_store.cc:276] 0709 13:03:46.325987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.612573 (+1286586us) service_pool.cc:224] Handling call
0709 13:03:47.613050 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:47.614153 (+  1103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:47.614181 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:47.614708 (+   527us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.962115 (+347407us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:47.962183 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:47.962290 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:48.622673 (+660383us) raft_consensus.cc:1685] finished
0709 13:03:48.622697 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.622837 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7541,"raft.queue_time_us":2644,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":2233,"thread_start_us":277313,"threads_started":1,"wal-append.queue_time_us":418348}
I20260709 13:03:48.603724  7630 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:48.625813  1591 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 13:03:48.627399  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2253) took 2754 ms (client timeout 1884 ms). Trace:
W20260709 13:03:48.628082  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.627812  1668 rpcz_store.cc:269] 0709 13:03:45.873213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.873390 (+   177us) service_pool.cc:224] Handling call
0709 13:03:48.627162 (+2753772us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:48.627300 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":732503808}
I20260709 13:03:48.629457  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.630982  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:48.631644  2312 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:48.631736  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.632076  2312 raft_consensus.cc:614] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:48.626906  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3062) took 1768 ms. Trace:
I20260709 13:03:48.632651  1591 rpcz_store.cc:276] 0709 13:03:46.859053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.625089 (+1766036us) service_pool.cc:224] Handling call
0709 13:03:48.625440 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.626631 (+  1191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.626851 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.616313  7611 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:48.639360  7630 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.643728  7630 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:48.643379  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:48.644625  2312 raft_consensus.cc:606] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:48.613854  7581 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.647979  7581 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.650684  7796 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.650913  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:48.651955  2312 raft_consensus.cc:606] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:48.652412  2312 raft_consensus.cc:614] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:48.653160  1665 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:48.654407  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.655601  1668 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:48.660573   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:03:48.669762  1663 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:48.655632  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.672789  1665 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:03:48.673954  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.675320  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.671316  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.676761  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.678102  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.678278  7810 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.679327  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:48.679641  7681 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:48.680266  1663 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:48.681128  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.682262  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:48.683156  7748 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:48.683689  7662 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.776s	user 0.011s	sys 0.000s
I20260709 13:03:48.684141  7662 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:48.684523  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 964) took 2063 ms (client timeout 2017 ms). Trace:
W20260709 13:03:48.684921  1592 rpcz_store.cc:269] 0709 13:03:46.621057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.652596 (+ 31539us) service_pool.cc:224] Handling call
0709 13:03:48.684340 (+2031744us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:48.684453 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62928128}
I20260709 13:03:48.686523  7807 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:48.687310  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.684939  1665 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:48.690174  7672 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:48.702797  7636 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.008s
W20260709 13:03:48.694387  7798 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.830s	user 0.003s	sys 0.009s
I20260709 13:03:48.694550  7596 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:48.697160  1592 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:03:48.728667  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3130) took 1411 ms. Trace:
I20260709 13:03:48.729198  1592 rpcz_store.cc:276] 0709 13:03:47.317327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.696382 (+1379055us) service_pool.cc:224] Handling call
0709 13:03:48.696832 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.728217 (+ 31385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.728571 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.693814  1663 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:48.732733  1663 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:03:48.703442  7609 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:48.738907  7735 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.795s	user 0.010s	sys 0.006s
I20260709 13:03:48.745097  7596 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.753396  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3051) took 2804 ms. Trace:
I20260709 13:03:48.729373  7636 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.760324  1590 rpcz_store.cc:276] 0709 13:03:45.949012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.449976 (+1500964us) service_pool.cc:224] Handling call
0709 13:03:47.458836 (+  8860us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.475929 (+ 17093us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:03:47.620985 (+145056us) log.cc:844] Serialized 14 byte log entry
0709 13:03:47.765164 (+144179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:47.765250 (+    86us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:47.765303 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.752982 (+987679us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:48.753070 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.753302 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50408,"prepare.queue_time_us":50069,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":121636,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":26993,"spinlock_wait_cycles":43209344,"thread_start_us":266421,"threads_started":3,"wal-append.queue_time_us":114730}
I20260709 13:03:48.777030  7833 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.778307  7833 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.781596  7833 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:48.756604  7609 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.782434  7833 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:48.760761  7582 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:48.760298  7831 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:48.785713  7672 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:48.761722  7596 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:48.811556  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:48.812173  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.812981  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3059) took 1961 ms. Trace:
I20260709 13:03:48.814509  1586 rpcz_store.cc:276] 0709 13:03:46.851615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.597208 (+1745593us) service_pool.cc:224] Handling call
0709 13:03:48.618577 (+ 21369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.618909 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:48.794509 (+175600us) log.cc:844] Serialized 14 byte log entry
0709 13:03:48.812482 (+ 17973us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:48.812563 (+    81us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:48.812625 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.812728 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.812911 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":118,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":184959,"threads_started":2,"wal-append.queue_time_us":19037}
I20260709 13:03:48.815531  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:48.816159  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.816764  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.817121  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.816494  7841 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:48.818084  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.819008  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:48.820374  2316 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 13:03:48.820933  1586 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:48.822775  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 970) took 1579 ms. Trace:
I20260709 13:03:48.823302  1591 rpcz_store.cc:276] 0709 13:03:47.243469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.633642 (+1390173us) service_pool.cc:224] Handling call
0709 13:03:48.634094 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.634350 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:48.634368 (+    18us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:48.659505 (+ 25137us) log.cc:844] Serialized 26 byte log entry
0709 13:03:48.811207 (+151702us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:48.811248 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.811320 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:48.822461 (+ 11141us) raft_consensus.cc:1685] finished
0709 13:03:48.822493 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:48.822682 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":72824,"threads_started":1,"wal-append.queue_time_us":95318}
I20260709 13:03:48.824594  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 974) took 1053 ms. Trace:
I20260709 13:03:48.825485  1586 rpcz_store.cc:276] 0709 13:03:47.771355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.820528 (+1049173us) service_pool.cc:224] Handling call
0709 13:03:48.820803 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.824349 (+  3546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.824538 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:48.825842  1591 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:48.827107  1586 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:03:48.827659  1585 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:48.826825  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 976) took 1044 ms. Trace:
I20260709 13:03:48.795763  7609 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.830478  1586 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:03:48.828629  1591 rpcz_store.cc:276] 0709 13:03:47.782245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.825023 (+1042778us) service_pool.cc:224] Handling call
0709 13:03:48.825394 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.826496 (+  1102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.826728 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:03:48.865880  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:48.818401  7841 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:48.987919  1632 leader_election.cc:336] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.648s (SENT)
W20260709 13:03:49.051309  1593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3040) took 3454 ms (client timeout 1958 ms). Trace:
I20260709 13:03:49.051718  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:49.052963  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65096.4us
I20260709 13:03:48.811324  7662 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.054138  7662 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.055132  7737 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.054836  7662 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.055802  7737 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.051344  7738 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.057850  7407 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:49.058264  7738 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:49.051744  1593 rpcz_store.cc:269] 0709 13:03:45.374524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.412916 (+ 38392us) service_pool.cc:224] Handling call
0709 13:03:48.827115 (+3414199us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54004001d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:48.829322 (+  2207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5068322176}
I20260709 13:03:49.061108  7738 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.062361  7737 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.062052  7662 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:48.797224  7833 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:48.798269  7831 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.808516  7672 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:48.833308  7660 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.050758  7748 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:49.096727  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1010) took 3023 ms (client timeout 2215 ms). Trace:
W20260709 13:03:49.097455  1664 rpcz_store.cc:269] 0709 13:03:46.073638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.497764 (+424126us) service_pool.cc:224] Handling call
0709 13:03:49.096388 (+2598624us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400260f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:49.096606 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":750906624}
I20260709 13:03:49.099176  1660 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
W20260709 13:03:49.108075  1706 connection.cc:256] RPC call timeout handler was delayed by -1.3359822859999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 13:03:49.110168  1706 connection.cc:256] RPC call timeout handler was delayed by -1.1883822859999782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
W20260709 13:03:49.110582  1706 connection.cc:256] RPC call timeout handler was delayed by -1.1712822860000074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
I20260709 13:03:49.068240  7662 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.108407  7711 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.092810  7660 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.129868  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2250) took 3250 ms. Trace:
I20260709 13:03:49.130426  1659 rpcz_store.cc:276] 0709 13:03:45.872135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.907309 (+ 35174us) service_pool.cc:224] Handling call
0709 13:03:45.907686 (+   377us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:45.925688 (+ 18002us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 13:03:45.937942 (+ 12254us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.613456 (+675514us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.618869 (+  5413us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:46.621326 (+  2457us) log.cc:844] Serialized 52 byte log entry
0709 13:03:46.621481 (+   155us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:46.621691 (+   210us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.621944 (+   253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:46.622046 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:48.695032 (+2072986us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:48.727664 (+ 32632us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b18003d5e80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:49.121965 (+394301us) raft_consensus.cc:1685] finished
0709 13:03:49.122006 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.122229 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":882,"prepare.queue_time_us":17768,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":151,"raft.queue_time_us":6754,"raft.run_cpu_time_us":531,"raft.run_wall_time_us":13341,"spinlock_wait_cycles":30428672,"thread_start_us":670469,"threads_started":1,"wal-append.queue_time_us":910917}
I20260709 13:03:49.141429  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.142695  1586 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:03:49.143575  1591 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:49.143842  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.144559  1593 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:49.102888  7748 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.122820  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:49.147722  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:49.144734  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.148397  1593 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:49.147974  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.149391  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.150561  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.120648  7636 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.152726  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.154914  1593 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:49.155552  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:49.158970  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.159793  1733 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:49.162657  1586 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:49.162714  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.163436  1591 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:49.164824  2281 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.545s (SENT)
I20260709 13:03:49.165140  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:49.166302  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.166759  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.134509  7836 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:49.171315  1741 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:49.169895  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.172389  1738 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:49.176194  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3041) took 3801 ms. Trace:
I20260709 13:03:49.153991  7663 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
W20260709 13:03:49.184473  2281 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.562s (SENT) [suppressed 38 similar messages]
I20260709 13:03:49.176874  1589 rpcz_store.cc:276] 0709 13:03:45.374758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.986269 (+611511us) service_pool.cc:224] Handling call
0709 13:03:45.986622 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:45.994216 (+  7594us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:46.580369 (+586153us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.974429 (+394060us) log.cc:844] Serialized 14 byte log entry
0709 13:03:46.974649 (+   220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:46.974719 (+    70us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:46.974790 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.691598 (+1716808us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b18004cdc20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:48.825329 (+133731us) spinlock_profiling.cc:243] Waited 20.6 ms on lock 0x7b18004cdc20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1
0709 13:03:48.826018 (+   689us) spinlock_profiling.cc:243] Waited 6.57 ms on lock 0x7b18004cdc20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:49.175711 (+349693us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b5400330c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:49.175766 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.176096 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":126515,"prepare.queue_time_us":793601,"prepare.run_cpu_time_us":147,"prepare.run_wall_time_us":148,"raft.queue_time_us":34083,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":38223,"spinlock_wait_cycles":109735168,"thread_start_us":852312,"threads_started":2,"wal-append.queue_time_us":478566,"wal-append.run_cpu_time_us":2040,"wal-append.run_wall_time_us":1071483}
I20260709 13:03:49.145180  7711 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:03:49.170096  7694 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.176188  7783 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.182431  7828 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.118250  7662 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.186305  7846 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:49.197472  7663 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:49.205189  7694 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.206214  7748 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.206954  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.205209  7783 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.210062  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.210891  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.241689  1742 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:49.211894  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.212329  7609 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.213043  7833 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:49.214879  7630 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.002s
W20260709 13:03:49.218107  7839 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:03:49.218981  7457 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:49.221859  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:49.227651  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.229959  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.255574  1738 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:49.235349  7596 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:49.204464  7832 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.377s	user 0.001s	sys 0.000s
I20260709 13:03:49.241859  1741 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:49.211745  7841 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:49.204061  7704 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.791s	user 0.013s	sys 0.002s
I20260709 13:03:49.239817  7494 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:49.244984  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.272217  1736 raft_consensus.cc:2376] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17. Re-sending same reply.
I20260709 13:03:49.251333  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.180964  7836 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.236268  7828 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.296568  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3131) took 1979 ms. Trace:
I20260709 13:03:49.297111  1592 rpcz_store.cc:276] 0709 13:03:47.317649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.730229 (+1412580us) service_pool.cc:224] Handling call
0709 13:03:48.730711 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:48.734193 (+  3482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:48.734244 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:48.735949 (+  1705us) log.cc:844] Serialized 26 byte log entry
0709 13:03:48.824919 (+ 88970us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:48.824958 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:48.825025 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:49.296320 (+471295us) raft_consensus.cc:1685] finished
0709 13:03:49.296357 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.296494 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3130,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":3789,"thread_start_us":86118,"threads_started":1,"wal-append.queue_time_us":352321}
I20260709 13:03:49.296576  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.300879  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:49.301976  1742 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:49.178587  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:49.304469  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 125910us
I20260709 13:03:49.277491  7672 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.308830  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.300580  7846 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:49.306845  7838 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 13:03:49.267956  1662 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:03:49.268968  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.257264  7457 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:49.308868  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.300257  7831 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.306962  7833 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:49.314502  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.316134  1741 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:49.315344  7820 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.794s	user 0.009s	sys 0.009s
I20260709 13:03:49.313133  7660 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.302404  7663 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.257902  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.336549  1738 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:49.232060  1591 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
W20260709 13:03:49.324352  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 139864us
I20260709 13:03:49.342555  7847 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:49.342376  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.343082  7847 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.343158  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.343601  7847 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:49.322938  7831 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 13:03:49.344724  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:49.324129  7774 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.762s	user 0.008s	sys 0.006s
I20260709 13:03:49.344652  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.344725  7828 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.350525  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.351713  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.314877  7846 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.353668  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.354091  7846 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.354293  1737 raft_consensus.cc:2376] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17. Re-sending same reply.
I20260709 13:03:49.348017  7694 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.334673  7845 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.355736  7845 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.356204  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.356904  1737 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:49.358919  2315 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:03:49.359246  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:49.360102  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2267) took 1292 ms (client timeout 1692 ms). Trace:
I20260709 13:03:49.349709  1593 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:03:49.360503  1666 rpcz_store.cc:269] 0709 13:03:48.067398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.077599 (+ 10201us) service_pool.cc:224] Handling call
0709 13:03:49.359043 (+1281444us) spinlock_profiling.cc:243] Waited 5.2 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:03:49.360037 (+   994us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11970944}
I20260709 13:03:49.362898  7783 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.320655  7704 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 13:03:49.364353  7704 consensus_queue.cc:1243] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:49.364866  7704 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:49.365216  7704 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:49.366051  7704 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.370770  1659 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:03:49.341684  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.375653  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:49.376320  1665 raft_consensus.cc:2376] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18. Re-sending same reply.
I20260709 13:03:49.378376  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.383960  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.384660  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:49.385300  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:49.385941  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.386745  1737 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
W20260709 13:03:49.387977  1705 connection.cc:256] RPC call timeout handler was delayed by -1.6169380950000232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
W20260709 13:03:49.388551  1705 connection.cc:256] RPC call timeout handler was delayed by -1.4846380950000366s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20260709 13:03:49.388978  1705 connection.cc:256] RPC call timeout handler was delayed by -1.4693380950000119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20779999999999998s for a response
W20260709 13:03:49.389474  1705 connection.cc:256] RPC call timeout handler was delayed by -1.3649380950000136s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2194s for a response
W20260709 13:03:49.389890  1705 connection.cc:256] RPC call timeout handler was delayed by -1.3566223610000065s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 13:03:49.390383  1705 connection.cc:256] RPC call timeout handler was delayed by -1.194622360999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2001s for a response
I20260709 13:03:49.392135  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:49.393249  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.394676  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.371809  7828 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.397063  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.398639  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.399272  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.399415  2315 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:49.401621  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.402436  1664 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:49.369969  7845 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.404031  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:49.397059  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.391343  7748 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:49.394068  7819 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:49.394272  7783 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:49.395979  7831 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.430800  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.416501  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:49.416141  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115297us
W20260709 13:03:49.432538  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49717.5us
I20260709 13:03:49.432601  7480 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:49.433169  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:49.435220  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46592us
I20260709 13:03:49.436317  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:49.436614  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:49.437386  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.439389  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.397902  7848 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.403316  7662 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.428562  7836 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.429698  7819 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.399953  7828 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.454147  7650 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:49.484539  7860 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:03:49.456545  7854 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:49.442515  7783 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:49.440482  7831 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.438974  7711 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:03:49.441143  7749 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:49.462087  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45690.4us
I20260709 13:03:49.464227  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.472934  7833 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:49.471326  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:49.480492  7774 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:49.480218  7848 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.481153  7819 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:49.481053  7748 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.483613  7828 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:49.488185  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49038.7us
I20260709 13:03:49.464993  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:49.507944  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:49.508662  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43764.1us
I20260709 13:03:49.510811  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:49.493134  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.478965  7775 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:49.504700  2316 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:49.464561  7845 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:49.513819  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.490767  7662 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:49.491693  7854 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.523931  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:49.521111  1632 leader_election.cc:336] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.075s (SENT)
I20260709 13:03:49.505201  7833 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.530308  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:49.535162  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.536149  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.533625  1739 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:49.540362  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.541603  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.513593  7711 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:49.517942  7828 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:49.518640  7775 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.525269  7694 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:49.532243  7851 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:49.548950  1735 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 13:03:49.550839  1735 pending_rounds.cc:85] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
I20260709 13:03:49.551195  1735 pending_rounds.cc:107] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:03:49.551465  1735 raft_consensus.cc:2889] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:49.551537  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.537739  7833 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:49.531884  7854 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:49.543285  7609 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:49.513206  7748 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.546206  7662 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:49.566586  7767 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:03:49.568634  7831 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:49.572372  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1038) took 1773 ms. Trace:
I20260709 13:03:49.547945  7694 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.531792  7846 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 13:03:49.573840  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:49.574438  7711 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.575731  7774 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:49.576014  7650 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:49.578194  7457 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:49.664204  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.664898  7457 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.664520  7819 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.658697  7855 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:49.580185  7847 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.666304  1593 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:49.667605  7847 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.581861  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:03:49.668567  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86772.3us
I20260709 13:03:49.668694  7846 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:49.669265  7846 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.669368  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:49.669697  7846 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.670354  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:49.548077  7775 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.555953  7663 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:49.670388  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3135) took 2351 ms. Trace:
I20260709 13:03:49.592794  7636 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.665540  7457 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:49.578558  7748 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.672742  1587 rpcz_store.cc:276] 0709 13:03:47.318807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.345904 (+ 27097us) service_pool.cc:224] Handling call
0709 13:03:49.670192 (+2324288us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540031db54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:49.670320 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":337474432}
W20260709 13:03:49.626147  7865 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 13:03:49.596609  1740 rpcz_store.cc:276] 0709 13:03:47.799274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.799752 (+   478us) service_pool.cc:224] Handling call
0709 13:03:47.800150 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.039368 (+239218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.572084 (+1532716us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:49.572302 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423147264}
I20260709 13:03:49.592657  7749 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.680454  1735 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 17 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:49.623770  7694 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:49.672878  7609 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:49.724087  2314 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
I20260709 13:03:49.695340  7831 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:49.727627  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:49.717399  7748 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.742107  1735 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:49.734323  7836 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 13:03:49.746536  7748 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.754258  1735 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:49.754170  7748 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.724159  7663 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 13:03:49.719710  7662 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:49.762703  7662 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:49.763799  7662 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.736963  7831 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:49.764930  7831 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:49.765933  7694 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.768199  7694 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.762739  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:49.769145  7775 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.764788  7848 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.773736  7848 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.774246  7848 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.776697  7848 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:49.777230  7848 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:49.559293  7783 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.756001  7819 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.778043  7775 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.783473  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 2969) took 3843 ms. Trace:
I20260709 13:03:49.721966  7847 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 13:03:49.787240  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3052) took 3837 ms. Trace:
I20260709 13:03:49.787729  1584 rpcz_store.cc:276] 0709 13:03:45.949330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.494065 (+1544735us) service_pool.cc:224] Handling call
0709 13:03:47.494494 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:47.754893 (+260399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.786882 (+2031989us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b5400431a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:49.787141 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":596678400}
I20260709 13:03:49.783938  1667 rpcz_store.cc:276] 0709 13:03:45.939919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.522288 (+582369us) service_pool.cc:224] Handling call
0709 13:03:48.629655 (+2107367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.629895 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:48.784765 (+154870us) log.cc:844] Serialized 14 byte log entry
0709 13:03:49.089469 (+304704us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:49.089554 (+    85us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:49.089613 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.089694 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:49.782762 (+693068us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b1800296b60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4
0709 13:03:49.783273 (+   511us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54000aa5d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:49.783406 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9867,"prepare.queue_time_us":191368,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":4071,"raft.queue_time_us":1174,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":5597,"spinlock_wait_cycles":4847754880,"thread_start_us":457719,"threads_started":2,"wal-append.queue_time_us":357091}
I20260709 13:03:49.717064  7457 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:49.697827  7636 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.725310  7749 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:49.807718  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.809732  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 288512us
I20260709 13:03:49.811605  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3121) took 3962 ms. Trace:
W20260709 13:03:49.813730  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.814682  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 282032us
I20260709 13:03:49.827805  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.863575  1584 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:49.864727  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.828059  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.829140  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.865679  1587 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:49.866334  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.867151  1591 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:49.829421  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.832453  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.867637  1592 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:49.867733  1588 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:49.868762  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.869613  1588 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:49.833781  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.835109  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:49.840200  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:49.841554  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.845940  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.859308  1585 rpcz_store.cc:276] 0709 13:03:45.848920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:46.524414 (+675494us) service_pool.cc:224] Handling call
0709 13:03:48.827365 (+2302951us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:48.827993 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:49.811311 (+983318us) spinlock_profiling.cc:243] Waited 2.3 s on lock 0x7b54004001d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:49.811524 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5292923264}
I20260709 13:03:49.873751  1589 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:49.872133  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.874308  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.874490  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.875325  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.875399  1584 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:49.876153  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:49.876480  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 324848us
I20260709 13:03:49.877689  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.877836  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:49.879724  1588 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20260709 13:03:49.881023  1586 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 17. Ops: []
W20260709 13:03:49.884138  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:49.882368  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:49.885991  1589 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:49.888118  1588 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:49.889571  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.890511  1659 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:49.892771  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.893973  2315 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:49.855695  7468 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.897891  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:49.890879  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 219120us
I20260709 13:03:49.902175  1585 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:49.904009  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.906415  1667 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:49.878492  7748 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:49.916777  1584 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:49.904475  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:49.904009  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:49.908905  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.899497  1592 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:03:49.918015  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.954195  1659 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 11: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:49.921859  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.924333  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.924347  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.954708  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:49.955394  1663 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:49.925931  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:49.929945  7875 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:49.938513  7749 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:49.904248  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:49.957064  1587 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:49.958707  2312 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:49.960522  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.966090  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.968621  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.969389  1591 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:49.969641  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.969425  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:49.957999  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 230362us
I20260709 13:03:49.964656  1589 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:49.940986  7871 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:49.976279  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.972363  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:49.951314  7468 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.974668  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.989529  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:49.959321  7873 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:49.990475  1659 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:49.903210  7854 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:49.998035  1740 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:49.992996  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:49.965517  7609 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:49.964634  7879 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:50.011322  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:50.012116  1741 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:50.013334  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:50.026726  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 150557us
I20260709 13:03:50.028837  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.032581  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:50.155562  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 143166us
W20260709 13:03:50.157362  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 124853us
I20260709 13:03:50.157478  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.158427  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:50.158880  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.160579  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:49.964118  7876 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:03:49.991740  7884 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:50.167279  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 379827us
W20260709 13:03:50.172760  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 263742us
I20260709 13:03:50.172658  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.175616  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.175984  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:50.176680  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 258728us
I20260709 13:03:50.178478  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.180410  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.192524  1667 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:50.002159  7882 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:50.156806  7416 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:49.987494  7871 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.167735  7875 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:50.163352  7867 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:49.991189  7468 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.169100  7749 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:50.156594  7662 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:50.197127  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.173027  7457 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.165002  7707 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:50.196506  7833 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.636s	user 0.012s	sys 0.000s
I20260709 13:03:50.200594  7879 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:50.220503  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:50.234303  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75908.3us
I20260709 13:03:50.165841  7855 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 13:03:50.009985  7609 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.199483  7882 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:50.230285  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52798.3us
I20260709 13:03:50.171125  7847 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.049s
W20260709 13:03:50.243150  7878 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 13:03:50.004436  7873 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.213248  7875 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.234944  1667 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:50.236986  1584 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:50.251726  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.232635  7867 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:50.240617  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43530us
I20260709 13:03:50.241957  7416 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:50.239573  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.240911  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.221222  7883 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:50.211323  7636 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:50.258589  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:50.260246  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.264065  1584 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:50.261248  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.265690  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.265817  1666 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:50.273905  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.244382  7882 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:50.246512  7662 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:50.225492  7831 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.273010  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.252063  7875 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:50.180470  7884 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:50.290499  7886 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 13:03:50.256667  7897 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:50.259207  7707 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 13:03:50.239012  7833 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.266235  7457 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:50.291314  7877 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.288s	user 0.000s	sys 0.001s
I20260709 13:03:50.302542  1584 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:50.314826  1586 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 13:03:50.286972  7846 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.617s	user 0.011s	sys 0.003s
I20260709 13:03:50.267735  7851 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.048s
W20260709 13:03:50.330204  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110508us
I20260709 13:03:50.301452  7883 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.278122  7881 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 13:03:50.270470  7879 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:50.342059  1735 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 17. Ops: []
I20260709 13:03:50.306082  7898 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:03:50.500223  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 119717us
I20260709 13:03:50.339093  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.324723  7748 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:50.326910  7847 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:50.298256  7893 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.218s
W20260709 13:03:50.567037  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 227949us
I20260709 13:03:50.568563  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.576010  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.327301  7846 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.500034  7894 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.250s
I20260709 13:03:50.591045  1737 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:03:50.571681  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 311337us
I20260709 13:03:50.643076  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:50.597997  1737 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:50.608317  1741 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.596s	user 0.009s	sys 0.005s
I20260709 13:03:50.506989  7867 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.572999  7457 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader election won for term 11
W20260709 13:03:50.585294  7888 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 13:03:50.633289  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 993) took 6640 ms. Trace:
I20260709 13:03:50.340729  7707 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.231s
I20260709 13:03:50.525774  7847 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.570971  7662 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:50.645215  1741 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:50.589421  7851 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:50.649210  1661 rpcz_store.cc:276] 0709 13:03:43.953706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:45.273578 (+1319872us) service_pool.cc:224] Handling call
0709 13:03:45.324508 (+ 50930us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:45.823224 (+498716us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:45.823269 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:45.843641 (+ 20372us) log.cc:844] Serialized 26 byte log entry
0709 13:03:47.742645 (+1899004us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b18004a1940. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:47.785194 (+ 42549us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:47.785327 (+   133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.593921 (+2808594us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:50.593984 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:50.594056 (+    72us) raft_consensus.cc:1685] finished
0709 13:03:50.594071 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.594251 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67227,"raft.queue_time_us":36094,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":16066,"spinlock_wait_cycles":1084667392,"thread_start_us":33802,"threads_started":1,"wal-append.queue_time_us":2098592,"wal-append.run_cpu_time_us":1908,"wal-append.run_wall_time_us":1105648}
I20260709 13:03:50.585755  7468 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:50.578316  7854 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:50.664927  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 319142us
I20260709 13:03:50.577905  7900 raft_consensus.cc:2764] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 13:03:50.667958  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.670457  1741 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 11.
I20260709 13:03:50.515693  7898 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:50.673863  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.645867  7894 consensus_queue.cc:1243] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 13:03:50.675176  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 99204us
I20260709 13:03:50.644531  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.648085  7871 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:50.694723  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:50.659896  7879 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:50.695428  1741 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:50.666273  7847 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:50.665745  7662 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.668090  7457 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:50.663096  7851 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:50.682108  1737 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.670236  7897 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:50.692071  7854 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:50.693861  7609 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:50.697803  7879 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:50.739578  7851 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.734326  7867 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.671000  7883 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.735118  7831 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:50.735359  7854 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.705948  7873 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.702421  7662 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:50.756910  7884 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:50.758275  7457 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.762319  7898 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:50.767763  1706 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.827s	user 0.012s	sys 0.036s
I20260709 13:03:50.768548  7879 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:50.777043  1741 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:50.767819  7872 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:50.779559  7903 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:03:50.780303  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 135783us
W20260709 13:03:50.783197  1706 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.012s	sys 0.036s
W20260709 13:03:50.787586  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.709990  1737 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:50.791548  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:50.779569  7661 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:50.783540  7851 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:50.814424  7894 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:50.816727  7846 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:50.790738  7831 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.817950  7831 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.820509  7831 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:50.826391  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3014) took 1620 ms (client timeout 2075 ms). Trace:
W20260709 13:03:50.826947  1736 rpcz_store.cc:269] 0709 13:03:49.205698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.206646 (+   948us) service_pool.cc:224] Handling call
0709 13:03:50.826180 (+1619534us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b54001095d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:50.826320 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72489216}
I20260709 13:03:50.829793  7766 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:50.830299  7766 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.831280  7766 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.833727  7507 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:50.834046  7480 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:50.834178  7507 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.834517  7480 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:50.835143  7507 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.835466  7480 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.836197  7507 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:50.836500  7480 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:50.842262  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2273) took 2735 ms. Trace:
I20260709 13:03:50.842715  1660 rpcz_store.cc:276] 0709 13:03:48.107388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.377825 (+270437us) service_pool.cc:224] Handling call
0709 13:03:49.098896 (+721071us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.099788 (+   892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.842010 (+1742222us) spinlock_profiling.cc:243] Waited 720 ms on lock 0x7b5400260f50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:50.842182 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1656516352}
W20260709 13:03:50.849164  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3044) took 1326 ms (client timeout 1755 ms). Trace:
W20260709 13:03:50.849685  1739 rpcz_store.cc:269] 0709 13:03:49.523084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.523583 (+   499us) service_pool.cc:224] Handling call
0709 13:03:50.848963 (+1325380us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b54000e3654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:50.849094 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8040704}
I20260709 13:03:50.850915  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:50.851723  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1057) took 1078 ms. Trace:
I20260709 13:03:50.852660  1739 rpcz_store.cc:276] 0709 13:03:49.773553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.850620 (+1077067us) service_pool.cc:224] Handling call
0709 13:03:50.851667 (+  1047us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.853564  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.854043  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1042) took 1558 ms. Trace:
I20260709 13:03:50.854566  1735 rpcz_store.cc:276] 0709 13:03:49.297969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.770239 (+472270us) service_pool.cc:224] Handling call
0709 13:03:49.770627 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.499537 (+728910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.853742 (+354205us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:50.853979 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312970624}
I20260709 13:03:50.854154  1739 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:50.855912  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:50.856525  1735 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:50.856863  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.857493  1739 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 10.
I20260709 13:03:50.858371  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.859392  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:50.859503  1593 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 980) took 1481 ms (client timeout 1648 ms). Trace:
W20260709 13:03:50.860152  1593 rpcz_store.cc:269] 0709 13:03:49.377923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.378088 (+   165us) service_pool.cc:224] Handling call
0709 13:03:50.859299 (+1481211us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b5400393ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:50.859438 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":659611392}
I20260709 13:03:50.860999  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1043) took 1562 ms. Trace:
I20260709 13:03:50.862867  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1044) took 1564 ms. Trace:
I20260709 13:03:50.863266  1733 rpcz_store.cc:276] 0709 13:03:49.298500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.862081 (+1563581us) service_pool.cc:224] Handling call
0709 13:03:50.862377 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.862563 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.862589 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.862631 (+    42us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:50.862660 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.862717 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.862813 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.864671  1733 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:50.865255  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1045) took 1566 ms. Trace:
I20260709 13:03:50.866173  1733 rpcz_store.cc:276] 0709 13:03:49.298739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.864208 (+1565469us) service_pool.cc:224] Handling call
0709 13:03:50.864498 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.865015 (+   517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.865198 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.828270  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.869922  1736 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:50.861778  1739 rpcz_store.cc:276] 0709 13:03:49.298252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.860227 (+1561975us) service_pool.cc:224] Handling call
0709 13:03:50.860505 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.860695 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.860720 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.860764 (+    44us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:50.860797 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.860851 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.860949 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.868302  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:50.786109  7903 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.872385  7903 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:50.795409  7872 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.874176  7741 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:50.875255  1739 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:50.875023  7741 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:50.876958  7741 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:50.877571  7872 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.879959  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1049) took 1580 ms. Trace:
I20260709 13:03:50.867689  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:50.882397  1739 rpcz_store.cc:276] 0709 13:03:49.300426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.870772 (+1570346us) service_pool.cc:224] Handling call
0709 13:03:50.871057 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.879667 (+  8610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.879894 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.792303  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:50.890043  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 97784.4us
W20260709 13:03:50.893113  2280 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.215s (SENT)
I20260709 13:03:50.906226  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:50.906306  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.896836  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1048) took 1597 ms. Trace:
I20260709 13:03:50.900460  2318 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:50.908052  1736 rpcz_store.cc:276] 0709 13:03:49.299607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.870622 (+1571015us) service_pool.cc:224] Handling call
0709 13:03:50.896116 (+ 25494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.896311 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.896334 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.896377 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:50.896406 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.896452 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.896740 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.902637  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:50.911850  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.000s	sys 0.007s
W20260709 13:03:50.912258  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.000s	sys 0.007s
I20260709 13:03:50.900097  7841 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:50.916357  7650 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:50.916404  7407 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:50.917209  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1050) took 1616 ms. Trace:
W20260709 13:03:50.917815  7801 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 3.145s	user 0.013s	sys 0.018s
I20260709 13:03:50.917774  1739 rpcz_store.cc:276] 0709 13:03:49.300757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.916270 (+1615513us) service_pool.cc:224] Handling call
0709 13:03:50.916669 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.916867 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.916892 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.916939 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:50.916971 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.917032 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.917149 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:50.900434  7869 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:50.921613  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.920795  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3030) took 1683 ms. Trace:
I20260709 13:03:50.922787  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.924685  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:50.928265  7841 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 13:03:50.932194  7841 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:50.925817  7407 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:50.932586  7841 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:50.932928  7407 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.919302  7869 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:50.926182  7855 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:50.939481  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.943153  1593 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:50.942133  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.944125  1664 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 14 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20260709 13:03:50.923503  7650 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.944334  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.947829  1593 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:50.944985  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.949057  1587 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:50.945720  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.946712  1584 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:50.927359  7619 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:50.926966  7872 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:50.941560  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:50.952436  1584 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:50.927623  7854 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.963179  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.922775  1737 rpcz_store.cc:276] 0709 13:03:49.237100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.521714 (+284614us) service_pool.cc:224] Handling call
0709 13:03:49.527319 (+  5605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.590911 (+1063592us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b54000350a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfdf1c0 00007fcdedfe0231 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c
0709 13:03:50.920415 (+329504us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:50.920452 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:50.920498 (+    46us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:50.920533 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.920605 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:50.920729 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":237,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":4028,"spinlock_wait_cycles":13889920}
I20260709 13:03:50.966066  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.974278  1736 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:50.975178  1593 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:03:50.943162  7869 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:50.976872  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:50.971558  7650 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.980043  7650 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.929661  7601 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:50.971930  7855 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:50.932593  7611 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:50.981846  7601 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.982398  7611 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.982743  7855 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.984196  7601 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.984439  7611 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:50.933645  7841 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.986265  7841 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:50.986759  7882 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.688s	user 0.010s	sys 0.001s
I20260709 13:03:50.987191  7882 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:50.988396  1667 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:50.990491  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.975781  1735 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:50.991526  1733 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:50.991629  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:50.934007  7407 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:50.992964  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:50.993852  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:50.997372  1593 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:50.981586  1664 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:50.995249  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:50.998287  1735 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:50.999841  1661 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:51.003139  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.043207  1660 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:51.044252  1660 pending_rounds.cc:85] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:03:51.044445  1660 pending_rounds.cc:107] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 13:03:51.044585  1660 raft_consensus.cc:2889] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:51.004727  1593 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:51.005362  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.045151  1736 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:50.989346  7882 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:50.960827  7619 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.023223  1584 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:50.995215  7873 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:50.995616  7880 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.001034  7901 raft_consensus.cc:2764] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 13:03:51.032977  7441 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.039386  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:51.040638  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167679us
I20260709 13:03:51.062274  1659 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:51.063521  1659 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:03:51.039573  7631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.599s	user 0.020s	sys 0.038s
I20260709 13:03:51.063606  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.035432  7898 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.053542  7619 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.053123  7873 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.089361  7869 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.097333  7873 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.098241  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3033) took 1848 ms. Trace:
I20260709 13:03:51.098618  1740 rpcz_store.cc:276] 0709 13:03:49.249434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.718624 (+469190us) service_pool.cc:224] Handling call
0709 13:03:49.719059 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.998793 (+279734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.097914 (+1099121us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:51.098171 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":639445760}
I20260709 13:03:51.099897  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.100535  1740 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:51.053345  7912 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.101979  7912 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:51.102721  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2287) took 2586 ms (client timeout 1974 ms). Trace:
W20260709 13:03:51.103149  1662 rpcz_store.cc:269] 0709 13:03:48.516832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.608293 (+ 91461us) service_pool.cc:224] Handling call
0709 13:03:51.102510 (+2494217us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.102652 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1507070336}
I20260709 13:03:51.058887  7879 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.046321  7847 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.083833  7917 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:51.104372  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.132817  1735 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:03:51.105424  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.096069  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1100) took 1698 ms. Trace:
I20260709 13:03:51.134730  2314 rpcz_store.cc:276] 0709 13:03:49.397224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.397823 (+   599us) service_pool.cc:224] Handling call
0709 13:03:49.405924 (+  8101us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:49.725332 (+319408us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.095732 (+1370400us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54005ae454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:51.095990 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729549056}
I20260709 13:03:51.101415  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.110029  1659 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:03:51.112784  1662 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:51.137912  1740 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 13:03:51.112931  7912 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.106194  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 972) took 3351 ms. Trace:
W20260709 13:03:51.114595  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56262.7us
I20260709 13:03:51.066835  7880 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.127745  7879 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:51.138595  1590 rpcz_store.cc:276] 0709 13:03:47.754455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:48.786837 (+1032382us) service_pool.cc:224] Handling call
0709 13:03:49.671113 (+884276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.807348 (+136235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.807398 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:49.807461 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:49.807508 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.096519 (+1289011us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b540031db54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:51.096571 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.105936 (+  9365us) spinlock_profiling.cc:243] Waited 883 ms on lock 0x7b540031db50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.106101 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2344028672}
I20260709 13:03:51.084913  7662 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.050151  1664 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:51.140681  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:51.141495  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:51.163447  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98238.9us
W20260709 13:03:51.068500  7631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.651s	user 0.020s	sys 0.039s
W20260709 13:03:51.167249  7494 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.612s	user 0.001s	sys 0.018s
W20260709 13:03:51.138012  7875 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.814s	user 0.011s	sys 0.001s
I20260709 13:03:51.115104  7869 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.176277  1634 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:51.179281  1665 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:51.208709  1736 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
W20260709 13:03:51.179881  7494 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.625s	user 0.002s	sys 0.019s
I20260709 13:03:51.212040  1734 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:51.128628  7881 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 13:03:51.214828  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.215135  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.215471  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.215551  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.182219  7875 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.216017  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.216184  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.115885  7898 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.222852  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.352105  1584 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:51.211500  7869 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:51.286557  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:51.270301  7915 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:03:51.310164  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:51.233566  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.355314  1593 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:51.377031  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:51.258647  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.356019  1735 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:51.356871  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:51.314455  7880 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:51.290170  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3025) took 2067 ms (client timeout 1522 ms). Trace:
W20260709 13:03:51.385226  1742 rpcz_store.cc:269] 0709 13:03:49.223012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.242483 (+ 19471us) service_pool.cc:224] Handling call
0709 13:03:51.289855 (+2047372us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b5400403b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.290051 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":65496448}
I20260709 13:03:51.179879  7912 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:51.173873  7917 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:51.292932  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.392624  1738 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.361486  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.392932  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.393718  1584 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:51.352860  1590 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:51.359769  1736 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.364714  2318 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:51.294598  7640 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:51.366374  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:51.235186  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.371631  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:51.397091  2315 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.374146  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:51.274237  7916 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 13:03:51.377687  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.383006  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.398739  1740 raft_consensus.cc:2376] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:03:51.399716  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.221096  7855 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.163s
I20260709 13:03:51.376595  7920 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.292622  7847 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.294013  7773 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.358498  7461 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.377125  7856 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:51.173609  7903 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.367069  7441 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.390201  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.433995  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.434358  1735 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:51.400911  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.286276  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.435597  1740 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:51.404911  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.402316  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:51.436472  2317 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:51.405016  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.437122  1738 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.437544  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:51.436511  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.405957  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.438320  1740 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:51.438402  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.438762  1736 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:51.409229  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.439268  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.439711  1742 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:51.377521  7662 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:51.387729  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:51.418037  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.424556  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.409608  7833 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.379175  7619 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.470525  7619 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.470350  7873 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.428810  2312 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:51.421353  1593 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:51.430733  2315 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:51.434207  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.439291  1659 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:51.472646  7873 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.400336  7640 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:51.398029  7917 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:51.442534  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.473978  1663 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:51.443094  1740 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:51.472780  1739 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:51.399853  7867 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 13:03:51.474243  7917 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.475224  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.475708  7917 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:51.475224  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.476768  1740 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.477056  7828 raft_consensus.cc:993] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:51.476778  7773 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.478096  7773 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.480333  7773 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:51.443724  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56072.3us
I20260709 13:03:51.445518  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.444893  7441 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.447227  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.482874  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.451529  7903 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:51.452466  7792 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:51.484412  7792 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:51.452476  1742 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:51.486229  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:51.486817  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.485348  7792 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.467562  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.467980  7898 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:51.407167  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2295) took 1463 ms. Trace:
I20260709 13:03:51.489852  1667 rpcz_store.cc:276] 0709 13:03:49.943544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.258240 (+314696us) service_pool.cc:224] Handling call
0709 13:03:51.406958 (+1148718us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b540013ced4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.407102 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640186752}
I20260709 13:03:51.490303  7773 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:51.475826  1739 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.481637  1738 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:51.490828  7773 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.491297  7773 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.491698  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.492817  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.483459  7441 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.493844  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.494429  1738 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:51.494797  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 988) took 1591 ms. Trace:
I20260709 13:03:51.495283  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.495980  1738 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:51.495216  1592 rpcz_store.cc:276] 0709 13:03:49.902915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.957194 (+ 54279us) service_pool.cc:224] Handling call
0709 13:03:49.957489 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:49.975459 (+ 17970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:49.975493 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:49.992290 (+ 16797us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.259094 (+266804us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:03:50.259233 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.494354 (+1235121us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b54001c4654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:51.494409 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.494462 (+    53us) raft_consensus.cc:1685] finished
0709 13:03:51.494475 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.494733 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":588,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":8985,"spinlock_wait_cycles":36842240,"thread_start_us":80447,"threads_started":1,"wal-append.queue_time_us":514672}
I20260709 13:03:51.483490  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:51.497718  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3153) took 1660 ms. Trace:
I20260709 13:03:51.498095  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.498981  1736 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:51.499958  7773 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.500108  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.486914  1742 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:51.501040  1736 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 10 FOLLOWER]: Advancing to term 11
I20260709 13:03:51.498124  1591 rpcz_store.cc:276] 0709 13:03:49.843579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.885512 (+ 41933us) service_pool.cc:224] Handling call
0709 13:03:49.969206 (+ 83694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:49.969913 (+   707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.497516 (+1527603us) spinlock_profiling.cc:243] Waited 83.3 ms on lock 0x7b540031d3d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.497655 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191685888}
W20260709 13:03:51.501911  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2298) took 1557 ms (client timeout 1650 ms). Trace:
W20260709 13:03:51.502357  1666 rpcz_store.cc:269] 0709 13:03:49.944243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.968315 (+ 24072us) service_pool.cc:224] Handling call
0709 13:03:51.501747 (+1533432us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b5400014ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:51.501852 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":681721600}
I20260709 13:03:51.502988  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.503762  1742 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.504540  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.505587  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.506556  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.423434  7920 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.509764  1736 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:51.510502  7860 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:51.510728  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.510996  7860 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.511536  1736 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.431239  7856 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.513103  7860 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.514079  7773 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.514295  1667 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:51.510504  7903 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 13:03:51.507194  1742 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:03:51.514917  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47405.1us
I20260709 13:03:51.496937  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:51.511581  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.428357  7461 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.528843  1740 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:51.530207  1662 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:03:51.530165  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.519923  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:51.536314  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.537190  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1051) took 2236 ms. Trace:
I20260709 13:03:51.537305  1586 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:51.520488  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.520707  1591 raft_consensus.cc:1217] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Deduplicated request from leader. Original: 13.8->[16.9-16.9]   Dedup: 16.9->[]
I20260709 13:03:51.538254  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:51.538872  1591 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:03:51.466559  7875 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.512166  1739 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:51.524613  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.539887  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:51.526960  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:51.527185  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:51.528091  1708 connection.cc:256] RPC call timeout handler was delayed by -1.2459345439999652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33230000000000004s for a response
I20260709 13:03:51.528213  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.546059  1590 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.549609  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:51.550021  1739 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:51.550637  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3042) took 2028 ms. Trace:
I20260709 13:03:51.514153  7856 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.551016  1739 rpcz_store.cc:276] 0709 13:03:49.522457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.549387 (+2026930us) service_pool.cc:224] Handling call
0709 13:03:51.549688 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.550391 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.550582 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:51.544761  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:51.551941  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:51.555383  1738 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:51.556481  1739 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 13:03:51.559038  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.558924  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:51.559470  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0352211809999403s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1814s for a response
I20260709 13:03:51.559811  2310 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:51.560110  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0028211809999448s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18500000000000003s for a response
I20260709 13:03:51.529572  7828 raft_consensus.cc:1081] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:03:51.552433  7833 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:51.514665  7847 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:51.537856  1740 rpcz_store.cc:276] 0709 13:03:49.301017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.536108 (+2235091us) service_pool.cc:224] Handling call
0709 13:03:51.536516 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.536751 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:51.536789 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:51.536853 (+    64us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:51.536900 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.536973 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:51.537118 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:51.568828  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:51.562410  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3043) took 2039 ms. Trace:
I20260709 13:03:51.569993  1738 rpcz_store.cc:276] 0709 13:03:49.522776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.552618 (+2029842us) service_pool.cc:224] Handling call
0709 13:03:51.553117 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.562122 (+  9005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.562346 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:51.570472  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:51.570016  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.561175  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:51.540059  1736 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 13:03:51.548209  7416 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:51.558305  7441 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:51.554718  7748 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:51.567430  7854 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 13:03:51.582253  7748 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.554271  7882 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 13:03:51.582810  7748 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.583591  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.584313  1587 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.509831  7920 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.590488  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.537591  7917 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:51.575209  1593 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:51.573874  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:51.568782  7847 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.570690  7823 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:51.578161  7441 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:51.578249  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.474265  7640 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.611711  1590 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:03:51.585774  7860 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 13:03:51.591835  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.589871  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:51.588114  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 422703us
W20260709 13:03:51.593583  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73714us
I20260709 13:03:51.548720  7898 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
I20260709 13:03:51.596475  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:51.597334  7851 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:51.567238  7833 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:51.571455  7875 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:51.599351  1738 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:03:51.594703  7416 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.581350  7494 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:51.561362  7855 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 13:03:51.614364  1736 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:51.619864  1739 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:51.610832  7847 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:51.587702  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:51.631201  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:51.631050  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52856.5us
I20260709 13:03:51.544196  7773 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
W20260709 13:03:51.637122  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47317.4us
I20260709 13:03:51.660812  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.642295  1587 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:51.595135  7619 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:51.595916  7672 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:51.649304  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.613022  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:51.627018  7441 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:51.628176  7917 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:51.690904  7917 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.640250  7851 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:51.693975  7851 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:51.634584  7823 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:51.673633  2312 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:03:51.647657  7416 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.708125  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.709021  1739 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:51.699893  2310 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:51.711642  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:51.675258  1587 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:03:51.677438  1590 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:51.678551  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:51.670414  7879 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:51.647955  7494 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.716342  1739 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:51.716055  1736 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 13: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:51.694459  7899 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.047s	user 0.003s	sys 0.000s
I20260709 13:03:51.682962  7748 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.687798  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:51.720403  1659 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:51.671387  7649 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:51.643718  7875 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:03:51.720180  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59411.6us
I20260709 13:03:51.682034  7922 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:03:51.725379  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.723953  1634 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.727989  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:51.676892  7672 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:51.751151  1736 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:03:51.733039  2312 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:03:51.727917  7875 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:51.727732  7856 raft_consensus.cc:2764] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 13:03:51.722599  7649 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.746726  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:51.758836  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 321254us
I20260709 13:03:51.695854  7924 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:51.716024  7876 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.719444  7494 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.721248  7884 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:51.761202  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.724016  7457 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:51.719193  7879 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:51.770891  1659 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:51.758239  7694 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:51.777001  7694 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:51.777418  7879 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.778067  7694 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:51.760926  7649 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:51.756029  7672 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:51.716215  7662 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 13:03:51.780660  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:51.838582  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:51.869167  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:51.879030  1590 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 13:03:51.888468  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.890844  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:51.892077  1660 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:51.896341  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.896651  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:51.897451  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 305543us
W20260709 13:03:51.878091  7921 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.202s	user 0.002s	sys 0.000s
I20260709 13:03:51.898362  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:51.907145  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:51.747817  7833 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:51.893882  7899 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 13:03:51.936048  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:51.774807  7876 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:51.942073  1659 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:51.944469  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:51.909031  1736 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:55272
I20260709 13:03:51.981212  1736 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:03:51.981812  1736 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:51.987766  1660 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:03:52.008275  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 371923us
W20260709 13:03:52.154230  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 566092us
W20260709 13:03:52.155738  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.009s	sys 0.020s
I20260709 13:03:52.167341  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 983) took 2297 ms. Trace:
I20260709 13:03:52.169854  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:52.176432  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:52.175256  1486 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:52.180186  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:52.182564  1590 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:52.185110  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:52.195268  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:52.012319  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:52.018297  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:52.248576  2314 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:51.938303  7407 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.002s	sys 0.031s
I20260709 13:03:52.249883  1659 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:52.250157  7407 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.832s	user 0.002s	sys 0.032s
I20260709 13:03:52.063153  7441 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.042552  7662 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:03:52.266254  1586 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:52.267932  1591 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:52.270991  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3058) took 1295 ms. Trace:
I20260709 13:03:52.272043  1660 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:52.272661  1589 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:52.274520  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2348) took 1366 ms. Trace:
I20260709 13:03:52.275494  7441 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:52.276477  7903 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:51.990903  7833 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:52.279230  1584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.003s	sys 0.006s
I20260709 13:03:52.022255  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:52.068923  7416 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.152930   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:03:52.016531  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:52.290999  2317 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:52.053802  7748 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:51.884207  7823 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.274957  1740 rpcz_store.cc:276] 0709 13:03:50.908193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.569021 (+660828us) service_pool.cc:224] Handling call
0709 13:03:51.569420 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.569666 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:51.891637 (+321971us) log.cc:844] Serialized 14 byte log entry
0709 13:03:52.273775 (+382138us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:52.273891 (+   116us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:52.274035 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.274155 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:52.274348 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11214,"prepare.queue_time_us":523238,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":13375,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":244,"thread_start_us":653817,"threads_started":2,"wal-append.queue_time_us":198878}
I20260709 13:03:52.110069  7879 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.301028  7879 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.027657  7661 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:52.070832  7876 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:52.092463  7933 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 13:03:52.071383  7941 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:52.325321  7941 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.325947  7941 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:52.197300  1662 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.016s	sys 0.024s
W20260709 13:03:52.329710  1662 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.017s	sys 0.024s
I20260709 13:03:52.334066  1593 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
W20260709 13:03:52.220275  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 319447us
I20260709 13:03:52.068977  7847 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:52.238737  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.010s	sys 0.021s
I20260709 13:03:52.232089  1585 rpcz_store.cc:276] 0709 13:03:49.870044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.903081 (+ 33037us) service_pool.cc:224] Handling call
0709 13:03:49.904201 (+  1120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:50.165689 (+261488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:50.165752 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:50.177834 (+ 12082us) log.cc:844] Serialized 26 byte log entry
0709 13:03:50.301956 (+124122us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:50.320747 (+ 18791us) log.cc:844] Serialized 14 byte log entry
0709 13:03:50.323076 (+  2329us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:50.323153 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:51.508559 (+1185406us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b4800080358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:03:51.693571 (+185012us) spinlock_profiling.cc:243] Waited 5.28 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:52.166531 (+472960us) raft_consensus.cc:1685] finished
0709 13:03:52.166624 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:52.167059 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7015,"prepare.queue_time_us":14873,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":67081,"raft.run_cpu_time_us":3655,"raft.run_wall_time_us":1187567,"spinlock_wait_cycles":405263616,"thread_start_us":341682,"threads_started":2,"wal-append.queue_time_us":301104}
I20260709 13:03:52.240744  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.260525  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1055) took 2791 ms. Trace:
I20260709 13:03:52.261250  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2346) took 1325 ms. Trace:
I20260709 13:03:52.050745  7884 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:52.357910  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.010s	sys 0.005s
W20260709 13:03:52.359197  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.358731  7884 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:52.360112  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.271651  1739 rpcz_store.cc:276] 0709 13:03:50.975150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.717493 (+742343us) service_pool.cc:224] Handling call
0709 13:03:51.717914 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:51.718316 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:51.718356 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:51.723135 (+  4779us) log.cc:844] Serialized 26 byte log entry
0709 13:03:52.025760 (+302625us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:52.025811 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.025899 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:52.231431 (+205532us) raft_consensus.cc:1685] finished
0709 13:03:52.231507 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:52.270714 (+ 39207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":300681,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":15379,"thread_start_us":263293,"threads_started":1,"wal-append.queue_time_us":356502}
W20260709 13:03:52.314074  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.321877  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.322480  2315 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: [16.12-18.13]
W20260709 13:03:52.347556  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.348682  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:52.355825  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 784997us
I20260709 13:03:52.356304  7847 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.339915  1735 rpcz_store.cc:276] 0709 13:03:50.907582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.568739 (+661157us) service_pool.cc:224] Handling call
0709 13:03:51.576750 (+  8011us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:51.580710 (+  3960us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:03:51.580742 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:51.591411 (+ 10669us) log.cc:844] Serialized 26 byte log entry
0709 13:03:52.021746 (+430335us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:52.021793 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.226677 (+204884us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:52.232831 (+  6154us) raft_consensus.cc:1685] finished
0709 13:03:52.232869 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:52.233340 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3389,"raft.queue_time_us":3502,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"thread_start_us":380801,"threads_started":1,"wal-append.queue_time_us":467652}
I20260709 13:03:52.383666  2314 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:03:52.386483  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 145985us
I20260709 13:03:52.388167  1659 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:52.391036  7876 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:52.391764  7876 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:52.392271  7876 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:52.393616  7704 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:52.339486  1742 rpcz_store.cc:276] 0709 13:03:49.469197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.548788 (+2079591us) service_pool.cc:224] Handling call
0709 13:03:51.549185 (+   397us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:51.549613 (+   428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:51.549642 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:51.550240 (+   598us) log.cc:844] Serialized 52 byte log entry
0709 13:03:51.978529 (+428289us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:51.978601 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:51.978718 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:52.228649 (+249931us) raft_consensus.cc:1685] finished
0709 13:03:52.228714 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:52.260281 (+ 31567us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1396,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9183,"thread_start_us":397519,"threads_started":1,"wal-append.queue_time_us":484014}
W20260709 13:03:52.408079  7672 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.540s	user 0.006s	sys 0.012s
I20260709 13:03:52.410315  1661 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
W20260709 13:03:52.415010  7494 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.011s
I20260709 13:03:52.419716  1737 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 13:03:52.445556  1736 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.461894  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3013) took 1417 ms. Trace:
I20260709 13:03:52.462370  1665 rpcz_store.cc:276] 0709 13:03:50.944376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.039311 (+ 94935us) service_pool.cc:224] Handling call
0709 13:03:51.051098 (+ 11787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.208821 (+157723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.360938 (+1152117us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400010c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:52.361339 (+   401us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":294187264}
W20260709 13:03:52.464457  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 143082us
I20260709 13:03:52.440893  7672 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.453040  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:52.443509  7494 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.471694  7672 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:52.471956  7946 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:52.474123  7614 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:03:52.475354  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:52.480027  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.479825  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94798.6us
I20260709 13:03:52.482265  1739 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 21. Ops: []
I20260709 13:03:52.452983  7935 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.486482  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2351) took 1575 ms. Trace:
I20260709 13:03:52.487203  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:52.488185  1736 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:52.442203  7879 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:52.446318  7649 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.578s	user 0.009s	sys 0.013s
W20260709 13:03:52.495561  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.498062  1737 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:52.501008  7845 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:03:52.502453  1659 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:52.439702  7941 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.503726  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:52.504417  7649 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.506541  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.506767  7649 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.508236  7833 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:52.510752  7833 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:52.511232  7833 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:52.477262  7880 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.452225  7823 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:52.459142  7847 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.517086  1742 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:52.521651  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:52.523058  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47883.2us
I20260709 13:03:52.525931  7935 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:52.530416  7851 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.836s	user 0.009s	sys 0.014s
I20260709 13:03:52.529968  7880 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.555182  7880 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.532029  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.558421  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.566820  7945 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 13:03:52.569461  7960 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 13:03:52.489348  7614 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:52.542371  7766 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:52.543174  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:52.491153  1738 rpcz_store.cc:276] 0709 13:03:50.910608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.570847 (+660239us) service_pool.cc:224] Handling call
0709 13:03:51.598896 (+ 28049us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:51.644155 (+ 45259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.467909 (+823754us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b54003511d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:52.486259 (+ 18350us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63752064}
I20260709 13:03:52.556082  1660 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:52.489948  7441 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 13:03:52.534291  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.574793  7836 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:52.575102  7823 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:52.585513  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.008s	sys 0.006s
W20260709 13:03:52.600867  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.009s	sys 0.006s
I20260709 13:03:52.580699  7614 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:52.605101  7968 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.060s	user 0.000s	sys 0.002s
I20260709 13:03:52.580806  7847 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.608994  7847 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.606751  7614 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:03:52.613162  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 160227us
I20260709 13:03:52.627504  1664 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:03:52.627795  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:52.628651  1587 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:52.632764  7910 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:52.633587  7910 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:52.634553  7494 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.634729  7910 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.637799  7694 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.637708  7952 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 13:03:52.638850  1585 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:52.638713  7694 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.646708  7873 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:52.647238  7873 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:52.648478  7873 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.629134  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:52.667043  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 185073us
I20260709 13:03:52.669951  1664 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:52.674826  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1047) took 3366 ms. Trace:
I20260709 13:03:52.675303  1733 rpcz_store.cc:276] 0709 13:03:49.299231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:50.867030 (+1567799us) service_pool.cc:224] Handling call
0709 13:03:50.991337 (+124307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:50.991933 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:52.662534 (+1670601us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b540012d9d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:52.666024 (+  3490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285005056}
I20260709 13:03:52.676762  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.645301  7461 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:52.582492  7851 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.639001  7952 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
W20260709 13:03:52.679358  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3015) took 3473 ms (client timeout 3234 ms). Trace:
I20260709 13:03:52.693217  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:52.686445  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 81998.9us
I20260709 13:03:52.681262  7441 raft_consensus.cc:2764] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 13:03:52.693442  1734 rpcz_store.cc:269] 0709 13:03:49.206146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.211546 (+  5400us) service_pool.cc:224] Handling call
0709 13:03:52.678725 (+3467179us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b540012e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:52.679094 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4527414144}
I20260709 13:03:52.673710  7847 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.693295  7952 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:52.709363  1667 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:52.701628  7748 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:52.710224  1590 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:52.701758  7882 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:52.711038  7941 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:52.706645  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2313) took 1138 ms. Trace:
I20260709 13:03:52.709240  1668 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 17 because replica has last-logged OpId of term: 16 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 13:03:52.705406  1664 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:52.512455  7833 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.683962  7945 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:03:52.721920  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45382.2us
I20260709 13:03:52.708299  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.716109  1661 rpcz_store.cc:276] 0709 13:03:51.568245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.761926 (+193681us) service_pool.cc:224] Handling call
0709 13:03:52.706285 (+944359us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400118854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:52.706535 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1240835968}
W20260709 13:03:52.719249  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 297840us
I20260709 13:03:52.726791  1740 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:03:52.709208  7823 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:52.733146  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 267606us
I20260709 13:03:52.674247  7876 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.738575  7882 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.714730  7748 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:52.739861  7947 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:52.740020  2280 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.648s (SENT)
I20260709 13:03:52.708760  7855 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 13:03:52.714829  7845 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.708361  7935 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.779803  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.785895  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:52.787165  7876 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:52.787047  7748 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.788113  7823 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.747704  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.785099  7882 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:52.742400  7945 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:52.830070  7935 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:52.830703  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:52.831043  7935 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.836793  7416 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:52.838820  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 122555us
I20260709 13:03:52.842149  7672 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:52.843256  7672 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:52.843472  7952 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.6, Last appended by leader: 6, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.845742  7946 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:52.843852  7903 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:52.846342  7946 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:52.846621  7890 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.532s	user 0.004s	sys 0.009s
I20260709 13:03:52.846849  7903 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:52.843248  1659 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:52.848660  7880 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:52.849629  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 109620us
I20260709 13:03:52.849334  7880 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:52.850565  2280 leader_election.cc:336] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.559s (SENT)
I20260709 13:03:52.851161  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.852398  7872 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:52.825829  7823 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.853233  7872 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.854136  7872 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.853178  7860 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:52.827818  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:52.864283  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.873930  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.741663  7847 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.845249  7952 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:52.887969  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 287382us
W20260709 13:03:52.890808  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:52.847798  7946 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.855801  7823 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:52.903693  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 124633us
I20260709 13:03:52.907312  7416 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:52.907428  7480 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:52.925158  7976 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 13:03:52.925254  7480 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:52.926148  7480 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:52.927824  7416 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:52.929155  7416 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.928375  7480 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.932490  7480 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:52.936058  7872 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:52.936961  7945 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.912272  7845 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:52.940201  7845 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:52.941396  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 141402us
I20260709 13:03:52.942641  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:52.944504  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:52.944916  7847 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:52.951932  7875 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.144s	user 0.008s	sys 0.022s
I20260709 13:03:52.952636  7875 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.954239  7935 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.957068  7845 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:52.960023  7845 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:52.801883  7947 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:52.912628  7823 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:52.920020  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:52.899485  7941 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 13:03:52.901232  7880 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.961047  1586 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:03:52.964066  7851 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:52.964751  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:52.965536  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.897217  7977 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.060s	user 0.001s	sys 0.003s
W20260709 13:03:52.977968  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:52.980590  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67306.2us
I20260709 13:03:52.999233  7741 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:03:53.000241  7741 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:53.000887  7741 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:52.975625  7946 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:53.003197  1587 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:52.978797  7823 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.007033  1590 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:53.008142  7823 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:53.013648  1665 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.024s	sys 0.072s
W20260709 13:03:53.014128  1665 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.024s	sys 0.072s
W20260709 13:03:53.022410  1735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.083s	sys 0.170s
W20260709 13:03:53.036396  1735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.083s	sys 0.170s
I20260709 13:03:53.037330  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1061) took 1394 ms. Trace:
I20260709 13:03:53.038240  1735 rpcz_store.cc:276] 0709 13:03:51.642866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.386123 (+743257us) service_pool.cc:224] Handling call
0709 13:03:52.401595 (+ 15472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:52.514349 (+112754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:52.514402 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:52.515280 (+   878us) log.cc:844] Serialized 26 byte log entry
0709 13:03:53.036907 (+521627us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:53.036960 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.037037 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:53.037084 (+    47us) raft_consensus.cc:1685] finished
0709 13:03:53.037097 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:53.037270 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78097,"raft.queue_time_us":88152,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":5595,"thread_start_us":521051,"threads_started":1,"wal-append.queue_time_us":517351}
W20260709 13:03:53.041826  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 333635us
W20260709 13:03:53.042229  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97872.3us
I20260709 13:03:53.043251  7836 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:53.057422  7875 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.064030  7935 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:53.076953  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:53.086751  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59860.5us
I20260709 13:03:53.058602  7903 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:53.091851  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:53.092797  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:53.098354  7974 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.316s	user 0.001s	sys 0.000s
I20260709 13:03:53.099153  7947 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.101696  7631 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:53.102335  7631 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:53.081457  7649 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:53.102849  7631 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:03:53.092427  1557 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.592s (SENT)
I20260709 13:03:53.121299  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.153981  7823 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:53.155367  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81465.9us
I20260709 13:03:53.159849  7903 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:53.165565  1486 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 11, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 11 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.174718  7979 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:03:53.175354  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:53.176244  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107770us
I20260709 13:03:53.177560  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.117147  7860 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 13:03:53.180440  7860 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:53.180892  7860 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:53.190146  7979 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:53.193111  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1059) took 1554 ms. Trace:
I20260709 13:03:53.193796  7836 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.193872  1740 rpcz_store.cc:276] 0709 13:03:51.638884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.296321 (+657437us) service_pool.cc:224] Handling call
0709 13:03:52.297016 (+   695us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:52.728187 (+431171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.192596 (+464409us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:53.192996 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":987115776}
I20260709 13:03:53.202375  1486 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.204523  1486 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 14 to 20, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 20 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.206851  1486 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 2 to 14. New cstate: current_term: 14 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.212467  7881 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:03:53.213131  7881 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:53.213397  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.215015  7860 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.193636  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.216557  7823 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:53.216691  7860 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:53.218568  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:53.271809  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.274140  7881 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.274351  1486 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 5 to 12, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 12 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:53.276046  7881 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:53.276659  1486 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:53.278757  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 535537us
I20260709 13:03:53.279141  7741 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.280118  1486 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.282384  1486 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.284298  1486 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:03:53.285591  7823 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:53.289721  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.291177  7741 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.291289  7833 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:03:53.294376  7833 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:53.295567  7833 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:53.297058  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 242557us
I20260709 13:03:53.297973  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1110) took 1723 ms. Trace:
I20260709 13:03:53.298382  2315 rpcz_store.cc:276] 0709 13:03:51.574690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:51.574853 (+   163us) service_pool.cc:224] Handling call
0709 13:03:51.575330 (+   477us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:52.380322 (+804992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.297569 (+917247us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b54005ae454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:53.297879 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1715227904}
W20260709 13:03:53.291663  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60510.6us
W20260709 13:03:53.278245  7972 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.449s	user 0.003s	sys 0.000s
W20260709 13:03:53.293396  7967 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.461s	user 0.000s	sys 0.002s
W20260709 13:03:53.292757  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74283.9us
W20260709 13:03:53.316867  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 208311us
I20260709 13:03:53.318372  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:53.315757  7480 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:53.320226  1556 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
W20260709 13:03:53.323757  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.324154  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:53.325181  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:53.326967  1556 leader_election.cc:403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:03:53.328979  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.330896  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:03:53.334892  7989 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.061s	user 0.001s	sys 0.011s
I20260709 13:03:53.342707  7965 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:53.343429  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:53.343673  7965 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:53.312798  7631 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.345163  7965 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.326375  7945 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:53.347687  7631 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.332991  7480 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:53.349234  7988 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.208s	user 0.001s	sys 0.000s
I20260709 13:03:53.350402  7480 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.351749  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:53.368014  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 157490us
I20260709 13:03:53.372479  7992 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 13:03:53.373759  7748 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:53.377885  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.378621  1659 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:53.378700  1660 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:53.380292  7457 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:53.380743  7441 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:53.381775  7945 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:53.382231  7945 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:53.382623  7945 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.391188  2317 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22. Ops: []
I20260709 13:03:53.380767  7457 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:53.395138  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:53.396234  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.396535  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:53.396739  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.398517  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:53.399948  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.400377  7993 raft_consensus.cc:2764] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:03:53.402354  7457 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.404405  7457 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:53.386631  7836 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:03:53.381389  7441 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.407212  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1065) took 1086 ms. Trace:
I20260709 13:03:53.407579  1742 rpcz_store.cc:276] 0709 13:03:52.321170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.494411 (+173241us) service_pool.cc:224] Handling call
0709 13:03:52.494872 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:52.535191 (+ 40319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.406954 (+871763us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b5400350cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:53.407153 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50355584}
W20260709 13:03:53.394563  7949 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.004s	sys 0.009s
W20260709 13:03:53.418627  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.406697  7992 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:03:53.427050  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:53.436333  7461 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.016s
W20260709 13:03:53.436884  7461 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.000s	sys 0.016s
I20260709 13:03:53.386611  7979 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:53.444448  7979 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:53.446084  7982 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 13:03:53.447283  7979 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:53.433924  7993 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:53.449002  7741 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.447419  2317 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:03:53.432860  7949 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.005s	sys 0.009s
W20260709 13:03:53.435067  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 321660us
I20260709 13:03:53.411643  7876 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:53.467463  7876 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:53.467978  7876 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:53.433519  7879 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:53.471464  7879 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:53.450313  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 104829us
I20260709 13:03:53.453176  7660 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:53.453886  1661 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:53.457769  7952 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:53.458874  7661 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:53.459553  2317 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:53.460793  7922 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.460147  7741 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.461457  7836 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:03:53.463405  7903 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:53.465560  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:53.465828  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:53.478092  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95578.3us
I20260709 13:03:53.491179  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.484730  7661 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.491001  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.496264  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.496534  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.497043  1735 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:53.497300  1734 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.491766  7993 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:53.497684  7640 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.488485  7922 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.499166  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.500385  7922 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.496280  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.502203  1741 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:53.507735  1659 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:03:53.489694  7987 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17.
I20260709 13:03:53.512424  1740 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:53.512835  7935 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:53.513411  7935 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:53.514185  7741 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:53.516244  7983 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.342s	user 0.000s	sys 0.021s
I20260709 13:03:53.491636  7660 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.520105  7660 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.520859  7879 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:53.506263  7993 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.522061  7993 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.523020  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.523743  1742 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:53.524438  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.525148  1733 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:53.525830  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.526345  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.527376  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:53.527698  8003 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 13:03:53.528080  1739 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:53.487511  7945 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:53.529193  7984 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.317s	user 0.002s	sys 0.016s
I20260709 13:03:53.529220  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.530061  1740 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:53.526726  1737 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 13:03:53.530589  7945 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.531725  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3119) took 1368 ms. Trace:
I20260709 13:03:53.532119  1586 rpcz_store.cc:276] 0709 13:03:52.163426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.267202 (+103776us) service_pool.cc:224] Handling call
0709 13:03:52.267799 (+   597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:52.986729 (+718930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.531451 (+544722us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b54003b1ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:53.531638 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1592873216}
I20260709 13:03:53.542546  1661 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 7.
I20260709 13:03:53.543267  1742 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:53.548530  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:53.557121  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.557749  1741 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.553463  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.560813  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.562263  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:53.564702  7985 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:53.566936  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.567344  1738 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 13:03:53.567798  1738 pending_rounds.cc:85] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:03:53.524479  7993 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.561304  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:53.570815  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.563284  1736 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:53.574548  1739 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:53.581704  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.582743  1659 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:53.552079  7946 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.586120  1735 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:53.568058  1738 pending_rounds.cc:107] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 13:03:53.586787  1738 raft_consensus.cc:2889] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:03:53.588824  7997 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 13:03:53.584867  7879 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:53.581189  7979 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 13:03:53.594934  7946 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.597657  7614 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:53.539526  7952 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:53.598912  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 274719us
I20260709 13:03:53.599798  7614 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:53.600057  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:53.600452  7952 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:53.605491  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 272143us
I20260709 13:03:53.605804  7441 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.607856  7582 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:53.608556  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:53.609422  7855 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 13:03:53.597637  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71428.5us
I20260709 13:03:53.613683  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1036) took 1138 ms. Trace:
W20260709 13:03:53.613862  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52484.6us
I20260709 13:03:53.614295  1667 rpcz_store.cc:276] 0709 13:03:52.475807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.482104 (+  6297us) service_pool.cc:224] Handling call
0709 13:03:52.709026 (+226922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:52.712770 (+  3744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.613346 (+900576us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400118850. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:53.613600 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":433738624}
W20260709 13:03:53.616434  7966 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.089s	user 0.002s	sys 0.000s
I20260709 13:03:53.616979  7903 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:03:53.617691  7903 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:53.618920  7903 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.601913  7962 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:03:53.591558  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3001) took 3704 ms (client timeout 2232 ms). Trace:
W20260709 13:03:53.623126  1668 rpcz_store.cc:269] 0709 13:03:49.888613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:49.888758 (+   145us) service_pool.cc:224] Handling call
0709 13:03:53.591204 (+3702446us) spinlock_profiling.cc:243] Waited 2.74 s on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:53.591439 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6313122688}
I20260709 13:03:53.623697  7935 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.626654  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:53.597317  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1033) took 1123 ms. Trace:
I20260709 13:03:53.637785  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:53.638463  7996 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 13:03:53.638491  7855 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:53.628192  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.630566  7952 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:53.616207  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.630663  7582 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:03:53.644896  1734 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:53.655979  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:53.646482  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:53.639955  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55967.1us
I20260709 13:03:53.668599  1587 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:53.670038  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:53.677331  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:53.645591  1665 rpcz_store.cc:276] 0709 13:03:52.474221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.481570 (+  7349us) service_pool.cc:224] Handling call
0709 13:03:52.486649 (+  5079us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:52.487091 (+   442us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:03:53.022175 (+535084us) log.cc:844] Serialized 14 byte log entry
0709 13:03:53.316450 (+294275us) log.cc:844] Serialized 14 byte log entry
0709 13:03:53.448712 (+132262us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b18001010e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:53.596616 (+147904us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b18001010e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:03:53.596839 (+   223us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:53.596910 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:03:53.596974 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.597081 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:53.597240 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":510206,"prepare.queue_time_us":19154,"prepare.run_cpu_time_us":104,"prepare.run_wall_time_us":105,"raft.queue_time_us":508591,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":8316,"spinlock_wait_cycles":12235136,"thread_start_us":557571,"threads_started":2,"wal-append.queue_time_us":227898}
I20260709 13:03:53.680665  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:53.678596  2310 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:53.597484  7946 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:53.689342  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.693889  2312 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:03:53.694383  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.695175  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:53.696954  1735 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:53.705052  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:53.705900  1588 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:53.708644  1586 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:53.710675  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:53.710703  1585 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:53.654115  7855 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:53.653725  7962 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:53.712139  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.713075  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.688551  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:53.693028  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:53.719288  1584 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:53.719027  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:53.720163  1592 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:53.720513  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.720817  1590 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:53.719126  2314 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:03:53.721830  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.666671  7407 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 13:03:53.723223  7694 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.723409  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:53.724634  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.725252  7856 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:53.725924  1632 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.522s (SENT)
W20260709 13:03:53.727049  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0585165759999882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
I20260709 13:03:53.731946  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.715782  7995 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 13:03:53.736454  7995 consensus_queue.cc:1243] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:03:53.737113  7995 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:53.737416  7995 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:53.738615  7995 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:53.739662  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.337911   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:53.748692  2315 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:53.753894  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.754931  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:53.763566  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 137335us
I20260709 13:03:53.711011  7660 raft_consensus.cc:2764] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:03:53.764853  1589 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:03:53.766183  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.767683  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:53.753316  7990 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.595s	user 0.005s	sys 0.006s
I20260709 13:03:53.768445  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:03:53.772356  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.773080  1735 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:53.774406  1735 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.744966  7748 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:53.753335  7661 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.775911  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.778326  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:53.767239  7995 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:53.791519  1739 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:53.766458  7847 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:53.788589  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60647.6us
I20260709 13:03:53.796126  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.766963  7935 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:53.796355  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.795647  1733 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:53.777980  7965 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:53.781973  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 286103us
I20260709 13:03:53.781957  7441 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:53.839495  1734 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:53.788137  7856 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:53.887598  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:53.895867  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57313.3us
I20260709 13:03:53.898733  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.899224  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.799713  7582 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:53.921880  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:53.922848  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:53.924732  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:53.896018  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:53.938460  1668 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:03:53.986922  1735 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:03:53.797734  7993 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.104s
I20260709 13:03:53.887180  7640 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:03:53.967326  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45501.8us
W20260709 13:03:53.993871  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 197490us
I20260709 13:03:54.043471  1584 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:54.044488  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:54.015091  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:54.046834  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.084728  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:53.930393  7880 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:53.999590  7640 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:54.097654  1592 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 13:03:54.098987  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:54.113021  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:53.809954  7847 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.119554  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:53.988397  7461 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:53.896914  7748 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.837729  7965 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.161531  1659 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:54.139628  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.145567  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.145306  1589 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 12.
I20260709 13:03:54.086881  7631 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:03:54.160645  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.101480  7880 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:53.930121  7876 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.170827  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.172659  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:54.176322  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 289843us
I20260709 13:03:54.179843  2315 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
W20260709 13:03:54.176553  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 280456us
I20260709 13:03:53.886080  7855 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:53.986948  7441 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.121809  7906 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:53.990464  7582 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.123023  7952 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:54.200445  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.178193  7880 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:54.224912  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.218981  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.219103  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.223592  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.202610  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:54.093708  7430 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.305s
W20260709 13:03:54.231518  8007 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.334s	user 0.000s	sys 0.001s
W20260709 13:03:54.217010  7457 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.812s	user 0.007s	sys 0.007s
I20260709 13:03:54.231429  7661 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.001927  7962 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.232069  7640 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.232978  7912 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:54.233492  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1044) took 1336 ms. Trace:
W20260709 13:03:54.233934  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 154422us
I20260709 13:03:54.235033  7922 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:03:54.246618  8011 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:03:54.245127  7965 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.249284  7441 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.250106  7952 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.195394  7992 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.308202  1659 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:03:54.252177  7985 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.687s	user 0.010s	sys 0.000s
W20260709 13:03:54.253665  1741 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.684s	user 0.010s	sys 0.003s
I20260709 13:03:54.254246  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:54.287461  7947 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:54.276262  1660 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:03:54.292932  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:54.299479  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116844us
I20260709 13:03:54.290875  1663 rpcz_store.cc:276] 0709 13:03:52.650240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:52.687290 (+ 37050us) service_pool.cc:224] Handling call
0709 13:03:52.724554 (+ 37264us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:53.038669 (+314115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:53.038718 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:53.060068 (+ 21350us) log.cc:844] Serialized 26 byte log entry
0709 13:03:53.138304 (+ 78236us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:03:53.138346 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:53.138425 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:53.729297 (+590872us) spinlock_profiling.cc:243] Waited 967 us on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:53.931399 (+202102us) raft_consensus.cc:1685] finished
0709 13:03:53.931423 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:53.986722 (+ 55299us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":455836,"raft.queue_time_us":242725,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":19807,"spinlock_wait_cycles":2224896,"thread_start_us":43247,"threads_started":1,"wal-append.queue_time_us":74234}
I20260709 13:03:54.292145  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:54.303962  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:54.307502  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.311314  1741 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:54.299175  7922 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:54.289309  7640 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:54.290822  7912 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.291446  7855 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:54.321843  1668 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:54.326956  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.328563  8019 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:54.332788  7922 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.334869  7922 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:54.339002  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:54.342499  1584 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:03:54.346192  7847 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.347908  7847 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:54.351380  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:54.352363  1592 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:03:54.354301  8013 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 13:03:54.356168  7876 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:54.356593  1662 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:03:54.357986  1662 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:54.320194  7985 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:54.327119  7912 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:54.367738  7952 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:54.369325  7748 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.313038  7992 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.372749  7992 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:03:54.376194  1663 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:54.291965  7906 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:54.382131  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43479.9us
I20260709 13:03:54.383308  8005 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:54.256068  7856 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:54.391197  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 137002us
I20260709 13:03:54.323351  7430 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:54.396502  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:54.404508  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.410586  1584 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:54.373433  7992 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:54.412209  1588 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.671s	user 0.012s	sys 0.001s
W20260709 13:03:54.413046  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.415221  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:54.420111  1588 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:03:54.424626  8015 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.191s	user 0.001s	sys 0.001s
W20260709 13:03:54.420799  7995 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.620s	user 0.011s	sys 0.002s
I20260709 13:03:54.424180  7480 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 13:03:54.459098  1739 pending_rounds.cc:85] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:03:54.479883  1739 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 13:03:54.480192  1739 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:54.480759  1739 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 13:03:54.481016  1739 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:54.419059  7631 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:54.495564  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:54.423400  7741 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:03:54.514642  7860 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:54.515236  7860 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:54.516435  7860 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.439337  7582 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.415376  8018 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:54.430609  7992 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:54.529268  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118765us
I20260709 13:03:54.530689  7860 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:54.422756  8019 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:03:54.451763  8025 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:54.424612  8009 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 13:03:54.538120  1668 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:54.547648  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2323) took 1075 ms. Trace:
I20260709 13:03:54.559247  1668 rpcz_store.cc:276] 0709 13:03:53.472444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.273479 (+801035us) service_pool.cc:224] Handling call
0709 13:03:54.547561 (+274082us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:03:54.558353  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:54.529907  7836 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:03:54.545085  7614 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.945s	user 0.011s	sys 0.004s
I20260709 13:03:54.543949  7939 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.539896  7672 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:54.546164  7952 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.540283  7660 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:54.578402  7457 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:03:54.575608  8022 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:54.577373  7614 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:54.581064  8023 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.583376  7875 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.581657  7836 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:54.598992  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52240.7us
I20260709 13:03:54.622999  7992 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.602708  8023 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.602231  7631 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.600669  7660 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.627898  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:54.628235  7660 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.629019  7962 raft_consensus.cc:2764] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:03:54.585551  7952 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.631178  8005 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:54.635299  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:54.636579  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:03:54.638697  7912 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.641304  8005 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:54.642786  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:54.644522  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:54.648166  8009 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:03:54.648710  8009 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:54.649084  8009 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:54.662292  7836 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.666317  2317 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22. Ops: []
I20260709 13:03:54.668146  8008 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:54.668704  8008 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.669140  8008 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:54.670305  8027 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:03:54.671271  7828 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:54.671864  7828 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:54.675886  1664 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:54.580549  7939 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.677599  1668 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:54.678722  1667 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:54.679195  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:54.638131  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.679737  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:54.680056  8010 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
W20260709 13:03:54.680657  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.681602  1660 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 13:03:54.682072  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54265.9us
I20260709 13:03:54.672806  7828 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.679569  7939 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.679374  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1051) took 1034 ms. Trace:
W20260709 13:03:54.637949  7999 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.153s	user 0.017s	sys 0.014s
I20260709 13:03:54.681190  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.699118  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:54.699273  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.595846  8022 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:54.702591  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.703336  2317 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:54.703338  1589 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:54.699721  1667 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:54.545821  8018 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.695477  1664 rpcz_store.cc:276] 0709 13:03:53.644703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.712751 (+ 68048us) service_pool.cc:224] Handling call
0709 13:03:54.679187 (+966436us) spinlock_profiling.cc:243] Waited 961 ms on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:54.679298 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2211305856}
I20260709 13:03:54.706952  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:54.710160  1585 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:03:54.690421  7480 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:54.690361  7833 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.718991  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.721710  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:54.712186  8028 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 13:03:54.611515  7875 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.706629  8022 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:54.712055  7828 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:54.718111  7480 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:54.721959  7614 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:54.715165  7939 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 13:03:54.708669  7979 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 13:03:54.748842  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.706786  7660 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
W20260709 13:03:54.740310  7872 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.338s	user 0.008s	sys 0.021s
I20260709 13:03:54.731053  7952 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.765080  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:54.771602  7872 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.372s	user 0.009s	sys 0.022s
I20260709 13:03:54.722675  7833 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:54.785090  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94357.5us
I20260709 13:03:54.732374  7880 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.792016  1660 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:03:54.786863  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:54.793973  1667 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:54.796913  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:54.748837  7935 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 13:03:54.779023  7614 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:54.783540  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54671.5us
W20260709 13:03:54.804355  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:54.801231  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 699786us
I20260709 13:03:54.810745  1661 raft_consensus.cc:2368] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
I20260709 13:03:54.807404  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:54.836787  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:54.837810  1667 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:03:54.801447  8023 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.782780  8032 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:03:54.814622  8019 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:03:54.841625  7946 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:03:54.856838  7875 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.858410  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:54.858963  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62113.1us
I20260709 13:03:54.853694  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:54.848773  7912 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.847792  8041 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:54.858009  7946 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:54.870187  8044 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:54.870698  7946 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.871939  7946 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:54.874719  8018 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:54.876554  1737 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:03:54.877005  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:54.877702  8023 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:54.877803  8034 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 13:03:54.878800  8034 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:54.879184  8034 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:54.879387  8023 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:54.880131  8034 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.878100  7741 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:03:54.881721  7741 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:54.881965  7947 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:54.882669  7741 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.884714  7947 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:03:54.886163  8023 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:03:54.886897  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:54.886761  8023 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:54.889227  8023 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.890297  8023 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:03:54.898746  8035 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:03:54.845386  7480 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.903230  7935 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:03:54.907029  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53484.7us
W20260709 13:03:54.908736  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58892us
W20260709 13:03:54.920641  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53782us
W20260709 13:03:54.923600  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 60888.1us
I20260709 13:03:54.919148  7875 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.930083  7875 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.931967  7833 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:54.932430  7875 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:54.919160  7935 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:54.933461  7935 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:54.934775  7935 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 15.7, Last appended by leader: 7, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.925154  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:54.938830  1486 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 12, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 12 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:03:54.865798  7631 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:03:54.954741  7631 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:54.955885  7631 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:54.922132  8032 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:54.958731  7995 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:54.919648  7704 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 13:03:54.947954  1486 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:54.949303  8033 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 13:03:54.866400  7614 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:54.969785  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44731.2us
I20260709 13:03:54.939458  7649 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:03:54.948508  8036 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 13:03:54.960263  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:54.963925  1665 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:03:54.991494  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2321) took 1519 ms (client timeout 1908 ms). Trace:
W20260709 13:03:54.991915  1665 rpcz_store.cc:269] 0709 13:03:53.471662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.716697 (+245035us) service_pool.cc:224] Handling call
0709 13:03:54.991259 (+1274562us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:54.991399 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317893888}
I20260709 13:03:54.852828  8031 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:54.993459  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3042) took 1191 ms. Trace:
I20260709 13:03:54.941823  7912 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 13:03:54.965090  7995 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:54.993938  1663 rpcz_store.cc:276] 0709 13:03:53.802292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.407432 (+605140us) service_pool.cc:224] Handling call
0709 13:03:54.418311 (+ 10879us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:54.432775 (+ 14464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:54.432826 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:54.482011 (+ 49185us) log.cc:844] Serialized 26 byte log entry
0709 13:03:54.696304 (+214293us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:54.696347 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.696418 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:54.993211 (+296793us) raft_consensus.cc:1685] finished
0709 13:03:54.993234 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:54.993401 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11689,"raft.queue_time_us":10471,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":1165,"thread_start_us":148628,"threads_started":1,"wal-append.queue_time_us":240242}
I20260709 13:03:54.996595  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.996302  7995 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:54.981134  8019 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:54.999303  8019 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:54.999754  8019 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:54.984754  1585 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:54.923373  8034 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:54.993743  8031 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.004318  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:54.958931  8032 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.004880  8031 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:54.958892  7880 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.027603  1593 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:54.993136  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:54.986407  7672 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:55.003654  7875 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:55.004078  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60455.8us
W20260709 13:03:55.034754  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63995.6us
I20260709 13:03:55.035573  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:54.994989  7416 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:54.976900  7649 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:55.010113  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:55.015553  7945 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:55.015656  8009 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.016325  7906 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.016801  7995 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.023105  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.026010  1588 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 13:03:55.026319  8032 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.026262  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:54.972535  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.037307  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:55.061141  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:55.037120  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:55.068179  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.068605  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:55.064011  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:55.065821  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 93361.2us
I20260709 13:03:55.052177  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.034358  7672 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:55.058494  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:55.081466  1584 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:55.084486  2317 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:55.088238  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:54.988409  8022 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.049733  7416 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:55.053134  7985 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:55.078069  7880 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:54.971017  8008 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.108978  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:55.078433  7979 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:55.090250  8048 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.089345  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:55.113276  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.118892  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.119753  1741 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 13:03:55.120118  1667 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:55.101564  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125622us
I20260709 13:03:55.120757  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.114774  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.115622  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:55.109809  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48759.5us
I20260709 13:03:55.116407  2317 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:55.125020  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.116765  1590 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:03:55.085287  7906 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:55.077931  7748 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:55.124082  1593 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:03:55.127789  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.129144  1741 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:55.130532  1588 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.130782  1734 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:55.130563  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.129932  7649 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.134286  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.134516  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.134953  7416 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.136575  7416 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.137322  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.139241  1740 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:03:55.118278  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:55.099117  7881 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:55.051621  7875 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.144050  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.134905  1738 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.107316  7962 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:55.121768  7979 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:55.094172  7631 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 13:03:55.158520  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.119033  7945 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:55.165966  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.131968  7906 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.131311  7748 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:55.145517  7881 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:55.167098  7979 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.180963  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.206652  1591 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.198756  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.191910  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.197014  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.144574  7856 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.148947  7985 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:55.178604  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:55.115779  7939 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 13:03:55.153203  7962 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.247145  1588 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
W20260709 13:03:55.275607  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 95590.4us
I20260709 13:03:55.282070  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.179317  8032 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:55.171346  7946 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:55.181835  7748 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.189976  7881 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:55.172089  7649 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:55.191231  8022 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.204953  7855 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:55.177635  7945 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.224879  7906 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.193171  7873 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:55.246618  7631 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:55.309855  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:55.321743  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.322139  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.273835  7985 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:55.143361  8048 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.291034  7962 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:55.183272  7869 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.214s
W20260709 13:03:55.301255  8013 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
W20260709 13:03:55.349789  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.351168  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.303843  8051 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 13:03:55.366366  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 187784us
I20260709 13:03:55.349411  8032 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:55.350389  8018 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:55.295931  8031 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.391170  1593 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:55.351884  7945 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.399665  2314 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:55.402590  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:55.404428  8049 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:03:55.406409  8031 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.409327  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.360674  7985 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:55.366050  7631 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:03:55.428647  1557 connection.cc:256] RPC call timeout handler was delayed by -2.2398070009999742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260709 13:03:55.416651  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 298318us
I20260709 13:03:55.418385  1587 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:55.299214  7856 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.421895  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:55.421948  1585 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:55.433727  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.434540  1737 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:03:55.431656  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.432605  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:55.435902  1663 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:55.398600  7694 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:03:55.442171  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.376514  8051 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:55.357185  7855 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.450841  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:55.420619  8032 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:55.456017  1557 connection.cc:256] RPC call timeout handler was delayed by -1.3558024289999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
I20260709 13:03:55.463158  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.428937  8009 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.467898  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:55.468978  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:55.473366  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.474623  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.468860  8031 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:55.479400  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.480307  1593 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:03:55.479949  8031 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.482601  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.484692  8031 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.485987  7631 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.490134  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1008) took 1503 ms. Trace:
I20260709 13:03:55.490307  8051 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.491529  8051 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:55.490603  1586 rpcz_store.cc:276] 0709 13:03:53.986637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:53.987249 (+   612us) service_pool.cc:224] Handling call
0709 13:03:54.085710 (+ 98461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:54.752018 (+666308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:54.752075 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:54.768517 (+ 16442us) log.cc:844] Serialized 26 byte log entry
0709 13:03:55.049953 (+281436us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:55.049997 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.050085 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.489760 (+439675us) raft_consensus.cc:1685] finished
0709 13:03:55.489785 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.490059 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":452717,"raft.queue_time_us":454430,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":7195,"thread_start_us":451929,"threads_started":2,"wal-append.queue_time_us":361099}
I20260709 13:03:55.493170  7880 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:55.495289  1586 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:03:55.496611  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.501039  1584 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 7.
I20260709 13:03:55.501917  8022 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:03:55.502976  8032 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.474987  8008 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.507831  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:55.509158  8023 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.005s
I20260709 13:03:55.511538  7855 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:03:55.512975  7828 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.765s	user 0.009s	sys 0.002s
I20260709 13:03:55.441987  7694 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:55.521878  7748 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.671710  7694 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.672905  7748 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.522662  7952 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.673832  7952 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.635999  7946 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:55.676076  7952 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.676775  1593 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:55.523490  1589 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:03:55.678882  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.680091  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.451917  7912 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:55.681672  1662 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:03:55.682303  7912 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.682816  7912 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.685055  7912 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.422272  8018 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.687232  8018 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.534732  7480 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:55.538645  7856 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.542487  8048 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.547053  7939 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.692662  7939 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.695138  7939 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.523499  7875 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.554412  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:55.477303  8019 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.696185  1668 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:03:55.697461  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:55.697854  8019 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.698979  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:55.526216  7741 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:03:55.563817  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3038) took 1870 ms (client timeout 2232 ms). Trace:
I20260709 13:03:55.564457  1660 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:55.568325  7992 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.573645  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1052) took 1928 ms. Trace:
I20260709 13:03:55.574671  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.703562  1741 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:55.703734  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.579943  7640 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:55.582643  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.584370  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:55.586581  7661 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:03:55.593595  7935 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.711967  7935 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.656927  7965 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 13:03:55.714223  7935 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:55.593914  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136108us
W20260709 13:03:55.595777  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 113220us
I20260709 13:03:55.681226  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.717116  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:55.596359  8031 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:55.597014  8067 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.720412  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:55.720563  8067 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:55.598569  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:55.603176  7672 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.505s	user 0.009s	sys 0.000s
I20260709 13:03:55.722383  7672 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.723201  8067 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.724736  7672 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.606844  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2328) took 2106 ms. Trace:
I20260709 13:03:55.707703  7912 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:55.725476  1659 rpcz_store.cc:276] 0709 13:03:53.500140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.308993 (+808853us) service_pool.cc:224] Handling call
0709 13:03:54.309329 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:54.378181 (+ 68852us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:54.378560 (+   379us) log.cc:844] Serialized 14 byte log entry
0709 13:03:54.494236 (+115676us) log.cc:844] Serialized 14 byte log entry
0709 13:03:54.535556 (+ 41320us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:54.535607 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:54.535645 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:54.675253 (+139608us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b1800462160. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:03:55.606590 (+931337us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x7b540030af54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:55.606644 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.606772 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78580,"prepare.queue_time_us":2291,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":57546,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":3093,"spinlock_wait_cycles":180782080,"thread_start_us":103779,"threads_started":1,"wal-append.queue_time_us":189736}
I20260709 13:03:55.700904  7833 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.728575  7833 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.609930  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:55.730526  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.731988  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.732411  7833 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.732607  1587 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:55.614140  1738 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:55.715296  7661 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:55.734443  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.735225  1738 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:55.735550  7661 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.736559  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.737105  1589 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:55.650341  7949 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.779933  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:55.567530  7979 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:55.638468  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65456.6us
I20260709 13:03:55.639076  1734 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:55.642642  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.643684  1591 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:55.613891  1556 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.908s (SENT) [suppressed 16 similar messages]
I20260709 13:03:55.659828  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:55.676141  7946 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.562362  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:55.700343  1661 rpcz_store.cc:269] 0709 13:03:53.693695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.180820 (+487125us) service_pool.cc:224] Handling call
0709 13:03:55.563572 (+1382752us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:55.563738 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1334250368}
W20260709 13:03:55.701773  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86128.7us
I20260709 13:03:55.708191  7962 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.707957  7992 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.703313  1666 rpcz_store.cc:276] 0709 13:03:53.645061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.393285 (+748224us) service_pool.cc:224] Handling call
0709 13:03:54.393602 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:54.413712 (+ 20110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:54.413754 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:54.413806 (+    52us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:54.413838 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.573220 (+1159382us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:55.573282 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.573425 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45752064}
I20260709 13:03:55.650643  7869 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:55.672977  7694 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:55.746255  1740 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:55.747117  1587 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:55.745334  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.809443  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:55.809923  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 167282us
W20260709 13:03:55.614738  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:55.782860  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62495.5us
I20260709 13:03:55.737342  7661 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:55.697221  7985 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.802886  1667 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:55.826674  1733 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:03:55.803282  1591 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:55.811784  1741 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:03:55.789270  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:55.812667  1589 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 13:03:55.709724  7640 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:55.825292  1587 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:03:55.835605  1741 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:55.655906  7881 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:55.780318  7833 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.710041  7922 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.104s
W20260709 13:03:55.846251  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 376569us
I20260709 13:03:55.795004  8067 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.843374  1587 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 11. (term mismatch)
I20260709 13:03:55.787698  8031 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:55.847164  1587 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:03:55.847482  1587 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 13:03:55.847777  1587 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:55.844065  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.849196  8067 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:55.844240  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99004.9us
I20260709 13:03:55.854389  1739 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [17.8-17.8]
I20260709 13:03:55.809012  7649 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:03:55.874738  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:55.874974  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:55.849416  1734 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 13:03:55.869403  2314 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:03:55.874063  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3059) took 1016 ms. Trace:
I20260709 13:03:55.868907  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:55.879678  2318 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:03:55.889638  1591 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:03:55.888087  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:55.849961  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:55.874331  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:55.893785  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46729.6us
W20260709 13:03:55.946192  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58186.2us
I20260709 13:03:55.945655  1665 rpcz_store.cc:276] 0709 13:03:54.857749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.149181 (+291432us) service_pool.cc:224] Handling call
0709 13:03:55.169319 (+ 20138us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.226568 (+ 57249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.226605 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.276319 (+ 49714us) log.cc:844] Serialized 26 byte log entry
0709 13:03:55.556953 (+280634us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b180049dce0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:03:55.822937 (+265984us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:55.822995 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.823081 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:55.873634 (+ 50553us) raft_consensus.cc:1685] finished
0709 13:03:55.873673 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:55.873995 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":67496,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":42859,"spinlock_wait_cycles":19794688,"thread_start_us":265435,"threads_started":1,"wal-append.queue_time_us":571397}
I20260709 13:03:55.813462  7480 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:03:55.967489  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.953578  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:55.958686  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:55.965699  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:55.962877  1666 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 13:03:55.825311  7879 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:55.949793  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:55.968640  1664 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:55.970408  2315 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:03:55.972366  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:55.972930  2312 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:03:55.972553  1587 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:55.871480  7962 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:55.977099  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:55.875788  7992 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:55.809654  7965 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.145s
I20260709 13:03:55.993932  1587 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:55.855815  7869 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:55.902750  7836 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:55.945374  7649 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:55.825579  7939 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.144s
I20260709 13:03:55.851094  8071 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:56.001613  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:55.865968  7855 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:55.989537  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:55.962818  7672 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:56.000770  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:56.012494  2315 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:56.003314  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:55.994989  7416 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.858s	user 0.013s	sys 0.002s
I20260709 13:03:56.013833  7672 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:56.026240  7672 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:56.029901  8076 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:56.030567  8076 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:56.031591  8076 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.033840  7873 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:56.034566  7873 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.013873  7836 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.038707  7836 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:56.041690  8064 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.042618  8064 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.043586  7920 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.044162  7920 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.045167  8064 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.022537  7935 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:55.998605  8031 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:56.118484  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:56.024930  7912 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:03:56.008167  7480 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 13:03:55.903082  7833 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:55.858376  7640 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.046315  7920 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:55.807884  7949 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.022255  8025 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:55.974318  8018 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:56.103932  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:56.162417  1661 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:56.056165  7881 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.092432  1588 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:56.087945  2310 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:56.057958  7828 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:03:56.108093  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102028us
I20260709 13:03:55.995090  7631 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.130s
I20260709 13:03:56.111701  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:56.023190  2318 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:03:56.003201  7906 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.634s	user 0.006s	sys 0.006s
W20260709 13:03:56.049350  8050 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.002s	user 0.000s	sys 0.003s
I20260709 13:03:56.007969  7649 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:56.123346  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:56.128377  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:56.129912  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:56.133035  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:56.177280  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:56.177479  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 66162.9us
I20260709 13:03:56.178933  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:56.136816  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:56.180341  2312 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:03:56.147034  2317 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:03:56.062222  8067 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.183815  7875 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.189675  7939 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.190451  7939 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.057341  7979 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.051065  7985 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.164752  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:56.168141  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:56.009438  7945 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.580s	user 0.012s	sys 0.000s
I20260709 13:03:56.170590  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:56.120340  7935 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.207229  7935 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:56.207335  8022 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.207893  8022 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.208923  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.211460  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.182520  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:56.118671  8031 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:56.214504  8031 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.214551  2317 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:56.091785  7855 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.192144  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.215747  1735 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
W20260709 13:03:56.197876  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.200351  1742 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:56.202154  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:56.220973  1661 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:03:56.213182  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:56.223359  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:56.223865  1738 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:56.224123  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 609912us
I20260709 13:03:56.229132  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:56.194746  7640 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:56.242172  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:56.123593  8071 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.092s
I20260709 13:03:56.243221  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.243229  2317 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:56.193125  7828 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:56.249428  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:56.264017  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:56.215273  7875 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:56.271574  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2332) took 2695 ms. Trace:
I20260709 13:03:56.228821  7992 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:56.273427  1662 rpcz_store.cc:276] 0709 13:03:53.576565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.358667 (+782102us) service_pool.cc:224] Handling call
0709 13:03:54.359182 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.682154 (+1322972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.266099 (+583945us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:56.271503 (+  5404us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3038872448}
I20260709 13:03:56.237722  8067 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:56.277323  1735 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:56.294210  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:56.295464  1661 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:03:56.297420  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.204120  7416 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 13:03:56.278206  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:56.261792  7672 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.259044  7880 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.307727  2317 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:56.302000  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:56.200512  7906 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.263054  7985 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:03:56.315351  1666 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:03:56.262564  7748 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:56.325958  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.318548  1738 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:03:56.271792  7946 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:03:56.327179  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:56.329612  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:56.226845  7945 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.333338  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3093) took 1221 ms. Trace:
I20260709 13:03:56.335685  1662 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 14 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 5.
I20260709 13:03:56.281574  7992 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:56.286602  7949 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:56.301975  7873 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.334295  1736 rpcz_store.cc:276] 0709 13:03:55.112032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.122000 (+  9968us) service_pool.cc:224] Handling call
0709 13:03:55.137830 (+ 15830us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:55.138179 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.138210 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:55.138790 (+   580us) log.cc:844] Serialized 52 byte log entry
0709 13:03:55.359303 (+220513us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:55.821554 (+462251us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.282813 (+461259us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.333013 (+ 50200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.333096 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.333144 (+    48us) raft_consensus.cc:1685] finished
0709 13:03:56.333156 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.333279 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101378,"prepare.queue_time_us":70659,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":95,"raft.queue_time_us":134649,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":70222,"thread_start_us":1072393,"threads_started":3,"wal-append.queue_time_us":456132}
I20260709 13:03:56.312178  7979 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.345372  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.333441  7946 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.338101  7741 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.337492  7947 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:56.260578  7869 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.328186  7880 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.353394  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:56.396040  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3096) took 1283 ms. Trace:
I20260709 13:03:56.399031  1733 rpcz_store.cc:276] 0709 13:03:55.112906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.113038 (+   132us) service_pool.cc:224] Handling call
0709 13:03:55.133832 (+ 20794us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.827101 (+693269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.395658 (+568557us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b5400284e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:56.395957 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1591935360}
W20260709 13:03:56.401963  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 385425us
I20260709 13:03:56.349519  7828 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:56.412638  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.014s	sys 0.026s
W20260709 13:03:56.413166  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.570s	user 0.014s	sys 0.026s
I20260709 13:03:56.352631  7979 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:56.413858  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 624342us
I20260709 13:03:56.414888  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:56.386332  8081 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 13:03:56.402732  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 445032us
I20260709 13:03:56.394794  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:56.389114  7694 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.564s	user 0.011s	sys 0.003s
W20260709 13:03:56.438380  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.378952  7748 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:56.450278  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55559.1us
W20260709 13:03:56.453069  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.386345  8070 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.398326  8077 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:56.455515  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.377908  7946 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:56.435938  7833 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.458340  8077 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.387283  7741 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.508901  7855 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.462124  8065 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 13:03:56.402738  7992 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:56.587297  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 131827us
I20260709 13:03:56.587671  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.379433  8078 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:03:56.386318  8069 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:03:56.591370  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 307087us
I20260709 13:03:56.591990  1589 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
W20260709 13:03:56.627318  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3144) took 1565 ms (client timeout 1616 ms). Trace:
I20260709 13:03:56.629839  2314 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:56.635620  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:56.690795  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2408) took 1264 ms (client timeout 1651 ms). Trace:
W20260709 13:03:56.689009  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 101377us
W20260709 13:03:56.708771  1742 rpcz_store.cc:269] 0709 13:03:55.426298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.507513 (+ 81215us) service_pool.cc:224] Handling call
0709 13:03:56.690540 (+1183027us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54000e3654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:56.690682 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1275803520}
I20260709 13:03:56.696436  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2341) took 1258 ms. Trace:
I20260709 13:03:56.637996  2310 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:03:56.710397  1737 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:56.691295  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3053) took 2014 ms. Trace:
I20260709 13:03:56.710263  1659 rpcz_store.cc:276] 0709 13:03:55.437625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.727823 (+290198us) service_pool.cc:224] Handling call
0709 13:03:55.728268 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:55.728551 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.728580 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:55.729166 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 13:03:55.973022 (+243856us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:03:55.973063 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:55.973138 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.696212 (+723074us) raft_consensus.cc:1685] finished
0709 13:03:56.696234 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.696375 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":243451,"threads_started":1,"wal-append.queue_time_us":322124}
I20260709 13:03:56.638278  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:56.716377  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 452240us
W20260709 13:03:56.708173  1585 rpcz_store.cc:269] 0709 13:03:55.022144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.022741 (+   597us) service_pool.cc:224] Handling call
0709 13:03:56.511747 (+1489006us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b540031c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:56.587210 (+ 75463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":834474880}
I20260709 13:03:56.711467  1660 rpcz_store.cc:276] 0709 13:03:54.676577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:54.808935 (+132358us) service_pool.cc:224] Handling call
0709 13:03:55.564230 (+755295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.702634 (+138404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.691102 (+988468us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54000a96d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:56.691234 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1735180032}
I20260709 13:03:56.592294  7949 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:03:56.725894  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:56.730690  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95120.3us
I20260709 13:03:56.593058  8077 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:56.738536  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.607309  7856 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:56.591735  8076 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:56.624396  7875 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:56.634738  7881 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:56.712328  8083 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.370s	user 0.006s	sys 0.000s
I20260709 13:03:56.633917  7985 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:56.685108  8032 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.713088  8064 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:56.623354  8070 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.739455  8018 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:56.729964  8069 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:56.753875  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3177) took 1635 ms. Trace:
I20260709 13:03:56.768540  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.746531  7912 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.773867  1584 rpcz_store.cc:276] 0709 13:03:55.118726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.455687 (+336961us) service_pool.cc:224] Handling call
0709 13:03:56.753628 (+1297941us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b54001fa254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:56.753797 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":84550912}
I20260709 13:03:56.760174  7992 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:56.762751  7920 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:56.752844  7672 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.784983  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:56.787261  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:56.803197  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.788846  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:56.766976  8099 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.781490  7912 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.762392  8076 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.801630  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1018) took 1731 ms. Trace:
I20260709 13:03:56.804059  1584 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:03:56.780483  8100 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.782693  7920 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:56.792439  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:56.789294  7952 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:56.770876  8064 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.767635  8032 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.763793  7881 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.840935  1739 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:56.809425  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:56.866773  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.867022  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:56.875408  8025 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.006s
I20260709 13:03:56.831135  7640 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:03:56.868316  8092 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:03:56.816666  8085 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.848493  8069 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.799449  7985 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.901757  8100 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:56.843623  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.903517  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:56.905378  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:56.812331  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:56.906656  1740 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:56.907133  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:56.906071  8097 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 13:03:56.906754  8098 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 13:03:56.808394  7672 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.091s
I20260709 13:03:56.899472  1736 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:03:56.891669  1593 rpcz_store.cc:276] 0709 13:03:55.069826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.481120 (+411294us) service_pool.cc:224] Handling call
0709 13:03:55.481407 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.677210 (+195803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.801305 (+1124095us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b540025c954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:56.801557 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448631168}
I20260709 13:03:56.932734  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:56.940349  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:56.941118  1593 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.893764  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3160) took 1027 ms. Trace:
I20260709 13:03:56.898764  8032 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.880359  7945 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.843279  8090 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:56.949004  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 12 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:03:56.935633  7935 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.728s	user 0.010s	sys 0.003s
I20260709 13:03:56.853233  7416 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 13:03:56.894624  1585 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:56.909114  1584 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:03:56.907750  7920 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:56.840824  7979 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:56.922466  1739 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:56.923673  8100 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.805938  7741 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.920301  7640 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:03:56.921473  8096 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.080s	user 0.001s	sys 0.001s
I20260709 13:03:56.895656  8025 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:56.779409  8089 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
W20260709 13:03:57.000233  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:56.831929  8099 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:56.919364  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 109970us
W20260709 13:03:56.985373  7649 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.777s	user 0.005s	sys 0.009s
I20260709 13:03:56.957211  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:56.966681  7912 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.967916  8076 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:56.969180  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3060) took 1968 ms. Trace:
I20260709 13:03:56.984725  8111 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:56.970798  7480 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:57.018935  7480 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.019373  7480 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:56.972137  7860 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:56.973002  7935 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.022593  7741 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:57.023063  7741 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:57.023815  7935 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:56.974315  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3195) took 1425 ms. Trace:
I20260709 13:03:57.023978  7649 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.024168  7741 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:56.978590  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.025867  1584 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:03:57.024633  1589 rpcz_store.cc:276] 0709 13:03:55.549148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.841064 (+291916us) service_pool.cc:224] Handling call
0709 13:03:56.587155 (+746091us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.712843 (+125688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.974096 (+261253us) spinlock_profiling.cc:243] Waited 745 ms on lock 0x7b540031c9d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:56.974251 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1714422144}
I20260709 13:03:56.873253  8107 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.028369  8107 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.941809  1586 rpcz_store.cc:276] 0709 13:03:55.866808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.011526 (+144718us) service_pool.cc:224] Handling call
0709 13:03:56.062213 (+ 50687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:56.062468 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:56.230872 (+168404us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.830878 (+600006us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:56.830953 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:56.893357 (+ 62404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.893428 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.893691 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":337222,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":195108,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":128,"thread_start_us":767339,"threads_started":3,"wal-append.queue_time_us":529841}
I20260709 13:03:57.003062  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.032629  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:56.941809  8106 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:57.034817  8106 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.036273  7979 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:57.037647  7979 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:57.011544  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3062) took 2010 ms. Trace:
I20260709 13:03:57.012108  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.018224  2315 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:56.831521  7875 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.039512  1664 rpcz_store.cc:276] 0709 13:03:55.001352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.456692 (+455340us) service_pool.cc:224] Handling call
0709 13:03:55.470764 (+ 14072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:55.989064 (+518300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.011194 (+1022130us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:57.011455 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1131982592}
I20260709 13:03:57.047326  1585 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:57.008807  1663 rpcz_store.cc:276] 0709 13:03:55.000764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.437885 (+437121us) service_pool.cc:224] Handling call
0709 13:03:55.448274 (+ 10389us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:03:55.823356 (+375082us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:55.823411 (+    55us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:03:56.187872 (+364461us) spinlock_profiling.cc:243] Waited 68.5 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:03:56.188602 (+   730us) log.cc:844] Serialized 78 byte log entry
0709 13:03:56.586144 (+397542us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:56.688734 (+102590us) log.cc:844] Serialized 14 byte log entry
0709 13:03:56.714500 (+ 25766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.714595 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:56.968698 (+254103us) raft_consensus.cc:1685] finished
0709 13:03:56.968856 (+   158us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:56.969096 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43725,"prepare.queue_time_us":102547,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":2785,"raft.queue_time_us":178125,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":29121,"spinlock_wait_cycles":166212224,"thread_start_us":746584,"threads_started":2,"wal-append.queue_time_us":654675}
W20260709 13:03:57.048286  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92146.4us
I20260709 13:03:57.048982  1593 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:56.955665  8090 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:56.999253  8100 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:57.055222  1585 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:57.052516  1593 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:57.057425  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.059602  7640 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:57.060719  7640 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:57.061098  7640 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:57.061293  8022 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.062055  7640 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.062840  8064 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.009578  8089 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:57.066847  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.070757  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.073385  1585 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:57.067967  7640 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:57.075580  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.053284  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:57.068892  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 136197us
I20260709 13:03:57.071550  7922 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:57.073280  7979 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:57.083562  1593 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 13:03:57.088130  1593 pending_rounds.cc:85] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 8
I20260709 13:03:57.088487  1593 pending_rounds.cc:107] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 13:03:57.088763  1593 raft_consensus.cc:2889] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:57.076625  7952 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:57.043653  8111 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:57.093463  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3181) took 1910 ms (client timeout 2166 ms). Trace:
I20260709 13:03:57.077862  8087 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:03:57.093993  1588 rpcz_store.cc:269] 0709 13:03:55.182711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.281491 (+ 98780us) service_pool.cc:224] Handling call
0709 13:03:57.093266 (+1811775us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b54003b1fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:57.093405 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1718787328}
I20260709 13:03:57.097589  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.099490  1590 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:57.083528  7881 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.104261  8084 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 13:03:57.105492  8090 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:57.098640  7965 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 13:03:57.108721  8099 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.110457  8107 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.070505  8100 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.099162  7922 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 13:03:57.114863  8084 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:57.115284  8084 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:03:57.116093  8108 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:03:57.116255  8084 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.117425  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3102) took 1650 ms. Trace:
I20260709 13:03:57.117849  1741 rpcz_store.cc:276] 0709 13:03:55.467413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.836885 (+369472us) service_pool.cc:224] Handling call
0709 13:03:55.837239 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:56.314107 (+476868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:56.314158 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:56.334458 (+ 20300us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.809191 (+474733us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:56.809247 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.809487 (+   240us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:57.117130 (+307643us) raft_consensus.cc:1685] finished
0709 13:03:57.117158 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.117354 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5448,"raft.queue_time_us":177331,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":851772,"threads_started":2,"wal-append.queue_time_us":606094}
I20260709 13:03:57.117866  8084 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:57.103956  1593 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:03:57.113951  8102 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 13:03:57.089224  8116 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:57.091377  7952 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:57.127411  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.080888  7949 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 13:03:57.134438  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:57.130182  8116 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.134774  7952 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.137666  8111 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:57.134796  7748 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.635s	user 0.009s	sys 0.005s
I20260709 13:03:57.141487  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:57.149828  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.150699  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.152348  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:57.152866  8031 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.929s	user 0.011s	sys 0.002s
I20260709 13:03:57.153534  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.154217  1738 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:57.154474  8106 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:57.156257  7946 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.517s	user 0.014s	sys 0.000s
I20260709 13:03:57.156705  7946 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.151257  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.160130  8077 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.134686  8089 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.174557  8075 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:03:57.174901  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:03:57.167393  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 421138us
I20260709 13:03:57.176842  8022 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:57.177546  8022 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.177973  8022 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.179827  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:57.182137  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:57.183189  7881 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:57.170660  7855 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:57.183810  7828 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:57.184324  8071 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:03:57.184546  7828 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:57.184796  8071 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:57.185791  8071 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.185868  7828 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.186875  8071 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:57.186746  7855 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:03:57.112994  7979 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:57.175047  8075 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.168684  7856 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:57.189751  8075 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:57.190389  7856 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.170192  8100 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.182847  8076 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:03:57.168030  7614 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:03:57.143186  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.204495  1588 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:57.143729  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.144409  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.144758  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.212257  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.214181  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.215338  1661 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:57.227150  1734 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 5. Preceding OpId from leader: term: 18 index: 8. (index mismatch)
W20260709 13:03:57.246229  1705 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.232s (SENT)
W20260709 13:03:57.246847  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49364.7us
I20260709 13:03:57.248803  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.177243  7873 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:57.249737  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:57.251897  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52988.7us
I20260709 13:03:57.255184  1665 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:57.251921  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:57.193426  8101 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 13:03:57.252573  1659 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:03:57.252790  1738 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:57.213016  1666 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:03:57.264489  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.265758  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.268359  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.142757  8111 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.196880  8076 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:03:57.269325  1661 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:57.142882  8116 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:57.271443  7694 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:03:57.153313  8031 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.142882  7480 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.272785  7855 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:57.273562  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.195899  8025 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 13:03:57.197180  7947 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:57.277549  1664 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 6. Preceding OpId from leader: term: 18 index: 8. (index mismatch)
I20260709 13:03:57.278937  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.148378  7880 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.279588  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.279737  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.147859  7748 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:57.283003  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.245955  8023 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:57.254508  8067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.912s	user 0.028s	sys 0.004s
I20260709 13:03:57.268443  7912 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 13:03:57.275501  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.291635  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:57.264407  7873 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:57.196573  8065 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 13:03:57.268271  8100 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.296516  2315 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.295612  8111 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.321791  7833 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:03:57.365867  7833 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.366325  7920 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.366675  8100 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.316202  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:57.282380  8076 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:57.318624  1666 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 13:03:57.295580  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 197997us
I20260709 13:03:57.366786  8111 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.303565  2314 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 13:03:57.260542  8022 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.355583  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.373129  2318 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:57.321425  1591 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:57.322978  1589 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 13:03:57.377849  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.379794  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:03:57.380251  1708 connection.cc:256] RPC call timeout handler was delayed by -1.6572325270000192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
I20260709 13:03:57.380558  1734 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:03:57.316604  8067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.029s	sys 0.004s
I20260709 13:03:57.385134  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.424476  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:57.396250  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:57.408561  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:57.423295  1739 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:57.426447  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:57.355190  8109 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.322250  8107 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:57.428759  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.429684  1588 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:03:57.356312  7741 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.366364  7833 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:57.355436  8121 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.359622  7480 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.423928  1589 pending_rounds.cc:85] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:03:57.443950  1589 pending_rounds.cc:107] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 13:03:57.444401  1589 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:57.428620  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.430421  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 924) took 1229 ms. Trace:
I20260709 13:03:57.445007  1735 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:57.293184  8018 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 13:03:57.435782  1659 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:03:57.300698  7704 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.133s
I20260709 13:03:57.409693  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.370911  7869 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:57.456552  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.443265  1665 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:57.295681  7947 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:57.368173  8008 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:57.450112  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.464622  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:57.384100  8066 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:57.465750  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:57.423756  8067 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:57.452234  1741 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 13:03:57.479797  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.480422  1736 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:57.444947  2312 rpcz_store.cc:276] 0709 13:03:56.201542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.201767 (+   225us) service_pool.cc:224] Handling call
0709 13:03:56.217081 (+ 15314us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:56.259933 (+ 42852us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:56.259969 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:56.271369 (+ 11400us) log.cc:844] Serialized 26 byte log entry
0709 13:03:56.571973 (+300604us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:56.572046 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:56.572155 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:57.192780 (+620625us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:57.385079 (+192299us) raft_consensus.cc:1685] finished
0709 13:03:57.385113 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.430319 (+ 45206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1856,"raft.queue_time_us":12566,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":10797,"spinlock_wait_cycles":36076800,"thread_start_us":244175,"threads_started":1,"wal-append.queue_time_us":498249}
I20260709 13:03:57.458591  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.458709  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.433984  8111 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:57.323266  8023 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.384565  7875 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 13:03:57.482107  1667 raft_consensus.cc:2376] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17. Re-sending same reply.
I20260709 13:03:57.519117  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.429363  7748 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 13:03:57.356451  7873 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:57.521263  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55551.8us
I20260709 13:03:57.388573  8087 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 13:03:57.520043  1659 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:57.451221  8106 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 13:03:57.524770  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48309.3us
I20260709 13:03:57.454909  7939 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.445700  8107 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.536659  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:57.469945  8089 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:57.537945  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:57.545738  8119 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 13:03:57.470723  7869 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.544600  7947 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.441859  8109 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.476162  7935 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:03:57.521554  8111 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.533893  7946 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.468250  8076 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.560213  8089 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.646826  8076 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:57.480224  7416 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:03:57.570541  2318 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:03:57.528916  7945 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:57.518499  8032 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.075s
I20260709 13:03:57.574753  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.554463  1667 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:03:57.479660  8066 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.586038  8075 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.568559  8077 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.461166  8121 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.594195  8022 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.594355  7920 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.598029  7935 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:57.577171  8111 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.600437  8109 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.604652  7881 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:57.702258  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.607831  1662 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 13:03:57.608052  8070 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:57.486529  7912 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 13:03:57.736374  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:57.736455  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.736896  7912 consensus_queue.cc:1243] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:03:57.737654  1662 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:57.605924  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3201) took 2055 ms. Trace:
I20260709 13:03:57.613101  1661 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:03:57.739694  1741 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.739948  1660 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22. Ops: [21.11-21.11]
W20260709 13:03:57.616523  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1085) took 2186 ms (client timeout 2007 ms). Trace:
I20260709 13:03:57.682402  7416 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:03:57.616704  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2346) took 2018 ms (client timeout 2005 ms). Trace:
W20260709 13:03:57.741847  1668 rpcz_store.cc:269] 0709 13:03:55.598390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.698697 (+100307us) service_pool.cc:224] Handling call
0709 13:03:57.616506 (+1917809us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54002f1f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:57.616636 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3198960512}
I20260709 13:03:57.742482  7416 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.743237  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.620711  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:57.621522  8123 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:03:57.623059  8031 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.745180  1590 rpcz_store.cc:276] 0709 13:03:55.550740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.842355 (+291615us) service_pool.cc:224] Handling call
0709 13:03:57.093399 (+1251044us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.099850 (+  6451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.605609 (+505759us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54003b1fd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:57.605827 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2765188352}
I20260709 13:03:57.624874  7869 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.624370  8067 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:57.628171  8084 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.508s	user 0.010s	sys 0.000s
I20260709 13:03:57.629019  7947 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:57.609073  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:57.639259  8128 consensus_queue.cc:1175] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e6adef9da7da44f6af0af5695cf9b704" responder_term: 19 status { last_received { term: 19 index: 12 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:03:57.589110  7906 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.649423  8089 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.563789  7985 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:03:57.650910  7952 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.508s	user 0.008s	sys 0.003s
I20260709 13:03:57.665992  1739 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:57.486282  8065 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.097s
I20260709 13:03:57.685832  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.774096  1735 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:03:57.479735  8008 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:57.715821  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.52274e+06us
I20260709 13:03:57.604230  7649 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.717494  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.699262  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:03:57.706840  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 178921us
I20260709 13:03:57.718672  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.716908  7935 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.723091  8066 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.700912  8122 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:57.729676  1733 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 13:03:57.734393  7881 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.783663  1735 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 13:03:57.784092  1735 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 8
I20260709 13:03:57.784308  1735 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 13:03:57.784544  1735 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:57.612138  8107 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.738624  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:57.743322  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.616922  8023 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.740783  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.745997  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:57.749481  1587 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.800060  1661 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 13:03:57.754364  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:03:57.800289  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62815.4us
I20260709 13:03:57.688723  8087 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:57.800766  1736 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:57.674787  7945 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.759599  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.761169  1664 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:57.750288  7836 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:03:57.767863  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 181174us
W20260709 13:03:57.768682  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 203911us
W20260709 13:03:57.770690  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3184) took 2576 ms (client timeout 2091 ms). Trace:
I20260709 13:03:57.772581  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.803756  1739 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:03:57.802842  1592 rpcz_store.cc:269] 0709 13:03:55.183581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.291808 (+108227us) service_pool.cc:224] Handling call
0709 13:03:57.757213 (+2465405us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b5400443c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:57.759458 (+  2245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2938078464}
I20260709 13:03:57.803977  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.804983  1590 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.804913  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.805208  1663 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:57.805884  1739 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.775300  1741 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:57.778204  1667 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:03:57.777863  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.785790  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:57.789227  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:03:57.741159  1737 rpcz_store.cc:269] 0709 13:03:55.430178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:55.609337 (+179159us) service_pool.cc:224] Handling call
0709 13:03:57.616325 (+2006988us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54003f9dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:57.616449 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1133757824}
I20260709 13:03:57.799834  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3173) took 1159 ms. Trace:
I20260709 13:03:57.807838  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:57.811182  2318 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:57.807395  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:57.809098  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.811937  1585 rpcz_store.cc:276] 0709 13:03:56.640237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.080374 (+440137us) service_pool.cc:224] Handling call
0709 13:03:57.085454 (+  5080us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:57.086134 (+   680us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.086173 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:57.100413 (+ 14240us) log.cc:844] Serialized 26 byte log entry
0709 13:03:57.257716 (+157303us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:57.257786 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.257899 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:57.799576 (+541677us) raft_consensus.cc:1685] finished
0709 13:03:57.799605 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.799766 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":4145,"thread_start_us":133669,"threads_started":1,"wal-append.queue_time_us":326300}
I20260709 13:03:57.813273  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.814270  1659 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:03:57.811105  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.815557  2312 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:57.816131  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.781924  8122 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:57.817586  1587 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.817157  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.818411  1667 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:03:57.819131  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.818688  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.819836  1592 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:03:57.820598  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:57.776845  7952 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.816906  1590 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:57.821595  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:57.834950  1585 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:03:57.835011  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.835243  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.822149  8032 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:57.822400  8135 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:03:57.823513  7992 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:57.813575  8066 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:57.839856  8066 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.840312  8066 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.840759  8018 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:57.841228  8018 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.841646  8018 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:57.797861  7833 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.842487  8066 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.843751  7833 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:57.844907  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.813632  7949 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 13:03:57.819554  8031 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:57.805991  7836 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.825623  7979 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 13:03:57.827672  7881 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:57.883922  1742 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
W20260709 13:03:57.830389  1487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 228) took 79968 ms (client timeout 15000 ms). Trace:
I20260709 13:03:57.830878  8065 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.806406  8089 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:57.837410  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.889271  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.890103  2317 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:03:57.890246  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.802309  7945 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:57.837929  1736 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:03:57.846439  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.848186  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.891749  1741 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:03:57.892045  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.845683  1592 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:03:57.853631  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:57.854487  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.893296  1587 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:57.894773  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.895568  1737 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:57.830288  8121 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.801388  8116 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.828804  8075 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.896265  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:57.888890  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.858120  8109 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:57.887506  7741 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.898221  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.899489  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.808007  8087 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:57.901299  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.882184  7836 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:57.898052  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43655.9us
I20260709 13:03:57.865473  8071 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 13:03:57.981338  2317 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:58.015137  1665 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:57.954516  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:57.889359  8089 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:57.897107  8065 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:57.957787  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.030912  1592 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:58.018242  7935 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.017856  8008 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.032199  8065 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:57.966918  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:58.044943  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 78118.4us
I20260709 13:03:58.045466  8106 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:03:58.045974  7833 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:58.046380  7880 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:58.046830  7880 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:58.045977  8106 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:58.032063  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.049121  8106 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.049611  7833 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:57.927843  1664 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:58.050819  7939 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:58.051287  7939 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:58.051481  1592 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:57.915022  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.929415  7748 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:57.962188  7952 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:57.977654  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.019408  7962 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:58.040104  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.017030  7881 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.016291  2318 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:58.041656  7949 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.135854  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:58.136587  2310 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:58.041782  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:57.891753  7992 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:57.976209  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3161) took 2109 ms. Trace:
I20260709 13:03:58.020412  7661 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:57.911235  2314 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:03:58.093885  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116318us
I20260709 13:03:58.140019  1591 rpcz_store.cc:276] 0709 13:03:55.867059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.013613 (+146554us) service_pool.cc:224] Handling call
0709 13:03:56.119603 (+105990us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.423710 (+1304107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.975843 (+552133us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400393fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:57.976040 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2762065664}
I20260709 13:03:58.016505  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:57.973069  1659 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:58.143675  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:57.974958   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:03:57.998525  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3162) took 2108 ms. Trace:
W20260709 13:03:58.107511  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.144639  1584 rpcz_store.cc:276] 0709 13:03:55.890604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.026632 (+1136028us) service_pool.cc:224] Handling call
0709 13:03:57.027002 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.036214 (+  9212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:57.036254 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:57.036300 (+    46us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:03:57.036331 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:57.998221 (+961890us) spinlock_profiling.cc:243] Waited 8.99 ms on lock 0x7b5400002654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:57.998279 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:57.998456 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20670336}
I20260709 13:03:58.055305  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:57.910378  1587 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:03:58.026255  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.147606  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:58.148137  1663 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:58.148406  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:57.991883  8100 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.081576  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.127381  1665 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 13:03:58.132877  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.021883  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.163821  1665 pending_rounds.cc:85] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 11
I20260709 13:03:58.164150  1662 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:03:58.164320  1665 pending_rounds.cc:107] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 13:03:58.164606  1665 raft_consensus.cc:2889] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:58.138732  2317 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:58.137284  1733 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:58.139545  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:58.092876  8106 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:58.044979  8085 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Stepping down as leader of term 21
W20260709 13:03:58.144595  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 128103us
I20260709 13:03:58.179093  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3163) took 2211 ms. Trace:
I20260709 13:03:58.179636  1589 rpcz_store.cc:276] 0709 13:03:55.968035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.028116 (+1060081us) service_pool.cc:224] Handling call
0709 13:03:57.028459 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.458436 (+429977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.178820 (+720384us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b540042e354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:58.179025 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":675570560}
I20260709 13:03:58.181393  1589 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:03:58.182132  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.187201  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 12 index: 5 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.145718  7875 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:58.125221  1666 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:58.077617  1741 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:03:58.151754  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:58.149945  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:03:58.197875  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46158.7us
I20260709 13:03:58.139406  7979 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 13:03:57.903965  7946 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:58.163401  1590 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:03:58.136588  7430 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:03:58.204084  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.172034  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:58.206195  2317 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: [15.13-17.14]
I20260709 13:03:58.173089  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.149827  7947 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.149266  7661 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.095580  8018 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.157109  2310 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:03:58.163800  7828 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:58.092811  8089 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:03:58.205731  7430 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.201159  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 13 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:58.222568  1741 raft_consensus.cc:2376] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13. Re-sending same reply.
W20260709 13:03:58.223299  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.223600  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.224182  1741 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:03:58.224993  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.225582  1741 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:58.226383  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.201787  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.204293  1590 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:03:58.205715  8078 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 13:03:58.228615  8078 consensus_queue.cc:1243] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:03:58.138513  7748 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.206480  2314 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:58.207777  1589 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:03:58.138502  7741 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.294737  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:58.296252  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.296655  7741 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:58.140021  7881 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.052386  7939 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.297948  7939 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:58.299912  7881 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:58.094049  8133 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 13:03:58.123986  7962 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:58.210879  7416 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:58.301527  7962 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.303288  1591 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 15 index: 7. (index mismatch)
I20260709 13:03:58.304948  2317 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:03:58.188752  8100 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:03:58.212522  8128 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.096s
I20260709 13:03:58.185174  8111 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.307251  1734 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 13:03:58.216137  7869 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:58.308423  7869 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.308864  7869 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:58.217044  7947 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.222456  8153 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
I20260709 13:03:58.231926  8070 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.317102  8070 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.232671  7430 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:03:58.319234  7661 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:58.320050  7661 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:03:58.320732  1586 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:58.233680  7875 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.099s
I20260709 13:03:58.323614  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.324628  1585 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:03:58.291559  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.021708  7945 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.141562  7949 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.206254  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.330718  2311 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:58.330948  2317 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:03:58.200960  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:58.331858  8128 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:58.204038  7649 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:03:58.332274  8128 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:58.332731  7949 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:58.239641  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3174) took 1525 ms. Trace:
I20260709 13:03:58.333449  1593 rpcz_store.cc:276] 0709 13:03:56.714067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.090185 (+376118us) service_pool.cc:224] Handling call
0709 13:03:57.090594 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.124161 (+ 33567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.230995 (+1106834us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b54003b79d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:58.239543 (+  8548us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29897984}
I20260709 13:03:58.222214  7992 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.294440  7748 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.323314  2314 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:03:58.338258  2314 raft_consensus.cc:606] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:58.338814  2314 raft_consensus.cc:614] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:58.330335  8018 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:03:58.331353  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125108us
I20260709 13:03:58.332738  8128 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:58.176406  8085 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:57.904487  8122 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.329275  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:03:58.356688  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 209315us
I20260709 13:03:58.368662  1736 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:03:58.334061  7962 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:58.374332  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45159.2us
I20260709 13:03:58.383070  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.373701  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.342896  7947 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:58.391775  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.397737  1736 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:58.391309  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.398718  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.401440  1665 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:03:58.343923  8111 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.404700  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.338050  7875 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 13:03:58.418340  8070 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.420441  7873 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:58.401751  8065 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:58.422171  8065 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.422572  8065 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:57.887490  1487 rpcz_store.cc:269] 0709 13:02:37.928062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:37.932108 (+  4046us) service_pool.cc:224] Handling call
0709 13:03:57.830088 (+79897980us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:43.197908 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:43.453846 (+255938us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:43.453923 (+    77us) write_op.cc:435] Acquired schema lock
0709 13:03:43.453975 (+    52us) tablet.cc:662] Decoding operations
0709 13:03:43.456060 (+  2085us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:43.456194 (+   134us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:43.456237 (+    43us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:03:43.508129 (+ 51892us) tablet.cc:701] Row locks acquired
0709 13:03:43.508190 (+    61us) write_op.cc:260] PREPARE: finished
0709 13:03:43.508388 (+   198us) write_op.cc:270] Start()
0709 13:03:43.508547 (+   159us) write_op.cc:276] Timestamp: P: 1783602223508315 usec, L: 0
0709 13:03:43.508581 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 13:03:43.509477 (+   896us) log.cc:844] Serialized 6353 byte log entry
0709 13:03:44.772766 (+1263289us) spinlock_profiling.cc:243] Waited 9.55 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdeaed30ac 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b
0709 13:03:46.375051 (+1602285us) op_driver.cc:464] REPLICATION: finished
0709 13:03:47.003035 (+627984us) write_op.cc:301] APPLY: starting
0709 13:03:47.003191 (+   156us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:47.186469 (+183278us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:47.186527 (+    58us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:47.198108 (+ 11581us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:47.198550 (+   442us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:03:47.198629 (+    79us) write_op.cc:312] APPLY: finished
0709 13:03:47.199263 (+   634us) log.cc:844] Serialized 148 byte log entry
0709 13:03:47.271633 (+ 72370us) spinlock_profiling.cc:243] Waited 24.8 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfda32a 00007fcdedf5ff14 00007fcdedf5fec3 00007fcdedf5fe38 00007fcdedf5fdf3 00007fcdedf5f0dc 00007fcdedf88e16 00007fcdedf70cbb 00007fcdedf5b39d 00007fcdedf5ed75
0709 13:03:47.498321 (+226688us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:47.522523 (+ 24202us) write_op.cc:454] Released schema lock
0709 13:03:47.603011 (+ 80488us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:47.630389 (+ 27378us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:47.633995 (+  3606us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22545,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":586449,"apply.run_cpu_time_us":46765,"apply.run_wall_time_us":612751,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71039,"num_ops":16,"prepare.queue_time_us":340798,"prepare.run_cpu_time_us":90734,"prepare.run_wall_time_us":2046808,"raft.queue_time_us":828649,"raft.run_cpu_time_us":42724,"raft.run_wall_time_us":1065418,"replication_time_us":2793225,"spinlock_wait_cycles":79073408,"thread_start_us":2933952,"threads_started":5,"wal-append.queue_time_us":2026449,"wal-append.run_cpu_time_us":75064,"wal-append.run_wall_time_us":2664444}]]}
I20260709 13:03:58.434124  8085 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.389986  8100 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:03:58.428117  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.438148  1593 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:03:58.448074  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.449918  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.451386  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:58.451977  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.376220  7748 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.065s
W20260709 13:03:58.458591  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 262083us
I20260709 13:03:58.457365  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:58.460914  1593 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:58.459645  1706 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.453111  8100 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:58.456406  7875 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:58.454986  8087 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:58.479769  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3118) took 1666 ms. Trace:
I20260709 13:03:58.480273  1740 rpcz_store.cc:276] 0709 13:03:56.813455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:56.918704 (+105249us) service_pool.cc:224] Handling call
0709 13:03:56.919135 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.075594 (+156459us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:03:57.281324 (+205730us) log.cc:844] Serialized 14 byte log entry
0709 13:03:57.449723 (+168399us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:57.449785 (+    62us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:03:57.449838 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.479487 (+1029649us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400284454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:58.479542 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:58.479701 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9197,"prepare.queue_time_us":214524,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":11771,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":21234,"spinlock_wait_cycles":359042816,"thread_start_us":345542,"threads_started":2,"wal-append.queue_time_us":239763}
I20260709 13:03:58.484179  1736 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:03:58.484318  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.485553  1740 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:03:58.486085  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:03:58.518584  1740 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:03:58.546584  1665 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 13:03:58.537720  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:58.548925  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.549441  1593 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:03:58.550706  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:58.551239  8087 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:03:58.551695  8087 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:58.552280  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.552559  8087 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:58.554236  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 94632.2us
W20260709 13:03:58.555974  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.556118  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:58.556838  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.557155  8066 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:58.557947  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:58.558710  2318 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:03:58.521371  7979 consensus_queue.cc:1243] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:03:58.568504  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.553386  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.586380  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:58.602840  1659 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:03:58.616459  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 224615us
W20260709 13:03:58.647480  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79024.7us
I20260709 13:03:58.648661  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.652081  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.561203  2280 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.660503  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:58.664408  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.665489  1593 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:03:58.665068  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46740.9us
I20260709 13:03:58.667707  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:58.670275  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:58.670682  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:58.671612  2315 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:03:58.660621  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 99480.9us
I20260709 13:03:58.674106  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.671105  1740 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:03:58.535027  8153 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.683005  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:58.546772  8032 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:58.693091  1738 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 7
I20260709 13:03:58.693610  1738 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 13:03:58.693889  1738 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:03:58.686651  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.683815  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:58.586496  7416 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:58.721534  1737 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:03:58.586098  8133 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:58.695740  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:03:58.742025  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46399.6us
I20260709 13:03:58.743294  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.697683  1593 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:03:58.646051  8159 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:58.721041  1735 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:03:58.748543  1735 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:58.689885  8143 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:03:58.681087  8008 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:58.681069  8156 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:58.587211  8106 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:03:58.678421  8071 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:58.685267  8087 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:03:58.684684  8111 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:03:58.757568  8067 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:03:58.692098  8154 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:58.693218  8153 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.754447  8066 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:03:58.676443  8100 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.786958  8071 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.727007  8032 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.745326  8133 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.757177  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.726477  7945 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:58.805470  8111 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:58.739285  8099 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:03:58.685150  8078 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
I20260709 13:03:58.691947  7741 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 13:03:58.754979  8143 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.769364  8070 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:03:58.761121  8008 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:58.807101  7748 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:58.758471  1735 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.823772  8066 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:58.756776  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:58.760684  8159 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.892037  1735 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:58.790212  8156 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:58.826184  8071 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:58.758162  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:03:58.895061  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:58.927028  1665 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:03:58.759447  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:58.928617  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 169178us
I20260709 13:03:58.930302  1659 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:03:58.846818  8099 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:03:58.858685  8143 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:58.925330  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:58.859036  8070 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:58.934856  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:58.935004  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.863453  8008 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:58.976666  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:58.976821  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.970744  1735 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:03:58.979097  1593 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:03:58.980535  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.986578  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.988134  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:58.993134  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:58.994241  1659 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:59.001605  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:59.003908  1660 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.004s	sys 0.027s
W20260709 13:03:59.005281  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.004s	sys 0.027s
I20260709 13:03:58.931923  7946 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:58.894459  7979 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:58.939556  8154 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.046571  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:03:59.047847  1593 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 13:03:58.926047  8163 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.020098  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:59.052461  1659 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:03:58.938385  8099 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:03:59.022835  1665 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:03:58.975225  8070 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:03:58.890865  7748 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:58.997855  7939 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:59.060698  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:59.062608  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 410637us
I20260709 13:03:59.063714  1665 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
W20260709 13:03:59.065914  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 381994us
I20260709 13:03:59.008239  8171 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:59.018137  7947 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:03:59.069559  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:03:59.073330  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 316473us
I20260709 13:03:59.073901  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:59.078199  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 101584us
I20260709 13:03:59.081789  1593 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:03:59.083369  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3177) took 1910 ms. Trace:
I20260709 13:03:59.070580  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.085115  2318 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:03:59.069779  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:59.083885  1588 rpcz_store.cc:276] 0709 13:03:57.173427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.265641 (+ 92214us) service_pool.cc:224] Handling call
0709 13:03:57.272597 (+  6956us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.436356 (+163759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.083147 (+1646791us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540025e754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.083310 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":360237440}
I20260709 13:03:59.084980  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.074384  7992 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 13:03:59.103796  8161 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 13:03:59.105386  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3215) took 1541 ms. Trace:
I20260709 13:03:59.105841  1585 rpcz_store.cc:276] 0709 13:03:57.563504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.325336 (+761832us) service_pool.cc:224] Handling call
0709 13:03:58.325821 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.366430 (+ 40609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.366527 (+    97us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.390650 (+ 24123us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.925168 (+534518us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:58.925215 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.925299 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.105083 (+179784us) raft_consensus.cc:1685] finished
0709 13:03:59.105110 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.105315 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7343,"raft.queue_time_us":6489,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":9599,"thread_start_us":483790,"threads_started":1,"wal-append.queue_time_us":454523}
I20260709 13:03:59.075011  7416 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:59.108840  1585 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 13:03:59.056139  7979 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.068517  7945 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 13:03:59.111359  8076 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.116719  8078 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:03:59.119879  8116 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.105835  7992 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:03:59.062409  8163 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.117570  7873 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:59.108717  8025 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.127012  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3185) took 1645 ms. Trace:
I20260709 13:03:59.169755  1584 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:03:59.067754  8066 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.126969  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3208) took 1747 ms. Trace:
I20260709 13:03:59.178673  1586 rpcz_store.cc:276] 0709 13:03:57.379407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.146250 (+766843us) service_pool.cc:224] Handling call
0709 13:03:59.118410 (+972160us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400431054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:59.126897 (+  8487us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":260716928}
I20260709 13:03:59.129456  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:59.144018  1708 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:03:59.154685  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 84964us
I20260709 13:03:59.186630  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.170167  1591 rpcz_store.cc:276] 0709 13:03:57.481204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.199562 (+718358us) service_pool.cc:224] Handling call
0709 13:03:58.210899 (+ 11337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.303728 (+ 92829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.122673 (+818945us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b540025dd54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.126948 (+  4275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211392256}
I20260709 13:03:59.127152  1588 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
W20260709 13:03:59.175909  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97711.2us
I20260709 13:03:59.191131  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:59.189893  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.199880  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:03:59.081650  8111 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.201977  8182 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:59.202394  7748 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.202543  8182 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.204080  7748 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:03:59.204533  7748 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.204818  8182 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.204888  7748 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:03:59.205673  8166 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.287s	user 0.001s	sys 0.001s
I20260709 13:03:59.205797  8186 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.206411  8186 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.177176  8153 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.208737  8186 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.209939  8186 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:03:59.211052  8008 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:03:59.212162  8089 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.906s	user 0.006s	sys 0.006s
I20260709 13:03:59.212616  8089 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.213812  1733 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:59.214464  8008 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.216792  8065 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.794s	user 0.010s	sys 0.000s
W20260709 13:03:59.217711  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 101254us
I20260709 13:03:59.218789  8075 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:03:59.219516  8075 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:03:59.220947  7430 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.902s	user 0.011s	sys 0.000s
I20260709 13:03:59.221346  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:03:59.221669  8170 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.316s	user 0.000s	sys 0.002s
W20260709 13:03:59.221728  7828 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.918s	user 0.013s	sys 0.000s
I20260709 13:03:59.177793  8116 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.174149  7881 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:59.146486  8076 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.181931  7873 consensus_queue.cc:1243] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:03:59.247170  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:59.250362  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:59.192209  8171 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:03:59.150251  7416 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.190824  8032 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.248334  8116 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.248565  7881 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.243520  8153 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:03:59.258279  8156 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.262283  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:59.217278  8065 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.176805  8025 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:59.263942  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77255.3us
W20260709 13:03:59.261382  8128 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.888s	user 0.009s	sys 0.000s
I20260709 13:03:59.257763  7979 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.242719  7828 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:03:59.256539  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 66770.7us
I20260709 13:03:59.271984  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.267266  7649 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:59.272374  8169 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.123s
I20260709 13:03:59.273099  8180 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.262706  8008 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:03:59.240736  7430 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.200376  7661 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
W20260709 13:03:59.273480  8162 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
W20260709 13:03:59.273612  8167 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
I20260709 13:03:59.283275  8153 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:59.329982  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 602) took 1004 ms. Trace:
I20260709 13:03:59.329648  8153 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.330406  2316 rpcz_store.cc:276] 0709 13:03:58.325671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.400509 (+ 74838us) service_pool.cc:224] Handling call
0709 13:03:58.411349 (+ 10840us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.411829 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.411867 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.412495 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.727591 (+315096us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:03:58.727637 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.727715 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.329597 (+601882us) raft_consensus.cc:1685] finished
0709 13:03:59.329621 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.329784 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7346,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"thread_start_us":314773,"threads_started":1,"wal-append.queue_time_us":518801}
I20260709 13:03:59.332489  1664 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:03:59.288678  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:59.291335  8025 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.278446  8172 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 13:03:59.264719  8183 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:59.302331  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.332624  8078 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
W20260709 13:03:59.349443  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1105) took 1309 ms (client timeout 1685 ms). Trace:
W20260709 13:03:59.278187  8174 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 13:03:59.264731  8159 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.299995  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.262425  7992 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.223007  7939 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:03:59.323433  1661 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 13:03:59.295969  7869 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.987s	user 0.010s	sys 0.005s
I20260709 13:03:59.374495  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3128) took 1655 ms. Trace:
I20260709 13:03:59.381057  1742 rpcz_store.cc:276] 0709 13:03:57.719607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.781003 (+ 61396us) service_pool.cc:224] Handling call
0709 13:03:57.801030 (+ 20027us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:57.891516 (+ 90486us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.374202 (+1482686us) spinlock_profiling.cc:243] Waited 82.5 ms on lock 0x7b5400403b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.374419 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":189775744}
I20260709 13:03:59.382426  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:59.383562  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:03:59.384119  1742 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:03:59.286851  8089 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.345358  8065 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.345712  8018 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
W20260709 13:03:59.351254  1737 rpcz_store.cc:269] 0709 13:03:58.040140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.137804 (+ 97664us) service_pool.cc:224] Handling call
0709 13:03:59.349149 (+1211345us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400350cd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:59.349361 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1325058816}
I20260709 13:03:59.334105  1659 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:03:59.326498  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54602.3us
W20260709 13:03:59.302367  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141074us
W20260709 13:03:59.279106  7962 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.888s	user 0.009s	sys 0.002s
W20260709 13:03:59.362425  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60131.2us
W20260709 13:03:59.275482  7836 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.135s	user 0.015s	sys 0.000s
I20260709 13:03:59.320627  8156 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.359946  8025 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:59.342216  8189 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:03:59.369473  8087 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.000s
I20260709 13:03:59.304670  8075 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:59.376130  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2437) took 1493 ms. Trace:
I20260709 13:03:59.328593  7430 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.301613  8128 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.332542  8071 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.482059  8128 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.483075  8071 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.377004  7875 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.893s	user 0.013s	sys 0.000s
I20260709 13:03:59.483806  7875 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:59.389317  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:59.369544  7741 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:03:59.312302  7748 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.487847  7748 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.387529  7869 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.490108  7869 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.404124  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.395628  7836 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.408154  1668 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 13:03:59.411243  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.492416  7836 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.492969  7836 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:03:59.463047  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 156678us
I20260709 13:03:59.406711  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:59.493368  1739 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:03:59.496658  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2429) took 1661 ms. Trace:
W20260709 13:03:59.497619  8164 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.559s	user 0.002s	sys 0.000s
I20260709 13:03:59.497920  1666 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [17.12-17.12]
W20260709 13:03:59.498273  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1109) took 1449 ms (client timeout 1835 ms). Trace:
W20260709 13:03:59.498682  1733 rpcz_store.cc:269] 0709 13:03:58.049185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.184678 (+135493us) service_pool.cc:224] Handling call
0709 13:03:59.498063 (+1313385us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54003fa2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:59.498213 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2267037568}
I20260709 13:03:59.499130  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2381) took 1061 ms. Trace:
I20260709 13:03:59.499547  1666 rpcz_store.cc:276] 0709 13:03:58.438243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.438945 (+   702us) service_pool.cc:224] Handling call
0709 13:03:58.439399 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.498333 (+1058934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.498896 (+   563us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.499073 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432555264}
I20260709 13:03:59.500547  7741 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:03:59.357263  8180 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.504175  8180 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.503974  8099 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.504535  8133 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.473069  1734 rpcz_store.cc:276] 0709 13:03:57.883168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.902932 (+ 19764us) service_pool.cc:224] Handling call
0709 13:03:57.903310 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.307826 (+404516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.375840 (+1068014us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400338954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.376066 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":927839488}
I20260709 13:03:59.505059  1665 raft_consensus.cc:606] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:03:59.505487  1665 raft_consensus.cc:614] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:03:59.507717  7430 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.508297  7430 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.511654  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:59.513093  1668 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.513855  1585 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 13:03:59.515057  8165 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:03:59.515439  8165 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:03:59.516201  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2379) took 1123 ms. Trace:
I20260709 13:03:59.516590  1661 rpcz_store.cc:276] 0709 13:03:58.392931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.393059 (+   128us) service_pool.cc:224] Handling call
0709 13:03:58.393371 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.369081 (+975710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.515958 (+146877us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b5400075dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.516139 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2136745216}
I20260709 13:03:59.497084  1739 rpcz_store.cc:276] 0709 13:03:57.834782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.834975 (+   193us) service_pool.cc:224] Handling call
0709 13:03:59.496455 (+1661480us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:03:59.496594 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3809556736}
W20260709 13:03:59.470660  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 270731us
I20260709 13:03:59.499205  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 937) took 1281 ms. Trace:
I20260709 13:03:59.526396  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.473241  7946 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:59.480172  8025 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:03:59.490249  8158 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.744s	user 0.003s	sys 0.006s
I20260709 13:03:59.528492  1668 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:03:59.474541  1659 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:03:59.526316  2310 rpcz_store.cc:276] 0709 13:03:58.221422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.230349 (+  8927us) service_pool.cc:224] Handling call
0709 13:03:58.238175 (+  7826us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.238787 (+   612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.238835 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.328079 (+ 89244us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.969176 (+641097us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:58.969227 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.969308 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.496848 (+527540us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:59.498919 (+  2071us) raft_consensus.cc:1685] finished
0709 13:03:59.498942 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.499144 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":284,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":117,"spinlock_wait_cycles":68787968,"thread_start_us":492144,"threads_started":1,"wal-append.queue_time_us":482010}
W20260709 13:03:59.531080  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 124407us
I20260709 13:03:59.540027  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.466558  8107 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:03:59.543339  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.544945  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3080) took 1784 ms. Trace:
I20260709 13:03:59.472981  8075 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.549307  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:59.546481  1667 rpcz_store.cc:276] 0709 13:03:57.760383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:57.827749 (+ 67366us) service_pool.cc:224] Handling call
0709 13:03:59.542102 (+1714353us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:03:59.542348 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.542559 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.542585 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.542633 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:03:59.542667 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.542737 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.544857 (+  2120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11031808}
I20260709 13:03:59.550254  1659 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:03:59.493440  7836 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.516230  8165 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.547792  7748 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:03:59.554930  8165 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:03:59.555476  7748 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.555976  7748 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.540052  8154 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:03:59.558157  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3211) took 2086 ms. Trace:
I20260709 13:03:59.515453  8070 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.542565  7939 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:59.558442  8189 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:03:59.559192  7939 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.542124  8133 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:03:59.540786  8180 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.562412  8180 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.548207  8071 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:03:59.543531  8018 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:03:59.565985  1742 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:59.542836  8159 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:59.567327  8159 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.469899  8156 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.543368  8089 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.494043  8087 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.547420  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.571231  2310 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:03:59.571050  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.594949  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:03:59.576206  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3222) took 1134 ms. Trace:
I20260709 13:03:59.577376  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3214) took 2014 ms. Trace:
I20260709 13:03:59.571892  2314 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:03:59.573160  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.572777  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.557081  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:03:59.572357  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.597451  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:59.573973  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:03:59.561333  7939 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.581154  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:59.641352  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93989.7us
I20260709 13:03:59.605979  2317 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:03:59.604350  1663 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:03:59.543054  8153 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:59.655567  2309 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:03:59.657680  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 60332.5us
I20260709 13:03:59.659201  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.544162  8107 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:03:59.660856  2312 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:03:59.558597  1589 rpcz_store.cc:276] 0709 13:03:57.472294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.182559 (+710265us) service_pool.cc:224] Handling call
0709 13:03:58.183372 (+   813us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.246180 (+ 62808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.557822 (+1311642us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b54002d19d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.558087 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54785792}
I20260709 13:03:59.640300  8018 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:03:59.592540  8075 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:03:59.641479  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 84463.2us
I20260709 13:03:59.578647  8025 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.673175  1706 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:03:59.569428  8159 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.576309  8160 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.670249  2318 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:03:59.604739  1590 rpcz_store.cc:276] 0709 13:03:57.563138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.227850 (+664712us) service_pool.cc:224] Handling call
0709 13:03:58.228200 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.228542 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.228568 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.229012 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.454562 (+225550us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:58.454625 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.454719 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.489674 (+1034955us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:03:59.576987 (+ 87313us) raft_consensus.cc:1685] finished
0709 13:03:59.577020 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.577289 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1159,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":63302,"spinlock_wait_cycles":10857728,"thread_start_us":225207,"threads_started":1,"wal-append.queue_time_us":446669}
I20260709 13:03:59.604380  1593 rpcz_store.cc:276] 0709 13:03:58.441799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.082737 (+640938us) service_pool.cc:224] Handling call
0709 13:03:59.083047 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:59.200755 (+117708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:59.200792 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:59.201307 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 13:03:59.479268 (+277961us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:59.479312 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.479387 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.575959 (+ 96572us) raft_consensus.cc:1685] finished
0709 13:03:59.575984 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.576142 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":120490,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":97,"thread_start_us":368971,"threads_started":2,"wal-append.queue_time_us":352212}
I20260709 13:03:59.585095  8133 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:03:59.593148  8116 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.574143  7430 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.674990  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:03:59.674360  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:03:59.671862  8180 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.657668  8154 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.684238  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:03:59.669086  8189 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.637208  8089 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.591331  7880 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.573980  8203 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 13:03:59.669767  8153 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.666918  8071 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 13:03:59.640977  8087 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.666755  8100 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 13:03:59.686081  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.787309  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3148) took 1118 ms. Trace:
W20260709 13:03:59.686049  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.693689  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:03:59.679872  7939 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:03:59.676047  7836 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.783238  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3151) took 1107 ms. Trace:
I20260709 13:03:59.894250  1740 rpcz_store.cc:276] 0709 13:03:58.675484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.734671 (+ 59187us) service_pool.cc:224] Handling call
0709 13:03:58.735031 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:58.735262 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:03:59.044462 (+309200us) log.cc:844] Serialized 14 byte log entry
0709 13:03:59.781081 (+736619us) log.cc:844] Serialized 14 byte log entry
0709 13:03:59.781328 (+   247us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:03:59.781382 (+    54us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:03:59.781427 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.783033 (+  1606us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.783176 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":734323,"prepare.queue_time_us":1779763,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":92,"raft.queue_time_us":25163,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":7145,"thread_start_us":216142,"threads_started":1}
W20260709 13:03:59.685626  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.716871  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:03:59.897686  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.682786  8159 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:03:59.798346  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.784576  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:03:59.867115  1736 rpcz_store.cc:276] 0709 13:03:58.669060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.681353 (+ 12293us) service_pool.cc:224] Handling call
0709 13:03:58.681778 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.860357 (+178579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.860413 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.863232 (+  2819us) log.cc:844] Serialized 26 byte log entry
0709 13:03:59.165457 (+302225us) raft_consensus.cc:1645] Marking committed up to 5
0709 13:03:59.165538 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:59.786718 (+621180us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:03:59.786793 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.786869 (+    76us) raft_consensus.cc:1685] finished
0709 13:03:59.786890 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:03:59.787228 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2361,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":66161,"spinlock_wait_cycles":386356480,"thread_start_us":158331,"threads_started":1,"wal-append.queue_time_us":321680}
I20260709 13:03:59.749456  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:03:59.789639  8143 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.789s	user 0.009s	sys 0.005s
I20260709 13:03:59.682220  7416 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:03:59.693930  8178 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.783946  8193 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:03:59.924817  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:03:59.835860  8031 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.698199  8179 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:03:59.927462  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 142884us
W20260709 13:03:59.778795  8207 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 13:03:59.705667  8194 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:03:59.683917  8154 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:03:59.697960  8099 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.730261  7430 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.792577  8156 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.938114  8008 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:03:59.838086  7875 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.782003  7947 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:03:59.687207  8032 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.890128  7939 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.959308  7741 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.959173  8154 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:03:59.767792  8163 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.984879  7947 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:03:59.957288  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 269266us
I20260709 13:03:59.990696  7939 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:59.686784  8018 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.253s
I20260709 13:04:00.001003  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:52744 (request call id 16) took 1215 ms. Trace:
I20260709 13:04:00.001421  8018 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:00.001868  8018 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:03:59.697587  8070 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.001389  1665 rpcz_store.cc:276] 0709 13:03:58.795458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.054502 (+259044us) service_pool.cc:224] Handling call
0709 13:04:00.000783 (+946281us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54000a91d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfd7853 00007fcdf2fa560a 00007fcdec52b7d4 00007fcdec52b762 00007fcdec52b691 00007fcdec52b60c 00007fcdec52a8c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:00.000906 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013077248}
I20260709 13:03:59.729923  7979 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.236s
I20260709 13:03:59.940469  7836 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.953958  8178 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.009027  8178 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.009841  8076 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:03:59.706981  8160 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.011830  8085 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:00.013943  8160 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.014556  8122 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:00.015738  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2372) took 1999 ms (client timeout 2228 ms). Trace:
W20260709 13:04:00.016146  1664 rpcz_store.cc:269] 0709 13:03:58.016126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.051037 (+ 34911us) service_pool.cc:224] Handling call
0709 13:04:00.015525 (+1964488us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540013d8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:00.015673 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2722947456}
I20260709 13:04:00.009048  7836 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.984362  7875 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.019913  7836 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.020808  7875 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:00.022931  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 413596us
W20260709 13:04:00.024312  8210 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 13:03:59.965373  8179 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.025480  8179 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:03:59.697768  8107 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.030777  8111 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:03:59.937119  8143 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.900044  7880 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.033070  2311 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:00.034276  8143 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.034827  7881 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.031914  8107 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:03:59.970669  8194 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:03:59.994715  8084 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:03:59.997944  8075 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.041779  8194 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.041935  8084 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.994830  8212 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:03:59.992439  7741 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:03:59.968148  8159 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.045110  8084 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:03:59.987730  8008 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:00.045892  7741 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:03:59.996912  8168 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.059s	user 0.003s	sys 0.000s
I20260709 13:04:00.046872  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.046543  8008 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:00.007936  8205 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 13:03:59.955595  8031 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:03:59.969163  7430 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.036710  7979 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:03:59.990293  8032 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.007766  8018 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.044296  8212 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.049482  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3137) took 2160 ms. Trace:
I20260709 13:04:00.013967  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.053123  1741 rpcz_store.cc:276] 0709 13:03:57.888965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.337953 (+448988us) service_pool.cc:224] Handling call
0709 13:03:58.338380 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:03:58.341471 (+  3091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.341537 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:03:58.347075 (+  5538us) log.cc:844] Serialized 26 byte log entry
0709 13:03:58.887274 (+540199us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:03:58.887344 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:03:58.887472 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:03:59.996274 (+1108802us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:00.049065 (+ 52791us) raft_consensus.cc:1685] finished
0709 13:04:00.049097 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.049393 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":90104,"raft.queue_time_us":185335,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":35589,"spinlock_wait_cycles":67882752,"thread_start_us":476517,"threads_started":1,"wal-append.queue_time_us":421361}
I20260709 13:04:00.056876  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:00.059949  8160 consensus_queue.cc:1175] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ff44a4643e01482cb6b6a79b099fa766" responder_term: 17 status { last_received { term: 17 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 17 index: 12 } }
I20260709 13:03:59.943625  8193 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.052094  8018 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:00.045195  8159 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.052012  7430 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.051934  8032 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.050714  8031 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:00.098297  8208 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 13:04:00.098982  8106 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:00.099462  8106 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:00.066737  8209 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
I20260709 13:04:00.054179  7979 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:00.123935  7748 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.567s	user 0.013s	sys 0.000s
I20260709 13:04:00.124431  7748 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.124679  7979 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.100783  8106 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.126524  8106 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:00.081772  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.068552  7952 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:00.092523  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 953153us
I20260709 13:04:00.072007  8212 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.094485  7949 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:00.161784  7949 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:00.162829  7949 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.069106  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.107563  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:00.164443  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95391us
W20260709 13:04:00.164752  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57273.5us
I20260709 13:04:00.165217  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.165549  8189 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:00.165729  8157 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:00.166333  8157 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:00.166587  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.095631  8025 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.116034  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.175738  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.176704  1664 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:00.116957  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.117800  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.118600  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.119316  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.180419  1668 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:00.120098  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.123193  1665 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 13:04:00.123083  8154 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.126036  7979 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.080144  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3195) took 1393 ms. Trace:
I20260709 13:04:00.140579  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.145671  8159 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.146837  1660 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:00.145710  8193 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.149008  8179 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.158120  8087 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.163398  8089 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.111492  1666 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:00.189049  8025 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:00.187793  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.206605  1660 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:04:00.188313  1585 rpcz_store.cc:276] 0709 13:03:58.687288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.169418 (+482130us) service_pool.cc:224] Handling call
0709 13:03:59.182444 (+ 13026us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:03:59.514210 (+331766us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.079751 (+565541us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b540025c454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:00.080049 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":761000704}
I20260709 13:04:00.184880  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:00.192397  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51903.4us
I20260709 13:04:00.196200  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:00.209012  2312 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:00.209118  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:00.205211  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 84086us
I20260709 13:04:00.210642  1664 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:00.168429  8075 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.210568  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.213013  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.216984  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.213034  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.219269  2316 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:00.219844  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.173448  8160 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:00.187014  7833 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:00.223443  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.225152  1736 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:00.227862  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.228925  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.192447  8183 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.231312  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.225800  1556 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.188107  7741 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.205569  7952 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:00.209920  8031 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.209830  8064 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:00.185379  7880 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 13:04:00.253939  1664 raft_consensus.cc:2368] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:00.250981  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.255681  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.218408  7949 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.260646  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.267972  1741 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:04:00.257618  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88607.9us
I20260709 13:04:00.261845  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.271502  1742 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:00.228681  8192 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:00.233106  8122 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:00.167348  8157 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.214628  8189 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.218870  8032 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.266517  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.302498  1738 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:00.266541  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:00.263414  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.304405  1733 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:00.268384  1736 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:00.270852  1660 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:00.269156  8180 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:00.258347  8159 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.305760  8189 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.333801  8159 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.267099  8031 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.336109  8159 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.276711  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.337636  8212 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:00.338168  8031 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.261580  8182 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:00.338954  8182 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:00.339663  8182 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:00.258809  7952 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:00.277900  7640 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.341778  7952 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:00.341928  7640 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.273789  7833 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:00.302354  8122 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.346283  7833 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.313776  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:00.349858  8122 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.273126  7939 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.260545  8183 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.351358  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.352414  2314 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:00.352483  8183 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.352974  8180 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:00.259716  8178 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.353986  8180 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.355152  8178 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.321211  8032 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.357322  8032 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.357897  8032 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:00.358299  8032 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:00.358659  8032 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:00.314229  2316 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:04:00.361186  7947 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.322149  8179 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.234191  1739 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:00.363826  7947 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.272418  8163 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 13:04:00.352483  7939 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.365388  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.366222  8154 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.301968  8173 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 13:04:00.366611  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.367328  1660 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17.
I20260709 13:04:00.363770  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.369192  1739 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:00.370121  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.370724  1739 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:00.371547  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.372291  1739 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 19 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 13:04:00.370754  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 104255us
I20260709 13:04:00.336112  8189 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:00.338881  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62207.1us
I20260709 13:04:00.374372  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.377568  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.377843  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:00.375655  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53875.3us
W20260709 13:04:00.378779  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.380738  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.357962  7748 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:00.394124  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.394523  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.262466  7741 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.396898  1736 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 13:04:00.398890  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:00.401476  2315 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:00.398983  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.402628  2314 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:00.411974  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:00.398386  8178 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:00.413367  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.457233  1663 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:00.383097  7962 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:00.416010  2316 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8e75da90507436ea52296522f44c15e"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:00.457947  2316 raft_consensus.cc:606] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:00.458418  2316 raft_consensus.cc:614] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:00.417225  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.417847  8183 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.419301  8194 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.421825  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.422194  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.422566  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:00.422739  7949 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.423184  8087 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.424250  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.424738  2280 leader_election.cc:400] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17.
I20260709 13:04:00.505338  2280 leader_election.cc:403] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:04:00.397585  8106 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:04:00.425696  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52206us
W20260709 13:04:00.506223  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84082.5us
I20260709 13:04:00.507691  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.426204  7430 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:00.426618  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.517728  1737 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:00.518915  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.429447  2315 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:00.436185  1733 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:00.442085  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47853.6us
I20260709 13:04:00.516983  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.521106  1742 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:00.450407  1739 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:00.522619  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.459435  1738 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:00.447961  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.505403  1661 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:00.425282  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.524056  1668 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:00.509347  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:00.503104  8066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.222s	user 0.011s	sys 0.006s
I20260709 13:04:00.403738  8107 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:00.520248  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.506959  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.525918  1733 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
W20260709 13:04:00.442121  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61400.4us
I20260709 13:04:00.516100  1665 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:04:00.529214  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 106674us
W20260709 13:04:00.529098  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 129231us
W20260709 13:04:00.530030  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:00.530954  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83017.2us
I20260709 13:04:00.533272  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.534555  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:00.535699  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.544274  1705 leader_election.cc:400] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:00.545248  1705 leader_election.cc:403] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:00.493036  8100 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:00.493671  7748 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:00.564185  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.493261  8189 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.497473  8122 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.567728  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.568008  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.568944  2315 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:00.556726  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.557340  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.557898  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:00.560463  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.562124  7836 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:00.562928  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.516911  8203 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:00.537532  8066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.256s	user 0.011s	sys 0.006s
I20260709 13:04:00.556074  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.590535  8203 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.591202  1588 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:00.592514  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.593151  1588 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:00.573215  7836 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.577806  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.579559  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.602519  1585 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.580392  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.573211  8212 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:04:00.539223  8180 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.586170  8194 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.586783  8231 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:00.621732  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.501439  8183 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.594086  1737 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:00.595042  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.623606  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.624061  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.624680  1592 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.548593  7880 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:00.625277  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.625783  7880 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:00.552662  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:00.626468  7880 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:00.626832  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74326.2us
I20260709 13:04:00.543001  8143 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17.
I20260709 13:04:00.632565  8032 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.633865  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.633983  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.634955  1734 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:00.635478  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.636229  1733 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:00.594837  8031 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 13:04:00.599817  8173 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 13:04:00.642005  8085 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:00.642647  8085 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:00.643142  8085 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:00.564145  7748 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.645789  7748 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:00.565201  8163 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:00.651410  7836 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:00.527890  8087 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.654728  8087 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.548425  7881 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:00.658239  7881 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.658747  7881 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:00.550309  8179 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:00.526793  7869 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.118s
I20260709 13:04:00.603487  1586 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.528631  8156 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.537041  8154 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:00.616886  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.498050  7947 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:04:00.513377  7949 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.564697  8159 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:00.662941  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:00.603211  1738 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:00.681712  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:00.664659  1584 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.663527  8180 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.664291  8194 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 13:04:00.528967  8107 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:00.540699  8078 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:00.548231  8018 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:00.598244  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.668213  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.668345  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.644426  8182 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.593240  8203 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.675580  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:00.677929  1592 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:00.682888  1739 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:04:00.692615  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.606525  8189 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.748648  8180 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:00.726071  8212 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:00.749781  8212 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.750185  8212 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:00.750655  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1153) took 1180 ms. Trace:
I20260709 13:04:00.751434  8182 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.752616  1593 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:00.753809  1667 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:00.753619  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.754459  2309 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:00.723587  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.726338  8100 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:00.755265  1584 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:00.755565  8100 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:00.725699  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44054.3us
I20260709 13:04:00.624698  8183 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.751113  2311 rpcz_store.cc:276] 0709 13:03:59.570464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:59.573695 (+  3231us) service_pool.cc:224] Handling call
0709 13:04:00.750390 (+1176695us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:00.750556 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":901345664}
I20260709 13:04:00.762779  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.777089  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:00.778643  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.739530  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3184) took 3260 ms. Trace:
I20260709 13:04:00.756577  1742 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:00.783349  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.785641  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.771822  1733 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
W20260709 13:04:00.772672  2312 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.563s	user 0.009s	sys 0.000s
I20260709 13:04:00.757238  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.786669  1664 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:00.786726  2312 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:00.680788  7979 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.780388  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.784293  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.790761  1660 raft_consensus.cc:2368] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:00.780014  1587 rpcz_store.cc:276] 0709 13:03:57.478853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:58.198149 (+719296us) service_pool.cc:224] Handling call
0709 13:03:58.198534 (+   385us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:03:58.213159 (+ 14625us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:03:58.213193 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:03:58.214209 (+  1016us) log.cc:844] Serialized 52 byte log entry
0709 13:04:00.051210 (+1837001us) spinlock_profiling.cc:243] Waited 75.6 ms on lock 0x7b180045e7a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:00.130161 (+ 78951us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:00.130205 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.739083 (+608878us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b540025d354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:00.739146 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:00.739216 (+    70us) raft_consensus.cc:1685] finished
0709 13:04:00.739232 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:00.739454 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2382,"raft.queue_time_us":3044,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":7954,"spinlock_wait_cycles":201047552,"thread_start_us":78743,"threads_started":1,"wal-append.queue_time_us":2061308}
I20260709 13:04:00.792654  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.794183  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.795898  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.607033  8122 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:00.797711  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.803697  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:00.803925  1662 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:00.756011  8100 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.798204  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:00.674769  7741 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:00.802560  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 266792us
I20260709 13:04:00.783707  8018 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.804747  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:00.822208  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:00.682449  8154 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.743708  7952 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.761708  8025 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.741335  8076 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:00.692474  8064 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:00.810659  2316 raft_consensus.cc:2368] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
W20260709 13:04:00.801642  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 251559us
I20260709 13:04:00.819350  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:00.717530  7949 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:00.776089  8107 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.825690  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:00.799095  8180 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:00.796728  8203 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:00.791059  8182 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.833205  1589 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:00.834128  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.797122  8157 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:00.836753  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.838675  1738 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:00.854346  1591 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.857322  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.687747  8156 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.803385  8183 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:00.867869  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.827279  8154 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:00.837244  8173 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:00.835374  8111 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:00.896689  1593 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.895380  2317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:00.835183  8106 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:00.856782  8076 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:00.908901  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.909670  1593 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:00.909801  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.898772  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.810782  7946 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 13:04:00.834832  8031 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 13:04:00.854832  8067 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:00.801676  7979 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.901918  1706 leader_election.cc:400] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:00.931928  1706 leader_election.cc:403] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:04:00.903443  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.933106  1589 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:00.797178  7833 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 13:04:00.905366  2317 raft_consensus.cc:606] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:00.937114  2317 raft_consensus.cc:614] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:00.905844  1586 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 20 index: 16. (term mismatch)
I20260709 13:04:00.938326  1586 pending_rounds.cc:85] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 15
I20260709 13:04:00.938673  1586 pending_rounds.cc:107] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 19.16
I20260709 13:04:00.939043  1586 raft_consensus.cc:2889] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:00.875339  1736 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:00.901229  8157 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:00.874742  8186 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:00.827463  8078 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 13:04:00.915316  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:00.827509  8018 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:00.835613  7947 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 13:04:00.924646  1585 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:00.930090  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:00.866966  8025 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 13:04:00.907613  8111 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.859915  7430 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 13:04:00.943349  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.984481  1586 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:00.943929  8157 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:00.946949  8186 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:00.952709  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.905093  8183 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.962289  1737 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:00.935735  8067 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:00.963047  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 95550.8us
W20260709 13:04:00.978711  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:00.987306  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72082us
I20260709 13:04:00.998736  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.988152  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:00.991829  1733 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:01.001338  1584 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:01.001724  1737 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:01.001987  1586 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
W20260709 13:04:00.958096  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 127830us
I20260709 13:04:00.994099  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:00.944458  7949 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:00.910948  8106 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:00.926959  8182 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.008922  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.009604  1585 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:01.010171  8183 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.010735  8183 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.011166  8183 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.001199  8032 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.017935  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.018631  8156 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.018981  1742 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:01.024199  8227 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 13:04:01.026242  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.027061  8189 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:00.986557  8076 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:00.998507  8233 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.157s	user 0.000s	sys 0.004s
I20260709 13:04:01.003855  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:00.960745  7939 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:00.982723  8111 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:01.004266  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:01.027591  1592 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:01.032387  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.034137  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:00.995540  8186 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.037382  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.038404  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:00.995465  7979 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.050621  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.000172  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:01.064992  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 207536us
I20260709 13:04:01.068152  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.027658  8122 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.071048  1740 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:01.006563  8066 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:01.028710  8180 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:01.074522  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74414.9us
I20260709 13:04:01.001377  8193 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 13:04:01.083575  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.084194  1740 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:01.001247  8143 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 13:04:01.082986  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:01.066829  8099 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.091552  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.066335  8159 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:01.094720  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.098352  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.065948  7430 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:01.069826  8078 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:01.078125  8066 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.137444  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:01.081820  7979 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:01.077656  8173 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:01.163458  1586 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 13:04:01.154974  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:01.088115  8076 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.143071  8178 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:01.090525  8032 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:01.215137  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.075187  7741 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
W20260709 13:04:01.154038  7836 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.000s
I20260709 13:04:01.130440  8156 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:01.069875  8239 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.138s
I20260709 13:04:01.087881  8070 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.126s
I20260709 13:04:01.147380  8008 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:01.231314  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.143072  8122 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.083302  8212 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.130035  7880 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.161559  8066 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:01.086879  8180 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.146431  8099 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:01.263698  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125949us
I20260709 13:04:01.202195  7979 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:01.360098  1737 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:01.156237  7952 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.229444  7833 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:01.201625  8078 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.385301  1667 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:01.230661  8159 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.149s
I20260709 13:04:01.312183  8008 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.160516  7430 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:01.397624  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.249101  8032 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.391161  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:01.395498  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 240479us
I20260709 13:04:01.358816  8156 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:01.383838  7748 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.737s	user 0.013s	sys 0.002s
I20260709 13:04:01.254160  7836 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.405714  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:01.382917  7979 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.377899  8183 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.402366  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.413132  1740 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:01.422820  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.417598  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:01.423916  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:01.402498  7881 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.743s	user 0.014s	sys 0.003s
I20260709 13:04:01.430281  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:01.427413  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.398248  8008 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.431241  1586 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:01.397439  7833 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.435171  1740 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:04:01.436297  1741 raft_consensus.cc:2368] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19.
I20260709 13:04:01.441743  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.442965  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.452560  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.453871  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.451539  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:01.425233  8122 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:01.459571  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53920.5us
I20260709 13:04:01.419710  8235 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.437541  8183 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:01.465793  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:01.438086  8245 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.417476  8156 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.395761  8173 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.416023  7748 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:01.473414  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58306.7us
I20260709 13:04:01.477101  1740 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:01.484777  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:01.464813  7952 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:01.465528  8252 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:01.469316  8183 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:01.440670  7881 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:01.494751  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53068.9us
I20260709 13:04:01.475363  7869 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:01.493407  7952 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:01.503014  8254 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 13:04:01.508906  8243 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 13:04:01.513020  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.482306  7880 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.458246  7833 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.521669  8259 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:01.511442  8183 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:01.522929  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 58782us
W20260709 13:04:01.495834  8238 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.278s	user 0.001s	sys 0.001s
I20260709 13:04:01.469568  8235 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.527665  8262 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:01.529893  7640 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:01.531668  8255 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:01.496829  8252 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:01.522401  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 137853us
I20260709 13:04:01.553141  7945 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:01.514690  8241 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:01.518493  7952 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:01.495963  8025 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
W20260709 13:04:01.569537  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 84791.1us
I20260709 13:04:01.526160  7869 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:01.593183  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.483219  8245 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.595504  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.595628  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:01.562940  7949 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.001s
I20260709 13:04:01.610944  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.560192  8240 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.586153  7881 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.614652  8156 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.615064  7833 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:01.616540  8100 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.789s	user 0.007s	sys 0.004s
W20260709 13:04:01.619171  8186 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.538s	user 0.011s	sys 0.000s
I20260709 13:04:01.619663  8186 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.620908  1587 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:01.621486  8262 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:01.586141  7946 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.623235  7946 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:01.624815  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:01.626014  7869 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.617012  8100 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.585809  7836 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.609566  8099 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.586660  8078 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.628768  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:01.605026  8157 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.607s	user 0.008s	sys 0.004s
I20260709 13:04:01.628425  1586 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:01.630596  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.630867  8157 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.631402  1584 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:01.619212  8066 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.632262  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:01.633154  8157 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.589637  8180 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.559641  8250 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.633433  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.589037  8153 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.649982  8235 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.563112  8259 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:01.627224  8186 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.647342  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2454) took 1378 ms. Trace:
W20260709 13:04:01.606577  8107 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.001s
W20260709 13:04:01.655440  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.963s (SENT) [suppressed 7 similar messages]
I20260709 13:04:01.654474  1735 rpcz_store.cc:276] 0709 13:04:00.270627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.270768 (+   141us) service_pool.cc:224] Handling call
0709 13:04:00.271122 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:00.278513 (+  7391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:00.278546 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:00.279034 (+   488us) log.cc:844] Serialized 26 byte log entry
0709 13:04:00.680760 (+401726us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b1800256d20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:00.930195 (+249435us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:00.930245 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:00.930336 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:01.613108 (+682772us) raft_consensus.cc:1685] finished
0709 13:04:01.613146 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:01.647243 (+ 34097us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6946,"raft.queue_time_us":7069,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":5846,"spinlock_wait_cycles":42421632,"thread_start_us":249211,"threads_started":1,"wal-append.queue_time_us":803329}
W20260709 13:04:01.656733  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43882.6us
I20260709 13:04:01.562037  8212 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.650835  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:01.666494  1659 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:01.629902  8100 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.594794  8252 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.560560  8189 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 13:04:01.672490  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.681028  1584 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:01.684721  8066 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.684564  8253 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 13:04:01.685071  8267 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:01.686012  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75107.8us
I20260709 13:04:01.685760  8267 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.687656  8259 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:01.689757  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44756.6us
I20260709 13:04:01.690793  7430 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.692143  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:01.689893  8249 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:01.696691  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:01.646646  8178 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:04:01.665261  8111 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.596s	user 0.007s	sys 0.005s
I20260709 13:04:01.694072  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.634637  7949 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:01.650770  8078 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:01.664690  8032 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.714385  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.720309  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.721951  1584 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:01.722905  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.723047  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:01.726245  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:01.716864  8111 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.730822  8258 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:01.732625  8111 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:01.738700  8018 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.752s	user 0.012s	sys 0.000s
I20260709 13:04:01.739144  7952 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.739305  8018 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.739853  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.740864  2315 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 13:04:01.741132  2309 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 13:04:01.652700  8240 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.723517  7881 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:01.742319  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.744045  1663 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:01.745884  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.669595  8107 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.754038  1593 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:01.751780  1740 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:01.755846  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:01.749082  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.758060  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.759865  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.760613  1584 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:01.760802  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:01.761616  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:01.704437  8235 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:01.689303  8267 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.771136  8257 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.771845  8257 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.773864  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:01.774480  1663 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:01.774596  8260 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:01.774899  2318 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:04:01.775436  8261 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:01.775635  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:01.776032  8261 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.777771  1660 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:01.750311  7952 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:01.723779  8025 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 13:04:01.765470  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:01.780575  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:01.781718  7945 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.785769  8032 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:01.786285  8032 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:01.786688  8032 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:01.787163  8085 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.143s	user 0.012s	sys 0.004s
I20260709 13:04:01.787634  8085 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.789319  1666 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:01.792169  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1122) took 1100 ms. Trace:
I20260709 13:04:01.792798  1738 rpcz_store.cc:276] 0709 13:04:00.691988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.692261 (+   273us) service_pool.cc:224] Handling call
0709 13:04:01.791885 (+1099624us) spinlock_profiling.cc:243] Waited 72.5 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:01.792074 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166799616}
I20260709 13:04:01.725013  8165 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:01.766251  8249 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:01.778707  8261 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.779278  7748 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.765267  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:01.766921  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.716472  8178 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:01.802829  8154 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.893s	user 0.011s	sys 0.000s
I20260709 13:04:01.751571  8107 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.793972  8235 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.763552  8111 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:01.824767  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:01.804116  8245 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.724539  8078 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.805797  8165 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:01.833307  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.830252  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:01.850718  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 141370us
I20260709 13:04:01.850692  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:01.853008  1584 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:01.856700  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.859665  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.861231  2310 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 14: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:01.830376  8235 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.857591  2318 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:01.884829  8179 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:01.885550  8179 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:01.885897  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.865784  8165 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.812857  8173 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 13:04:01.858420  8260 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:01.871377  8226 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.876s	user 0.009s	sys 0.002s
I20260709 13:04:01.872120  8245 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:01.848490  8070 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:01.886487  8179 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.892974  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:01.897430  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:01.874102  8076 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.649s	user 0.014s	sys 0.000s
I20260709 13:04:01.827387  8154 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.906345  1738 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:04:01.907763  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:01.907357  1742 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:01.891273  8018 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.888273  8249 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:01.896341  8107 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.907631  8076 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:01.903846  8070 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:01.905457  7748 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:01.902940  8078 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:01.910261  8165 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:01.936834  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:01.952409  1738 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:01.934429  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:01.911834  8189 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.955312  8189 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.918346  8173 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:01.958091  8173 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.927434  8179 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:01.931145  8064 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:01.940303  8107 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:01.905882  8261 consensus_queue.cc:1175] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e6adef9da7da44f6af0af5695cf9b704" responder_term: 15 status { last_received { term: 13 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: [14.12-14.12]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:01.903050  8260 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:01.950851  8070 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:01.976293  8173 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.937841  8257 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:01.948896  7748 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:01.965641  8008 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.503s	user 0.009s	sys 0.004s
W20260709 13:04:01.995113  8273 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:04:01.917445  8178 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:01.966256  8064 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:01.991125  7979 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.562s	user 0.014s	sys 0.000s
I20260709 13:04:02.003316  7939 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:01.935622  8249 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.006289  8064 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.021574  7945 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:02.022428  7979 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.024717  7979 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:02.026574  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.028081  8154 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.028810  8154 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:02.029192  8122 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.003s
I20260709 13:04:02.029109  8018 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:02.029863  8018 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.032496  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.056092  1738 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:02.060637  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:02.061189  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 139873us
I20260709 13:04:02.033612  8189 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.058175  8122 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:02.069741  8018 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.005003  8008 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.029343  8154 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.103634  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:02.114442  2311 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:02.119504  1738 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 13:04:02.077338  7748 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:02.132484  8225 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.145s	user 0.005s	sys 0.006s
I20260709 13:04:02.150198  7962 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:02.150760  7962 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:02.151739  7962 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.152846  7962 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:02.133471  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:02.157248  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.134589  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3169) took 1117 ms. Trace:
I20260709 13:04:02.158066  1733 rpcz_store.cc:276] 0709 13:04:01.016803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.072457 (+ 55654us) service_pool.cc:224] Handling call
0709 13:04:01.072754 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.073090 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:01.073116 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:01.073814 (+   698us) log.cc:844] Serialized 26 byte log entry
0709 13:04:01.895896 (+822082us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b1800273d60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:02.000161 (+104265us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:02.000217 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.000407 (+   190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.134202 (+133795us) raft_consensus.cc:1685] finished
0709 13:04:02.134228 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.134505 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":611,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":5731,"spinlock_wait_cycles":7558528,"thread_start_us":103993,"threads_started":1,"wal-append.queue_time_us":1000903}
I20260709 13:04:02.160435  1733 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:02.135425  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1117) took 1619 ms (client timeout 1929 ms). Trace:
W20260709 13:04:02.161710  1739 rpcz_store.cc:269] 0709 13:04:00.515822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.522401 (+  6579us) service_pool.cc:224] Handling call
0709 13:04:02.135234 (+1612833us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.135351 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":306004352}
I20260709 13:04:02.163281  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2463) took 1850 ms. Trace:
I20260709 13:04:02.163763  1736 rpcz_store.cc:276] 0709 13:04:00.313208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.401861 (+ 88653us) service_pool.cc:224] Handling call
0709 13:04:00.402149 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:00.941848 (+539699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.163021 (+1221173us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b5400436054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:02.163210 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1087120256}
I20260709 13:04:02.152870  8064 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.170235  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3113) took 1382 ms. Trace:
I20260709 13:04:02.170696  1667 rpcz_store.cc:276] 0709 13:04:00.787690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.798977 (+ 11287us) service_pool.cc:224] Handling call
0709 13:04:00.820947 (+ 21970us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.386140 (+565193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.169960 (+783820us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:02.170156 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1287027072}
I20260709 13:04:02.172091  8064 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.173938  8106 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.176946  8085 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.179167  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3160) took 1165 ms. Trace:
I20260709 13:04:02.179558  1737 rpcz_store.cc:276] 0709 13:04:01.013791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.033838 (+ 20047us) service_pool.cc:224] Handling call
0709 13:04:02.178975 (+1145137us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54001090d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.179092 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331656960}
W20260709 13:04:02.181032  8183 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.609s	user 0.007s	sys 0.002s
I20260709 13:04:02.182359  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.183350  1585 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:04:02.184240  8008 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.184866  8107 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.190696  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3172) took 1148 ms. Trace:
I20260709 13:04:02.191181  1742 rpcz_store.cc:276] 0709 13:04:01.041971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.067419 (+ 25448us) service_pool.cc:224] Handling call
0709 13:04:02.190467 (+1123048us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b540012e8d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.190607 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1915534080}
I20260709 13:04:02.208449  8157 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.210973  1733 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:02.142673  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.146066  8032 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.223555  1589 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:02.225368  1586 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:02.226697  7741 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.225433  8008 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:02.227397  7741 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.227799  8008 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.231362  8008 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.240510  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:02.245008  8183 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.177006  8252 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.182569  8154 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.250901  8252 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.253769  8032 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.139369  2314 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.265460  8032 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.266872  8032 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.184288  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.272948  1585 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:02.274212  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.209184  8157 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.278837  7952 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.294775  7952 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.297891  7952 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.281276  7741 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.279457  1737 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:02.304422  1742 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:02.304288  8032 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.306229  8264 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.306694  8064 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:04:02.307935  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.309264  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.288268  1734 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:02.343818  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.344240  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3173) took 1275 ms. Trace:
I20260709 13:04:02.317816  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.344566  1734 rpcz_store.cc:276] 0709 13:04:01.069127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.070809 (+  1682us) service_pool.cc:224] Handling call
0709 13:04:02.133299 (+1062490us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8e9c 00007fcdeaeb8a5b 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:02.344170 (+210871us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6734336}
I20260709 13:04:02.345479  1585 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:02.287883  1736 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:02.319360  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.347299  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.354106  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.355175  1665 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:02.357659  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.313884  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:02.363570  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49725.4us
W20260709 13:04:02.324431  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83998.1us
I20260709 13:04:02.368383  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:02.369997  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:02.372023  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:02.326831  1589 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:02.375139  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.375903  1589 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.344588  1663 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.322583  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:02.392869  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:02.395756  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73233.8us
I20260709 13:04:02.397120  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3212) took 1101 ms. Trace:
I20260709 13:04:02.397756  1585 rpcz_store.cc:276] 0709 13:04:01.295455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.275294 (+979839us) service_pool.cc:224] Handling call
0709 13:04:02.397032 (+121738us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.332358  8178 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:02.404827  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.409547  1667 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.405397  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:02.404891  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.406994  1660 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.388535  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:02.411628  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.406893  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:02.411872  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.411937  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.412640  1661 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:02.412163  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:02.419672  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:02.341912  8064 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:02.425736  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.428212  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:02.434303  1661 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.434468  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.435480  1662 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:02.408595  8252 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:02.406235  8107 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.438238  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.414552  8216 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:02.440089  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.429097  2310 raft_consensus.cc:606] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:02.413427  8268 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:02.447223  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:02.346109  8264 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.416018  8183 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.448004  2316 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:02.458099  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.459578  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3209) took 1165 ms. Trace:
I20260709 13:04:02.460040  1585 rpcz_store.cc:276] 0709 13:04:01.294586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.442572 (+1147986us) service_pool.cc:224] Handling call
0709 13:04:02.459477 (+ 16905us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.436858  8085 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:02.462498  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.464721  1661 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:02.464972  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.466097  1662 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.468999  1664 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:02.472831  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53256.5us
I20260709 13:04:02.473986  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:02.495934  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3230) took 1939 ms (client timeout 1963 ms). Trace:
I20260709 13:04:02.473276  1668 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
W20260709 13:04:02.496418  1591 rpcz_store.cc:269] 0709 13:04:00.556981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.557108 (+   127us) service_pool.cc:224] Handling call
0709 13:04:02.495655 (+1938547us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400393fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.495858 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606235904}
I20260709 13:04:02.497649  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.479190  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3099) took 2291 ms. Trace:
I20260709 13:04:02.501320  8264 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.501681  1666 rpcz_store.cc:276] 0709 13:04:00.187713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.212088 (+ 24375us) service_pool.cc:224] Handling call
0709 13:04:00.213762 (+  1674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.789958 (+1576196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.478851 (+688893us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:02.479074 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3623842176}
I20260709 13:04:02.482472  8067 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.503681  1591 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:02.480819  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:02.501338  8249 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.507458  7939 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:02.508890  1705 leader_election.cc:400] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:02.452876  8194 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.510257  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1058) took 1601 ms. Trace:
I20260709 13:04:02.510007  8194 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.510733  1593 rpcz_store.cc:276] 0709 13:04:00.909166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.922032 (+ 12866us) service_pool.cc:224] Handling call
0709 13:04:00.922498 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.754543 (+832045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.509971 (+755428us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:02.510177 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1910912384}
I20260709 13:04:02.512467  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.483564  8183 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:02.513338  8183 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.445866  8216 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:02.493862  8236 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.550s	user 0.008s	sys 0.023s
W20260709 13:04:02.494383  8282 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 13:04:02.494938  8189 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:02.496106  1661 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 21 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:04:02.469092  8064 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.497269  8070 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:02.499408  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.447567  8107 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:02.477414  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3243) took 1665 ms (client timeout 2072 ms). Trace:
I20260709 13:04:02.417536  8157 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.512598  8194 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.521416  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:02.524645  2317 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:02.534578  1592 rpcz_store.cc:269] 0709 13:04:00.812587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.833866 (+ 21279us) service_pool.cc:224] Handling call
0709 13:04:02.477165 (+1643299us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540031c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.477322 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302084864}
I20260709 13:04:02.464282  8085 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.513470  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3224) took 1030 ms. Trace:
I20260709 13:04:02.516398  1589 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:02.532349  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.533327  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:02.522256  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3228) took 1966 ms. Trace:
W20260709 13:04:02.537076  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.498276  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:02.531953  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:02.536283  1591 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:02.538360  8251 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 13:04:02.603461  8251 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 13:04:02.603776  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.609074  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 87734.9us
W20260709 13:04:02.609359  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 111134us
I20260709 13:04:02.611294  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:02.610610  1708 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:02.614125  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:02.614786  2316 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.616048  1660 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:04:02.619282  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:02.508220  7939 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:02.513895  8183 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:02.523358  8064 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:02.622601  8235 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.724s	user 0.016s	sys 0.001s
I20260709 13:04:02.514297  8087 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:02.612466  8018 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.634388  7965 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election won for term 14
I20260709 13:04:02.521481  8189 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.526324  8070 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.603901  8251 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:02.610656  8085 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:02.638120  7952 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:02.624610  8157 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:02.615909  8122 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:02.515287  8216 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.617754  8252 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 13:04:02.649677  7965 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:02.650350  8087 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:02.642864  8235 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.655055  8157 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:02.666930  8157 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:02.664609  8216 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:02.664009  8251 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.674826  8251 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:02.680145  8107 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.688103  8235 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:02.748073  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.754259  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3206) took 1460 ms. Trace:
I20260709 13:04:02.758199  1585 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.758586  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:02.759300  1665 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.755132  1589 rpcz_store.cc:276] 0709 13:04:01.293716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.374834 (+1081118us) service_pool.cc:224] Handling call
0709 13:04:02.753871 (+379037us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.759059  1593 rpcz_store.cc:276] 0709 13:04:01.483056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.512102 (+1029046us) service_pool.cc:224] Handling call
0709 13:04:02.513392 (+  1290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.747603  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.749763  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.751534  1590 rpcz_store.cc:276] 0709 13:04:00.556190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.556330 (+   140us) service_pool.cc:224] Handling call
0709 13:04:00.556694 (+   364us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:00.630356 (+ 73662us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:00.630409 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:00.631267 (+   858us) log.cc:844] Serialized 52 byte log entry
0709 13:04:00.898176 (+266909us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:00.952276 (+ 54100us) log.cc:844] Serialized 14 byte log entry
0709 13:04:00.952656 (+   380us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.492873 (+1540217us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:02.494541 (+  1668us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b180042b140. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:04:02.494731 (+   190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.494837 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.521960 (+ 27123us) raft_consensus.cc:1685] finished
0709 13:04:02.521985 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.522165 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91392,"prepare.queue_time_us":486809,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":2763,"raft.queue_time_us":189182,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":6665,"spinlock_wait_cycles":191063424,"thread_start_us":642008,"threads_started":3,"wal-append.queue_time_us":312147}
I20260709 13:04:02.771284  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1132) took 1043 ms. Trace:
I20260709 13:04:02.790040  1592 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:02.771292  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1129) took 1111 ms. Trace:
I20260709 13:04:02.801898  7945 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:02.803249  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3258) took 1178 ms. Trace:
I20260709 13:04:02.806749  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3235) took 1061 ms. Trace:
I20260709 13:04:02.807310  1586 rpcz_store.cc:276] 0709 13:04:01.745223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.032203 (+286980us) service_pool.cc:224] Handling call
0709 13:04:02.806181 (+773978us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540025d354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.806476 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":441738624}
W20260709 13:04:02.810981  1587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1052) took 1974 ms (client timeout 1634 ms). Trace:
W20260709 13:04:02.811733  1587 rpcz_store.cc:269] 0709 13:04:00.836366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.836530 (+   164us) service_pool.cc:224] Handling call
0709 13:04:02.810320 (+1973790us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540025c954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.810646 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1665320192}
I20260709 13:04:02.795105  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3204) took 1502 ms. Trace:
I20260709 13:04:02.786854  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.814101  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.818765  1589 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:02.819705  8194 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:02.827260  8194 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.825922  8107 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:02.830318  8107 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.781085  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:02.831159  2316 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:02.832202  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51719.8us
W20260709 13:04:02.833207  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.792894  1735 rpcz_store.cc:276] 0709 13:04:01.659712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.659998 (+   286us) service_pool.cc:224] Handling call
0709 13:04:01.665299 (+  5301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:01.665808 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:02.028706 (+362898us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x7b1800318e80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:04:02.064372 (+ 35666us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.434682 (+370310us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.444207 (+  9525us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.444564 (+   357us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:02.444882 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.445173 (+   291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.445539 (+   366us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.770600 (+325061us) raft_consensus.cc:1685] finished
0709 13:04:02.770668 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.771036 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":437231,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":4671,"raft.run_cpu_time_us":391,"raft.run_wall_time_us":20786,"spinlock_wait_cycles":40738816,"thread_start_us":403310,"threads_started":2,"wal-append.queue_time_us":548341}
I20260709 13:04:02.834633  8216 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:02.835299  8032 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:02.836066  8216 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:02.836305  8032 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.803308  7945 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.838153  8216 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.841928  7880 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:02.841996  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:02.842638  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.843139  1666 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:02.843765  1661 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:02.844180  8107 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.804402  7965 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 12.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:02.838979  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 308342us
I20260709 13:04:02.846947  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.848384  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2401) took 1120 ms. Trace:
I20260709 13:04:02.849174  8179 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.857141  8179 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.815099  1592 rpcz_store.cc:276] 0709 13:04:01.292887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.749185 (+1456298us) service_pool.cc:224] Handling call
0709 13:04:02.795012 (+ 45827us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.859817  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.860612  8179 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.860672  1592 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:02.850209  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3228) took 1124 ms. Trace:
W20260709 13:04:02.842020  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 308484us
I20260709 13:04:02.861876  1591 rpcz_store.cc:276] 0709 13:04:01.691122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.746557 (+1055435us) service_pool.cc:224] Handling call
0709 13:04:02.815896 (+ 69339us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.863727  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.864954  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:02.854933  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 97152.4us
I20260709 13:04:02.855360  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.880465  8197 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:02.855966  8252 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:02.881441  8197 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:02.881798  8252 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:02.882370  8252 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:02.883208  8197 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.854332  2318 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:02.886637  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.887280  8087 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:02.888240  1590 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:02.888283  8087 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.827020  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3262) took 1032 ms. Trace:
I20260709 13:04:02.892057  1589 rpcz_store.cc:276] 0709 13:04:01.794186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.813637 (+1019451us) service_pool.cc:224] Handling call
0709 13:04:02.826902 (+ 13265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.892663  8087 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:02.895015  8183 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.897830  8183 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:02.866210  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:02.867143  8025 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:02.894462  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.873977  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.804067  1584 rpcz_store.cc:276] 0709 13:04:01.624349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.815249 (+190900us) service_pool.cc:224] Handling call
0709 13:04:02.802506 (+987257us) spinlock_profiling.cc:243] Waited 920 us on lock 0x7b5400430b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:02.802909 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2115328}
I20260709 13:04:02.899468  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.865274  1591 raft_consensus.cc:2368] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:02.848826  1659 rpcz_store.cc:276] 0709 13:04:01.727961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.728067 (+   106us) service_pool.cc:224] Handling call
0709 13:04:01.728325 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.760667 (+ 32342us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:04:01.793141 (+ 32474us) log.cc:844] Serialized 14 byte log entry
0709 13:04:01.927982 (+134841us) log.cc:844] Serialized 14 byte log entry
0709 13:04:02.022529 (+ 94547us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b1800280c00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:02.264873 (+242344us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b1800280c00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:04:02.268481 (+  3608us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:02.268537 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:02.268613 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.474940 (+206327us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b1800280c00. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:02.847831 (+372891us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:02.847926 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.848153 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":374691,"prepare.queue_time_us":34663,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":78,"raft.queue_time_us":343456,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":12392,"spinlock_wait_cycles":97735936,"thread_start_us":110972,"threads_started":1,"wal-append.queue_time_us":231726}
I20260709 13:04:02.809319  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.913693  1586 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:02.918736  8025 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.790251  1739 rpcz_store.cc:276] 0709 13:04:01.727779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.164931 (+437152us) service_pool.cc:224] Handling call
0709 13:04:02.165235 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:02.189869 (+ 24634us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:02.189904 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:02.200614 (+ 10710us) log.cc:844] Serialized 26 byte log entry
0709 13:04:02.637821 (+437207us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:02.637888 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:02.637987 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:02.770179 (+132192us) raft_consensus.cc:1685] finished
0709 13:04:02.770241 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:02.770870 (+   629us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":41299,"raft.run_cpu_time_us":629,"raft.run_wall_time_us":622,"spinlock_wait_cycles":15872,"thread_start_us":296092,"threads_started":1,"wal-append.queue_time_us":342490}
I20260709 13:04:02.939780  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:02.942648  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.948911  1587 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:04:02.953980  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3232) took 1261 ms. Trace:
I20260709 13:04:02.954499  1584 rpcz_store.cc:276] 0709 13:04:01.692105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.910252 (+1218147us) service_pool.cc:224] Handling call
0709 13:04:02.953883 (+ 43631us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.954967  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:02.957192  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:02.963514  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:02.964826  8179 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:02.968809  1585 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:02.970045  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3257) took 1401 ms. Trace:
I20260709 13:04:02.970609  1585 rpcz_store.cc:276] 0709 13:04:01.569114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.460974 (+891860us) service_pool.cc:224] Handling call
0709 13:04:02.969914 (+508940us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:02.972203  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:02.968019  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:02.973817  8179 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:02.974424  8179 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:02.975591  8179 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:02.977702  1591 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:02.983099  7946 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:02.985728  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:02.987433  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51179.1us
I20260709 13:04:02.990048  1666 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:02.984225  8085 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:02.965544  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.001482  2317 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:03.003471  1592 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:03.002625  8025 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:03.010092  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.023895  1587 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:03.024741  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3261) took 1276 ms. Trace:
I20260709 13:04:03.025871  8240 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:03.027762  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 162460us
I20260709 13:04:03.033607  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.034900  8261 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:03.035380  8261 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.035740  8261 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:03.037343  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3245) took 2160 ms. Trace:
I20260709 13:04:03.037820  1588 rpcz_store.cc:276] 0709 13:04:00.876331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.876587 (+   256us) service_pool.cc:224] Handling call
0709 13:04:00.877002 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:01.031249 (+154247us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:01.311563 (+280314us) log.cc:844] Serialized 14 byte log entry
0709 13:04:01.387348 (+ 75785us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:01.387409 (+    61us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:01.387467 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.036764 (+1649297us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54003b1fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:03.036850 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.037113 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":123587,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":91,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":11939,"spinlock_wait_cycles":355049600,"thread_start_us":322287,"threads_started":2,"wal-append.queue_time_us":160066,"wal-append.run_cpu_time_us":5724,"wal-append.run_wall_time_us":1130916}
I20260709 13:04:03.040652  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.017187  8203 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:03.043274  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3250) took 1582 ms. Trace:
I20260709 13:04:03.044641  1588 rpcz_store.cc:276] 0709 13:04:01.461471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.040211 (+1578740us) service_pool.cc:224] Handling call
0709 13:04:03.043205 (+  2994us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.021253  8189 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.047930  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.051112  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:03.054960  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 168016us
I20260709 13:04:03.044090  8203 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:03.025139  1587 rpcz_store.cc:276] 0709 13:04:01.748084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.813048 (+1064964us) service_pool.cc:224] Handling call
0709 13:04:03.024639 (+211591us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.058790  1591 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:03.060803  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53067us
I20260709 13:04:03.064069  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:03.066473  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.072641  7965 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.073454  7965 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.079335  2318 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:03.085635  8018 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:03.087328  1592 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:03.088331  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3236) took 1069 ms. Trace:
I20260709 13:04:03.089136  1592 rpcz_store.cc:276] 0709 13:04:02.018959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.048904 (+1029945us) service_pool.cc:224] Handling call
0709 13:04:03.088252 (+ 39348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.092923  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.093941  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.088315  8203 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.098658  8203 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:03.102624  8085 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:03.105404  1663 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.668s	user 0.014s	sys 0.007s
I20260709 13:04:03.106171  7430 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.106459  1663 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
W20260709 13:04:03.114475  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46339.5us
I20260709 13:04:03.115972  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:03.116999  8032 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.107126  7430 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.119992  7430 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.120736  7430 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:03.121623  7430 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.122049  1591 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:04:03.122262  7430 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.122887  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3218) took 1709 ms. Trace:
I20260709 13:04:03.123746  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.124991  7430 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.121439  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:03.123387  1591 rpcz_store.cc:276] 0709 13:04:01.413125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.067409 (+1654284us) service_pool.cc:224] Handling call
0709 13:04:03.121659 (+ 54250us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.122546 (+   887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.122787 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.129436  1593 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:03.142875  1587 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:03.126534  7965 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.153235  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3214) took 1745 ms. Trace:
I20260709 13:04:03.153807  1587 rpcz_store.cc:276] 0709 13:04:01.403708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.058670 (+1654962us) service_pool.cc:224] Handling call
0709 13:04:03.071627 (+ 12957us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.148855 (+ 77228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.149117 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:03.157909  7939 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.515s	user 0.004s	sys 0.012s
I20260709 13:04:03.158515  7939 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.160960  7939 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.146919  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.149271  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.168237  1586 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
W20260709 13:04:03.170563  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50241.6us
I20260709 13:04:03.171559  8122 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.172492  8182 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.172369  8122 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.173306  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:03.174289  2314 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
W20260709 13:04:03.179214  8303 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
W20260709 13:04:03.179425  8302 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.169s	user 0.002s	sys 0.001s
I20260709 13:04:03.182497  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3220) took 1768 ms. Trace:
I20260709 13:04:03.182978  1586 rpcz_store.cc:276] 0709 13:04:01.413709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.125074 (+1711365us) service_pool.cc:224] Handling call
0709 13:04:03.127623 (+  2549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.182134 (+ 54511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.182409 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.188763  8122 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.193773  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.195501  8259 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:03.196004  8259 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.196375  8259 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:03.196713  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:03.199267  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3162) took 2184 ms (client timeout 1503 ms). Trace:
W20260709 13:04:03.199806  1741 rpcz_store.cc:269] 0709 13:04:01.014655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:01.017602 (+  2947us) service_pool.cc:224] Handling call
0709 13:04:03.198721 (+2181119us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54003534d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:03.198989 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853508480}
I20260709 13:04:03.203193  8252 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:03.207209  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 137016us
I20260709 13:04:03.208503  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:03.209018  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3186) took 1496 ms. Trace:
I20260709 13:04:03.209241  2316 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:03.192355  8157 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.211005  8252 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.212239  8157 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.213629  2318 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:03.215322  1736 rpcz_store.cc:276] 0709 13:04:01.712379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.164901 (+452522us) service_pool.cc:224] Handling call
0709 13:04:02.165222 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:02.345985 (+180763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.208487 (+862502us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b5400338e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:03.208826 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281370112}
I20260709 13:04:03.220610  8182 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:03.220877  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:03.222932  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.228986  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3223) took 1814 ms. Trace:
I20260709 13:04:03.229473  1587 rpcz_store.cc:276] 0709 13:04:01.414568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.167401 (+1752833us) service_pool.cc:224] Handling call
0709 13:04:03.226227 (+ 58826us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.228462 (+  2235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.228500 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.228601 (+   101us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:03.228698 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.228790 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.228915 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:03.232877  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.233777  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.233896  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:03.234866  8179 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:03.234926  8257 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.235499  8179 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:03.236006  8179 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.235708  8257 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:03.238389  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 626672us
I20260709 13:04:03.238659  8257 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:03.239271  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:03.253979  8197 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:03.254142  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.254712  8197 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:03.255260  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.244850  8261 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.246670  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.255929  8197 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.257483  8261 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.253221  8032 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 13:04:03.257663  1666 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:03.258402  8032 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:03.258792  8032 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:03.259931  8032 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.260572  1557 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:03.261037  1557 leader_election.cc:403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:03.261241  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:03.232085  8157 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:03.261847  7946 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:03.257485  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.262365  8157 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.263491  1659 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:03.239846  8122 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:03.264783  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.265030  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.249032  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:03.253262  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.250413  2318 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:04:03.247828  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1063) took 1800 ms. Trace:
I20260709 13:04:03.266131  8157 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.266310  1739 raft_consensus.cc:2368] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19.
I20260709 13:04:03.267047  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.266151  2310 raft_consensus.cc:606] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:03.268913  2310 raft_consensus.cc:614] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:03.246512  8182 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:03.268353  1741 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:03.270759  7430 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:03.271539  7430 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.272804  7952 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:03.265874  1735 raft_consensus.cc:2376] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19. Re-sending same reply.
I20260709 13:04:03.232436  8203 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:03.268334  1736 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:03.280392  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.275836  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:03.266499  1586 rpcz_store.cc:276] 0709 13:04:01.447323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.219443 (+1772120us) service_pool.cc:224] Handling call
0709 13:04:03.247134 (+ 27691us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.247410 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.247439 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.247509 (+    70us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:03.247556 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.247615 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.247739 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.266537  1668 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:04:03.273348  7952 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:03.277814  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.270779  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:03.270781  7945 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.273775  8240 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:03.289616  8257 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:03.292282  7430 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.293237  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.274061  8032 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:03.297799  7945 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:03.298372  1668 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:04:03.301446  7945 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:03.307946  8085 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.308411  1708 leader_election.cc:400] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:03.308755  1708 leader_election.cc:403] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:03.309468  8122 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:03.309896  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.310781  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:03.312242  7945 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.315722  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.316593  1590 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:03.316461  7748 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:03.317129  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1065) took 1280 ms. Trace:
I20260709 13:04:03.318257  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.319123  8194 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.319370  8122 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
W20260709 13:04:03.320860  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.321622  7430 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:03.322399  7430 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.327023  7965 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:03.327210  8203 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.328318  8157 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:03.328758  8157 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:03.329859  8157 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.330615  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.331914  8157 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:03.332270  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.303161  8252 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:03.333546  8252 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:03.334179  8307 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:04:03.336074  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3215) took 1923 ms. Trace:
W20260709 13:04:03.336648  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.336910  7430 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.336448  1584 rpcz_store.cc:276] 0709 13:04:01.412246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.062184 (+1649938us) service_pool.cc:224] Handling call
0709 13:04:03.072446 (+ 10262us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.073065 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.073105 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.073942 (+   837us) log.cc:844] Serialized 26 byte log entry
0709 13:04:03.146053 (+ 72111us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:03.168989 (+ 22936us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.170621 (+  1632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.170707 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.335795 (+165088us) raft_consensus.cc:1685] finished
0709 13:04:03.335818 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.335979 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":710,"prepare.queue_time_us":46150,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":3914,"raft.run_cpu_time_us":488,"raft.run_wall_time_us":20321,"spinlock_wait_cycles":2432,"thread_start_us":71513,"threads_started":1,"wal-append.queue_time_us":156774}
I20260709 13:04:03.338647  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.339399  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.339597  1593 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.340147  1584 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:03.341812  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1062) took 1943 ms. Trace:
I20260709 13:04:03.342724  1741 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:03.343081  8260 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.341204  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.344290  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:03.345623  8122 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:03.345764  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:03.346657  8122 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:03.333039  1586 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:03.317493  1590 rpcz_store.cc:276] 0709 13:04:02.037049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.315346 (+1278297us) service_pool.cc:224] Handling call
0709 13:04:03.317059 (+  1713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.334097  8252 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.348570  8122 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.342217  1588 rpcz_store.cc:276] 0709 13:04:01.398625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.047368 (+1648743us) service_pool.cc:224] Handling call
0709 13:04:03.076394 (+ 29026us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.086171 (+  9777us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:03.128517 (+ 42346us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.166072 (+ 37555us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.225636 (+ 59564us) log.cc:844] Serialized 26 byte log entry
0709 13:04:03.229931 (+  4295us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:03.231025 (+  1094us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.232351 (+  1326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.232448 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.341582 (+109134us) raft_consensus.cc:1685] finished
0709 13:04:03.341606 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.341752 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1320,"prepare.queue_time_us":42272,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":93,"raft.queue_time_us":2280,"raft.run_cpu_time_us":485,"raft.run_wall_time_us":1792,"thread_start_us":77443,"threads_started":2,"wal-append.queue_time_us":103362}
I20260709 13:04:03.373926  7939 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:03.375020  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:03.376835  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:03.375121  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.377290  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.378680  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.379356  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.379431  1584 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:03.388504  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.381914  1589 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
W20260709 13:04:03.421574  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 60624.1us
W20260709 13:04:03.423915  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.362923  8203 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.421638  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:03.427551  1589 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:03.437933  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.438163  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.440343  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.442813  1735 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:03.437448  7430 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.446002  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.445693  7430 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.448062  7430 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.451534  8064 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 13:04:03.453320  8064 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:03.455817  8064 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:03.457094  8064 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.457830  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.458631  1739 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:03.440181  1590 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:03.469581  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 198928us
I20260709 13:04:03.423686  8260 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:03.441727  8018 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:03.479995  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3219) took 2066 ms. Trace:
I20260709 13:04:03.459652  7939 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:03.480409  1592 rpcz_store.cc:276] 0709 13:04:01.413417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.091190 (+1677773us) service_pool.cc:224] Handling call
0709 13:04:03.091687 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.099601 (+  7914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.099633 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.100185 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 13:04:03.149911 (+ 49726us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:03.478518 (+328607us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.479572 (+  1054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.479687 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.479740 (+    53us) raft_consensus.cc:1685] finished
0709 13:04:03.479762 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.479932 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17794,"prepare.queue_time_us":44795,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":485,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":94,"thread_start_us":332080,"threads_started":2,"wal-append.queue_time_us":118463}
I20260709 13:04:03.487064  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.489754  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.490969  1588 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:03.495916  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.479617  8064 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.498718  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.499990  1584 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:04:03.476858  7430 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.497759  1741 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:03.504034  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.510231  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.511701  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.513365  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.515755  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.518128  8031 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:03.518810  8031 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.519330  8031 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:03.524880  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.490309  8018 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:03.534701  1556 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:03.485702  8259 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.561831  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.565905  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:03.564064  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.579715  1735 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:03.564205  8212 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:03.594897  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.595717  1588 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:03.596714  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.596984  8261 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:03.597361  8261 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:03.597904  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.598323  8261 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.599440  8261 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:03.561944  8178 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:03.601853  8178 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:03.604503  7952 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:03.605655  8312 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.105s	user 0.002s	sys 0.000s
I20260709 13:04:03.608004  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3221) took 2194 ms. Trace:
I20260709 13:04:03.608353  1591 rpcz_store.cc:276] 0709 13:04:01.413992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.143168 (+1729176us) service_pool.cc:224] Handling call
0709 13:04:03.169345 (+ 26177us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:03.224511 (+ 55166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.224549 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:03.226889 (+  2340us) log.cc:844] Serialized 52 byte log entry
0709 13:04:03.352926 (+126037us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:03.385144 (+ 32218us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.398447 (+ 13303us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.398786 (+   339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.398892 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.607753 (+208861us) raft_consensus.cc:1685] finished
0709 13:04:03.607784 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.607939 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11430,"prepare.queue_time_us":35993,"raft.queue_time_us":1663,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":29709,"spinlock_wait_cycles":68965632,"thread_start_us":117518,"threads_started":1,"wal-append.queue_time_us":182175}
I20260709 13:04:03.611387  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3226) took 1920 ms. Trace:
I20260709 13:04:03.611752  1591 rpcz_store.cc:276] 0709 13:04:01.690596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.610386 (+1919790us) service_pool.cc:224] Handling call
0709 13:04:03.610796 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.611003 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.611029 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.611095 (+    66us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:03.611157 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.611208 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.611313 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.564831  8018 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.532492  7430 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.543655  8189 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:03.605103  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 109642us
W20260709 13:04:03.619961  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68118.6us
I20260709 13:04:03.621555  1591 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:03.629876  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 103491us
I20260709 13:04:03.634008  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.634522  1590 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:03.634970  1736 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:03.636099  1590 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:04:03.636827  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3229) took 1945 ms. Trace:
I20260709 13:04:03.562914  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.640689  1742 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.644170  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.645054  1737 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:03.646027  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.647274  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.647936  1737 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.651636  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.651830  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.652647  1741 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:03.653048  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.654531  1733 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 13:04:03.662398  8259 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.665706  1585 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
W20260709 13:04:03.616166  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 131) took 16305 ms (client timeout 15000 ms). Trace:
I20260709 13:04:03.556882  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:03.672391  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3227) took 1937 ms. Trace:
I20260709 13:04:03.672868  1591 rpcz_store.cc:276] 0709 13:04:01.690879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.612683 (+1921804us) service_pool.cc:224] Handling call
0709 13:04:03.613073 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.628332 (+ 15259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.628581 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.637183  1590 rpcz_store.cc:276] 0709 13:04:01.691353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.635364 (+1944011us) service_pool.cc:224] Handling call
0709 13:04:03.635827 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.636520 (+   693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.636745 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.669356  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:03.686262  1589 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:03.690462  1737 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:03.692544  7952 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.693392  8252 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.702350  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.701905  8179 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.701938  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.704399  8085 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:03.621641  8212 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:03.710981  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.682693  8182 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.715824  8197 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.717149  8189 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:03.718869  8165 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:03.719276  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.720629  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3225) took 2236 ms. Trace:
I20260709 13:04:03.722476  1741 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:03.619781  7430 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.726809  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3269) took 1413 ms. Trace:
I20260709 13:04:03.732517  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.761861  1738 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:03.764513  1737 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:03.765585  1585 rpcz_store.cc:276] 0709 13:04:02.313134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:02.971754 (+658620us) service_pool.cc:224] Handling call
0709 13:04:03.726610 (+754856us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b5400443c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:03.726737 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1538928512}
I20260709 13:04:03.742689  1742 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
W20260709 13:04:03.745647  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:03.750432  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.768455  1586 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:03.769858  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.770012  8240 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:03.773871  8179 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.775388  8234 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:03.775781  8252 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:03.776629  1584 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:03.750252  8212 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:03.831512  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3233) took 2134 ms. Trace:
I20260709 13:04:03.832005  1584 rpcz_store.cc:276] 0709 13:04:01.697267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.721223 (+2023956us) service_pool.cc:224] Handling call
0709 13:04:03.776309 (+ 55086us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.831092 (+ 54783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.831432 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.779331  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1066) took 1742 ms. Trace:
W20260709 13:04:03.785877  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98315.9us
I20260709 13:04:03.756067  8182 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.839524  1584 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:03.840355  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1072) took 1760 ms. Trace:
I20260709 13:04:03.840718  8182 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.841099  8251 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:03.842039  8251 consensus_queue.cc:1243] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:03.842566  8251 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:03.843007   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:04:03.844141  1585 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:03.844870  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1069) took 1776 ms. Trace:
I20260709 13:04:03.845202  1585 rpcz_store.cc:276] 0709 13:04:02.069078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.818891 (+1749813us) service_pool.cc:224] Handling call
0709 13:04:03.843952 (+ 25061us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.844593 (+   641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.844809 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.846663   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:03.846866  1585 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:03.847512  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3268) took 1534 ms. Trace:
I20260709 13:04:03.847856  1585 rpcz_store.cc:276] 0709 13:04:02.312813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.846134 (+1533321us) service_pool.cc:224] Handling call
0709 13:04:03.846519 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.847270 (+   751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.847461 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.848871  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:03.848608  1489 catalog_manager.cc:5697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:03.850313  1659 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:04:03.851635  1489 catalog_manager.cc:5697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 7, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:03.852098  8165 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.854573  8165 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:03.855341  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.848368  1665 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:03.840751  1584 rpcz_store.cc:276] 0709 13:04:02.080484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.838721 (+1758237us) service_pool.cc:224] Handling call
0709 13:04:03.839284 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.840073 (+   789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.840294 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.792522  7939 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:03.794076  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.794761  8107 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 13:04:03.762696  1587 rpcz_store.cc:276] 0709 13:04:01.483903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.230467 (+1746564us) service_pool.cc:224] Handling call
0709 13:04:03.262237 (+ 31770us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.296507 (+ 34270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.296549 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.299571 (+  3022us) log.cc:844] Serialized 26 byte log entry
0709 13:04:03.474556 (+174985us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:03.474617 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.474732 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:03.720381 (+245649us) raft_consensus.cc:1685] finished
0709 13:04:03.720403 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.720563 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23786,"raft.queue_time_us":23587,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"thread_start_us":174250,"threads_started":1,"wal-append.queue_time_us":403135}
I20260709 13:04:03.858853  1666 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:03.789343  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1068) took 1742 ms. Trace:
I20260709 13:04:03.859364  1590 rpcz_store.cc:276] 0709 13:04:02.046800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.787895 (+1741095us) service_pool.cc:224] Handling call
0709 13:04:03.788550 (+   655us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.788799 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.788841 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.788937 (+    96us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:03.789009 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.789092 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.789252 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.810000  1735 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:03.823460  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.830555  1733 raft_consensus.cc:2368] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15.
I20260709 13:04:03.830260  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.832914  1592 rpcz_store.cc:276] 0709 13:04:02.037303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.778279 (+1740976us) service_pool.cc:224] Handling call
0709 13:04:03.778701 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.778953 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.778982 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:03.779028 (+    46us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:03.779073 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.779131 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:03.779263 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.833914  1593 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:03.864068  1589 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [14.9-15.10]
W20260709 13:04:03.885555  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.849280  1585 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:03.890551  1741 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:03.847513  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.894675  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.894860  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:03.894886  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 460636us
I20260709 13:04:03.895480  1742 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:03.900492  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:03.900695  1586 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:03.901641  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1070) took 1829 ms. Trace:
I20260709 13:04:03.901973  1586 rpcz_store.cc:276] 0709 13:04:02.072025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.831725 (+1759700us) service_pool.cc:224] Handling call
0709 13:04:03.900503 (+ 68778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:03.901249 (+   746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.901467 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:03.905593  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:03.905783  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.906649  1735 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:03.907233  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:03.907572  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.907804  2316 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.908291  1735 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:03.911963  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.912951  1733 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260709 13:04:03.914611  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 120620us
I20260709 13:04:03.916744  7945 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:03.917223  7945 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:03.918920  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.918716  1737 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 13:04:03.917368  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:03.935273  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.936296  1739 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [18.11-19.12]
I20260709 13:04:03.898967  8178 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 13:04:03.938566  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:03.940472  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.941169  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.944608  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.945312  1733 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:03.800688  8197 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:03.946135  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3231) took 2246 ms. Trace:
I20260709 13:04:03.946015  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.946695  1589 rpcz_store.cc:276] 0709 13:04:01.691854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.691291 (+1999437us) service_pool.cc:224] Handling call
0709 13:04:03.863680 (+172389us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:03.888837 (+ 25157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:03.938511 (+ 49674us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54002d19d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:03.938672 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395062016}
I20260709 13:04:03.927269  7945 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:03.949110  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:03.951524  1737 raft_consensus.cc:2376] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:04:03.949903  1735 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:03.962893  1741 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:03.963805  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46496.2us
I20260709 13:04:03.943773  8178 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:03.972693  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:03.975425  1737 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:03.952708  7946 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:03.967577  1742 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:03.980631  1738 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:03.931274  8234 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:03.987598  1735 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 13:04:03.988777  1741 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 13:04:03.668077  1486 rpcz_store.cc:269] 0709 13:03:47.310306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:47.310611 (+   305us) service_pool.cc:224] Handling call
0709 13:04:03.615826 (+16305215us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:03:57.115370 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:03:57.116196 (+   826us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:03:57.116253 (+    57us) write_op.cc:435] Acquired schema lock
0709 13:03:57.116285 (+    32us) tablet.cc:662] Decoding operations
0709 13:03:57.120220 (+  3935us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:03:57.120358 (+   138us) write_op.cc:641] Partition lock acquired for write op
0709 13:03:57.120383 (+    25us) tablet.cc:685] Acquiring locks for 13 operations
0709 13:03:57.121774 (+  1391us) tablet.cc:701] Row locks acquired
0709 13:03:57.121830 (+    56us) write_op.cc:260] PREPARE: finished
0709 13:03:57.122039 (+   209us) write_op.cc:270] Start()
0709 13:03:57.122217 (+   178us) write_op.cc:276] Timestamp: P: 1783602237121960 usec, L: 0
0709 13:03:57.122256 (+    39us) op_driver.cc:348] REPLICATION: starting
0709 13:03:57.123071 (+   815us) log.cc:844] Serialized 5259 byte log entry
0709 13:03:57.839285 (+716214us) op_driver.cc:464] REPLICATION: finished
0709 13:03:57.976996 (+137711us) write_op.cc:301] APPLY: starting
0709 13:03:57.985587 (+  8591us) tablet.cc:1367] starting BulkCheckPresence
0709 13:03:58.024447 (+ 38860us) tablet.cc:1370] finished BulkCheckPresence
0709 13:03:58.024610 (+   163us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:03:58.330125 (+305515us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:03:58.330492 (+   367us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 13:03:58.330554 (+    62us) write_op.cc:312] APPLY: finished
0709 13:03:58.333483 (+  2929us) log.cc:844] Serialized 121 byte log entry
0709 13:03:58.333743 (+   260us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:58.351792 (+ 18049us) write_op.cc:454] Released schema lock
0709 13:03:58.381628 (+ 29836us) write_op.cc:341] FINISH: Updating metrics
0709 13:03:58.412958 (+ 31330us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:03:58.413010 (+    52us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4555898,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":109576,"apply.run_cpu_time_us":22878,"apply.run_wall_time_us":449927,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58493,"num_ops":13,"prepare.queue_time_us":91639,"prepare.run_cpu_time_us":26185,"prepare.run_wall_time_us":410620,"raft.queue_time_us":217867,"raft.run_cpu_time_us":12440,"raft.run_wall_time_us":29554,"replication_time_us":716827,"spinlock_wait_cycles":35712,"thread_start_us":593142,"threads_started":4,"wal-append.queue_time_us":494278,"wal-append.run_cpu_time_us":41168,"wal-append.run_wall_time_us":1852556}]]}
I20260709 13:04:04.028007  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:03.953105  7939 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:04:04.035418  1668 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:04.035679  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:04.036568  1738 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:04.037761  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55078.6us
I20260709 13:04:04.038641  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:04.039451  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 321007us
I20260709 13:04:04.040509  2316 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:04:04.027323  8031 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.507s	user 0.015s	sys 0.000s
I20260709 13:04:04.054128  1741 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:04:04.055616  8182 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:04.056205  8032 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:04.057013  8240 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.058037  1632 leader_election.cc:400] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19.
I20260709 13:04:04.144752  1632 leader_election.cc:403] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:04.057770  8165 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:04.062808  8257 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:04.065398  8317 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:04.069638  7946 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:04.070111  8216 raft_consensus.cc:993] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:04:04.071182  8197 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.079404  8064 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:04.080983  8318 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:04.088442  8234 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:04.089715  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.092221  1665 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:04.093454  1489 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 6 to 7, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 7 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:04.101631  8259 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:04.114203  8178 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:04.143083  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:04.147857  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 307837us
W20260709 13:04:04.152751  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 242134us
W20260709 13:04:04.160715  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 254805us
I20260709 13:04:04.168114  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:04.143994  8182 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:04.178213  1741 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:04:04.179380  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.152280  8317 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:04.182704  1742 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:04.186519  1733 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:04.199419  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:04.204079  8260 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.713s	user 0.012s	sys 0.000s
W20260709 13:04:04.204643  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.205773  8260 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:04.208038  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:04.208073  1585 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:04.170004  8216 raft_consensus.cc:1081] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Signalling peer 316f3bf18ed94e32b238130a6bfedf1a to start an election
I20260709 13:04:04.210664  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.211993  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:04.166761  8257 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:04.197247  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:04.147954  8031 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.166309  8032 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 13:04:04.221166  8178 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:04.232939  1590 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:04.235025  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.236131  1592 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:04.237867  1586 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:04:04.151988  8240 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.244683  1589 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:04.198624  7939 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.262081  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.196651  7946 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:04.262871  1587 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:04.266342  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:04.267098  1741 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:04.268064  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.181783  8197 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:04:04.286060  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.224159  8085 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:04.220450  8319 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.233659  8107 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.196851  8064 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:04.295598  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 117394us
I20260709 13:04:04.291496  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.301028  1584 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 12 index: 13. Preceding OpId from leader: term: 14 index: 14. (index mismatch)
I20260709 13:04:04.301478  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:04.306151  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1064) took 2269 ms. Trace:
I20260709 13:04:04.306723  1591 rpcz_store.cc:276] 0709 13:04:02.036739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.774283 (+1737544us) service_pool.cc:224] Handling call
0709 13:04:03.774832 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.832591 (+ 57759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.832648 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.865171 (+ 32523us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.146023 (+280852us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:04.146071 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.146291 (+   220us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.305790 (+159499us) raft_consensus.cc:1685] finished
0709 13:04:04.305823 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.306031 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25339,"raft.queue_time_us":49928,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":25846,"thread_start_us":194391,"threads_started":1,"wal-append.queue_time_us":364984}
I20260709 13:04:04.308233  1665 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:04.312390  1591 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
W20260709 13:04:04.295559  8322 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.063s	user 0.001s	sys 0.001s
I20260709 13:04:04.290791  8234 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:04.225483  8257 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:04.339110  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.293573  8018 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:04.261263  8261 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 13:04:04.267210  7939 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.422765  1733 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:04.417832  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:04.349996  8329 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:04.314926  8319 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.333792  8107 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.295111  8182 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.375902  8259 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
W20260709 13:04:04.453352  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 107515us
I20260709 13:04:04.377264  8216 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.096s
W20260709 13:04:04.456851  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 140272us
W20260709 13:04:04.453974  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 229504us
I20260709 13:04:04.458643  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:04.458393  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:04.442937  8343 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.458374  8067 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.128s
W20260709 13:04:04.466655  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 408568us
I20260709 13:04:04.460695  1733 raft_consensus.cc:2376] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16. Re-sending same reply.
W20260709 13:04:04.454111  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 219080us
I20260709 13:04:04.454847  8197 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:04.322131  8317 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.14, Last appended by leader: 14, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.474169  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:04.454414  8329 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.484272  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:04.415571  8018 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.486389  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.490007  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:04.411475  7946 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.490453  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.492223  1668 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:04.478350  8085 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:04.475979  8197 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:04.455482  8335 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 13:04:04.500890  8319 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.473522  8343 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.501770  8064 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.502902  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.538825  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.503115  8018 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:04.540238  8018 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:04.540457  8320 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:04.540709  8018 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.540987  8320 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:04.541139  8018 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:04.503866  8257 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.504343  8330 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:04.504894  7939 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:04.505985  8031 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.498260  8182 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:04.486162  8260 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 13:04:04.507990  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:04.502900  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3255) took 3018 ms. Trace:
I20260709 13:04:04.599131  7939 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:04.599815  7939 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:04.598019  1588 rpcz_store.cc:276] 0709 13:04:01.484166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.598837 (+2114671us) service_pool.cc:224] Handling call
0709 13:04:03.599240 (+   403us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:03.601139 (+  1899us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:03.601546 (+   407us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.974717 (+373171us) log.cc:844] Serialized 14 byte log entry
0709 13:04:03.974949 (+   232us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:04.027233 (+ 52284us) log.cc:844] Serialized 52 byte log entry
0709 13:04:04.027452 (+   219us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:04.027716 (+   264us) log.cc:844] Serialized 14 byte log entry
0709 13:04:04.027940 (+   224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.028021 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.135632 (+107611us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b180013bbe0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdf1eb158f 00007fcdeaef0571 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd
0709 13:04:04.502665 (+367033us) raft_consensus.cc:1685] finished
0709 13:04:04.502692 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.502833 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32620,"prepare.queue_time_us":102816,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":2307,"raft.queue_time_us":5570,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":6087,"spinlock_wait_cycles":67598208,"thread_start_us":372505,"threads_started":1,"wal-append.queue_time_us":629292}
I20260709 13:04:04.509886  8216 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 13:04:04.526098  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47071.5us
I20260709 13:04:04.601004  7939 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.603474  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:04.526935  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.527387  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.606242  8260 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:04.606673  8260 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:04.607010  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.605489  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:04.528326  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.528613  8197 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.526878  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.527933  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.497225  8240 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:04.524375  8122 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:04.563737  8107 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.608916  1667 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:04.620606  1668 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:04.568630  8212 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.670s	user 0.006s	sys 0.005s
I20260709 13:04:04.570145  8320 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.571750  8338 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:04.572507  8329 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:04.573611  8194 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:04.650718  8194 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:04.652160  8194 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.573984  8189 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.653985  8189 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.656932  8189 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:04.574906  8339 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:04.658066  1664 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:04.659116  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.660791  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.660981  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.661592  8339 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.662079  8339 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:04.576900  8343 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.536852  8179 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 13:04:04.584332  8323 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.240s	user 0.002s	sys 0.001s
W20260709 13:04:04.585095  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.586028  8317 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:04.508003  8327 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:04:04.542912  8261 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:04.588838  8085 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.589627  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:04.667133  8194 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:04.672441  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:04.674682  8107 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:04.675346  8107 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:04.677698  7748 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:04.677922  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116332us
I20260709 13:04:04.678318  7748 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.678947  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:04.680603  7748 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.624872  8122 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.682538  8122 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:04.683094  8332 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:04.685055  8067 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:04.685760  8067 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:04.686957  8067 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.688230  8197 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:04.607810  1663 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:04:04.679577  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.527901  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.608549  1660 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
W20260709 13:04:04.629062  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90272.3us
I20260709 13:04:04.630393  8338 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:04.644848  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.648073  8018 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.648597  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.607057  8260 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:04.691563  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:04.689819  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 84370.3us
I20260709 13:04:04.692803  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.697021  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.667029  8320 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:04.531373  8252 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:04.628881  7945 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.656s	user 0.007s	sys 0.007s
I20260709 13:04:04.681219  7748 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:04.689459  8067 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:04.697664  1668 raft_consensus.cc:2368] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:04.710992  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.717444  8107 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:04.630610  8240 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:04.726563  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.718811  8018 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:04.719012  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.696795  8348 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:04.697007  8261 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:04.738444  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 272052us
W20260709 13:04:04.738729  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.749890  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.750533  1660 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:04.752622  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.757279  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:04.759289  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.757370  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:04.762393  8351 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:04.766105  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:04.766458  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.735127  8261 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.736526  8348 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:04.714905  8252 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.769199  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:04.630754  8212 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.778899  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:04.782133  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:04.746053  7946 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:04.725009  7945 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:04.786919  2310 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 13:04:04.787776  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.773793  8261 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:04.769959  8331 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:04:04.793777  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.826726  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.775480  8018 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:04.832702  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44665.8us
I20260709 13:04:04.829082  1584 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:04.777401  8348 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:04.770653  8351 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:04.846747  1665 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:04.855182  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65212.3us
I20260709 13:04:04.860985  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.862053  1666 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:04.832656  8240 raft_consensus.cc:2764] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 13:04:04.875448  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 109101us
I20260709 13:04:04.837523  7430 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:04.841524  8018 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:04.877862  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:04.842001  8343 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:04.837069  7748 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:04.751525  8338 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:04.893294  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111232us
I20260709 13:04:04.903785  1660 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:04.914183  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.900241   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:04.869724  8353 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:04.837697  7946 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:04.890545  8234 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.543s	user 0.009s	sys 0.004s
W20260709 13:04:04.926046  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 147187us
I20260709 13:04:04.892398  8018 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:04.929291  8337 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:04:04.929728  8357 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:04.930097  8361 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:04.935201  7946 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:04.979911  1589 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: [14.9-15.10]
I20260709 13:04:04.938242  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.940951  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2471) took 1299 ms. Trace:
I20260709 13:04:04.945993  1592 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db"
dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:46322
I20260709 13:04:04.981930  1592 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:04.982376  1592 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:04.946866  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.947718  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:04.948869  1585 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 13:04:04.950995  8354 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
W20260709 13:04:04.945161  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:04.881577  7430 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:04.892637  8343 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:04.974736  2281 leader_election.cc:400] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15.
W20260709 13:04:04.977603  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75019.3us
I20260709 13:04:04.999001  2281 leader_election.cc:403] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 13:04:04.999435  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:04.923990  8353 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:04.981662  1734 rpcz_store.cc:276] 0709 13:04:03.642086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.648542 (+  6456us) service_pool.cc:224] Handling call
0709 13:04:03.648964 (+   422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:03.649563 (+   599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:03.649594 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:03.650418 (+   824us) log.cc:844] Serialized 26 byte log entry
0709 13:04:04.037409 (+386991us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:04.037456 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:04.038945 (+  1489us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:04.939936 (+900991us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b24000c36f8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2efb0 00007fcdedba5488 00007fcdedba5be8 00007fcdedb7d398 00007fcdedb7876a 00007fcdedb77bb6
0709 13:04:04.940583 (+   647us) raft_consensus.cc:1685] finished
0709 13:04:04.940613 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:04.940874 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1606,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":116,"spinlock_wait_cycles":8579968,"thread_start_us":386479,"threads_started":1,"wal-append.queue_time_us":679945}
I20260709 13:04:04.972460  8234 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19.
I20260709 13:04:05.007278  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:04.998297  1591 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:05.040688  1660 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.001986  8338 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.000624  8343 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.006564  8353 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:05.005053  7946 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.004536  8064 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 13:04:04.983875  8357 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:05.094347  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 123144us
I20260709 13:04:05.091418  7430 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.100153  7430 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.091970  8365 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.101928  8212 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.102052  8365 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:05.102679  8342 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.103292  8342 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.061384  7748 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.104055  8064 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:05.104506  8064 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:05.105314  8178 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:05.105437  8064 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.106848  8064 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:05.106947  8178 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:05.067607  8122 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:05.110762  8360 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
W20260709 13:04:05.113435  8368 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 13:04:05.069859  8367 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 13:04:05.102494  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.117272  1659 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:05.115119  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.131528  1592 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:05.145380  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98602.8us
I20260709 13:04:05.090054  8252 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:05.179877  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.204886  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:05.170145  7945 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.170025  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.178021  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.211030  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.211635  1665 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:05.179106  8122 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:05.218403  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 218979us
W20260709 13:04:05.183955  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.185103  2316 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 14. Preceding OpId from leader: term: 14 index: 14. (term mismatch)
I20260709 13:04:05.221397  2316 pending_rounds.cc:85] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 13
W20260709 13:04:05.185526  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.186721  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.207329  1661 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:05.207650  1659 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:05.225914  1662 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:05.209095  1584 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:05.179838  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:05.227038  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47276.4us
W20260709 13:04:05.197435  1557 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.679s (SENT)
I20260709 13:04:05.228121  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:05.229391  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.232580  1592 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:05.203637  8339 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.541s	user 0.010s	sys 0.004s
I20260709 13:04:05.005584  8351 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.242748  1708 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:05.157070  8367 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:05.221886  2316 pending_rounds.cc:107] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 13.14
I20260709 13:04:05.257936  2316 raft_consensus.cc:2889] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:05.260099  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.252080  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:05.266732  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48943.6us
I20260709 13:04:05.233875  8252 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.270823  1665 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:05.236359  8349 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:05.290388  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:05.274693  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:05.291934  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.270376  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:05.140125  7946 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:05.243912  8340 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:05.324550  8318 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.325214  8318 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:05.326094  8342 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.245574  8178 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:05.292701  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:05.296232  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.261471  8361 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:05.202095  8338 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 13:04:05.297071  1741 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:05.243489  8356 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 13:04:05.303483  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.304252  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.304309  7748 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:05.341741  1740 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:05.306576  8332 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 13:04:05.313004  8320 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.000s
I20260709 13:04:05.313625  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:05.318131  8182 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.726s	user 0.007s	sys 0.011s
W20260709 13:04:05.321084  8197 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.632s	user 0.004s	sys 0.013s
I20260709 13:04:05.333128  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.302525  8378 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:05.339510  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:05.344449  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:05.305825  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 108382us
I20260709 13:04:05.296940  8358 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 14. Current term is 15.
I20260709 13:04:05.262861  8339 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.365937  1733 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:05.334678  8178 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:05.411080  1664 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
W20260709 13:04:05.377310  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63813.3us
W20260709 13:04:05.408676  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 142044us
I20260709 13:04:05.300894  8251 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
W20260709 13:04:05.407584  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74090us
I20260709 13:04:05.412673  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.424566  1663 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.413602  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:05.360469  8261 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.001s
W20260709 13:04:05.418977  1557 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.716s (SENT)
I20260709 13:04:05.426044  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.427165  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:05.427356  1740 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:04:05.428663  1665 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.299989  8252 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:05.433266  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.360054  8318 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.426220  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:05.441920  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:05.362246  8365 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:05.376813  7748 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:05.377573  8342 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:05.425271  8178 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:05.499713  8379 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 13:04:05.499395  8361 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.498538  8251 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.463973  8067 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:05.417829  8212 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:05.417830  8380 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 13:04:05.463150  7946 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:05.511503  8260 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.790s	user 0.013s	sys 0.008s
I20260709 13:04:05.465991  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:05.532943  1584 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:05.469985  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:05.471107  8342 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:05.470677  8122 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:05.419142  8372 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.063s
I20260709 13:04:05.518808  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:05.464834  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.444257  8261 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.507153  8064 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:05.422000  8367 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:05.507500  8318 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:05.545061  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:05.535444  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93604.5us
W20260709 13:04:05.535777  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 116783us
I20260709 13:04:05.600121  1589 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:05.506526  8251 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:05.521435  8386 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 13:04:05.590229  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:05.504436  8358 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 13:04:05.530968  8212 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:05.610103  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:05.614001  1665 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:05.613204  1662 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:05.640451  1660 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.592s	user 0.015s	sys 0.000s
I20260709 13:04:05.644883  1660 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:05.646682  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.656934  1587 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:05.658574  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:05.524425  8387 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:05.563513  8342 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:05.665956  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:05.667098  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 249553us
I20260709 13:04:05.667351  1737 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 13:04:05.686276  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 327502us
I20260709 13:04:05.683187  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:05.668615  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67958.2us
W20260709 13:04:05.610705  8018 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.635s	user 0.000s	sys 0.019s
I20260709 13:04:05.682075  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:05.683034  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.692409  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:05.615145  8318 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:05.695200  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 269866us
I20260709 13:04:05.696288  1666 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:05.690183  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.702072  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:05.535477  8260 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.704146  1736 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:05.710551  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:05.709335  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:05.710794  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.716858  1663 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:05.665961  8064 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:05.719614  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:05.694533  8122 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:05.755158  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:05.723752  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.707875  8018 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.690830  8387 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:06.004249  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 306411us
I20260709 13:04:06.004351  1733 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:05.816051  8357 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.705243  8340 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:06.005926  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:05.817647  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:05.830595  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:05.790428  2311 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:05.761659  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:06.006315  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:06.020149  1663 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:05.764051  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.010076  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.763429  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:05.778249  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.756825  1737 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:06.021740  1661 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.006057  8212 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.005313  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.019274  2310 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:04:05.803715  8371 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 13:04:05.987964  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.023016  1734 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.018508  1733 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
W20260709 13:04:05.935549  8381 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
W20260709 13:04:05.803431  8394 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
W20260709 13:04:05.829830  8353 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.712s	user 0.012s	sys 0.000s
W20260709 13:04:05.934597  8393 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260709 13:04:05.779814  8386 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 13:04:05.807261  8348 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.934s	user 0.012s	sys 0.000s
W20260709 13:04:05.933243  8373 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.483s	user 0.000s	sys 0.002s
I20260709 13:04:06.020802  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3141) took 1085 ms. Trace:
W20260709 13:04:05.933790  8383 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.156s	user 0.001s	sys 0.001s
W20260709 13:04:05.922948  7430 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.822s	user 0.012s	sys 0.003s
I20260709 13:04:06.035491  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.035450  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:06.042292  8402 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:04:05.984395  8064 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.042902  1660 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:06.043191  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:06.043901  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:05.933516  8397 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.044056  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:06.044718  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:06.048381  1738 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:05.847021  8318 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.002547  7939 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.018126  8260 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.018992  8252 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:05.859962  8122 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:05.998832  8251 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:05.755550  7748 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:05.802165  8389 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.355s
I20260709 13:04:05.811707  8367 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.180572  1665 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:04:06.217206  1739 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:06.171749  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.217280  1737 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:04:06.098332  1584 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.548s	user 0.012s	sys 0.001s
I20260709 13:04:06.218783  1589 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:06.065863  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:06.220371  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 215076us
I20260709 13:04:06.058259  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.094496  1659 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:06.078341  1734 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:06.219229  2314 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:06.217381  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3136) took 1283 ms. Trace:
I20260709 13:04:06.165408  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.232467  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1175) took 1854 ms. Trace:
I20260709 13:04:06.164113  8339 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.232996  2317 rpcz_store.cc:276] 0709 13:04:04.378471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.378610 (+   139us) service_pool.cc:224] Handling call
0709 13:04:04.411203 (+ 32593us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.456427 (+ 45224us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:04.734276 (+277849us) log.cc:844] Serialized 14 byte log entry
0709 13:04:05.067519 (+333243us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.081679 (+ 14160us) log.cc:844] Serialized 26 byte log entry
0709 13:04:05.081966 (+   287us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:05.082050 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.082170 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.232236 (+1150066us) raft_consensus.cc:1685] finished
0709 13:04:06.232260 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.232402 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20610,"prepare.queue_time_us":332123,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":29012,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":8834,"thread_start_us":536371,"threads_started":2,"wal-append.queue_time_us":472946}
I20260709 13:04:06.235026  7939 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.232983  8397 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.245347  8397 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.220091  1584 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:06.222831  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.280503  8212 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.229959  1660 rpcz_store.cc:276] 0709 13:04:04.933946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.042275 (+1108329us) service_pool.cc:224] Handling call
0709 13:04:06.042658 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.217046 (+174388us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.217312 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:06.180883  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.245663  8398 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.164273  8365 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:06.301681  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78906.7us
W20260709 13:04:06.301836  8406 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.317s	user 0.004s	sys 0.000s
I20260709 13:04:06.246464  8338 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:06.302626  8338 consensus_queue.cc:1243] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:06.303028  8397 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:06.222640  8349 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:06.249609  8343 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.163s	user 0.017s	sys 0.002s
W20260709 13:04:06.253180  8342 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.000s
I20260709 13:04:06.255795  8356 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.083052  8261 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.257326  8390 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:06.257813  8386 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 13:04:06.257169  1736 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:06.260205  8378 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.263268  8179 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:06.263469  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:06.133605  8371 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:04:06.268433  1740 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.269284  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.269644  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3137) took 1284 ms. Trace:
I20260709 13:04:06.271562  7430 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.221464  1733 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:06.274535  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3135) took 1394 ms (client timeout 1547 ms). Trace:
W20260709 13:04:06.362165  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98773.3us
W20260709 13:04:06.275379  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2451) took 1583 ms (client timeout 1700 ms). Trace:
I20260709 13:04:06.275579  8387 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.098630  8348 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.078495  8407 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 13:04:06.056074  8353 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.278518  8399 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:06.278699  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.373755  1734 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:04:06.079511  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1109) took 1331 ms. Trace:
I20260709 13:04:06.285919  1587 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.219240  1663 rpcz_store.cc:276] 0709 13:04:04.935277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.761160 (+825883us) service_pool.cc:224] Handling call
0709 13:04:06.020734 (+259574us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.223429  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.300029  1661 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:06.300987  8398 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.246899  1667 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:06.303690  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1149) took 1025 ms. Trace:
W20260709 13:04:06.342278  8380 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.005s	sys 0.022s
I20260709 13:04:06.305238  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:06.381348  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76174us
I20260709 13:04:06.374310  1668 rpcz_store.cc:276] 0709 13:04:04.748114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.748256 (+   142us) service_pool.cc:224] Handling call
0709 13:04:04.766287 (+ 18031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:04.766685 (+   398us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:04.766712 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:04.794786 (+ 28074us) log.cc:844] Serialized 501 byte log entry
0709 13:04:05.338453 (+543667us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:05.338505 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:05.338601 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.079233 (+740632us) raft_consensus.cc:1685] finished
0709 13:04:06.079259 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.079436 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":467748,"threads_started":1,"wal-append.queue_time_us":804896}
W20260709 13:04:06.306758  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.307251  1739 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
W20260709 13:04:06.171687  8178 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.664s	user 0.010s	sys 0.003s
I20260709 13:04:06.310799  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1116) took 1361 ms. Trace:
I20260709 13:04:06.387144  1660 rpcz_store.cc:276] 0709 13:04:04.949850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.309603 (+1359753us) service_pool.cc:224] Handling call
0709 13:04:06.310040 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.310324 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.310363 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.310423 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:06.310488 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.310571 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.310722 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.392565  8338 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:06.393182  8361 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:06.322333  8349 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.393766  8361 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:06.394881  8361 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.395855  8349 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:06.306159  8212 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.398270  8252 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:06.321592  1589 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:06.257193  8085 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.359426  8182 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:06.445943  8182 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:06.400260  8380 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.006s	sys 0.023s
I20260709 13:04:06.448333  8182 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.326651  8343 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.372351  7430 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.370620  8400 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:06.450726  8343 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.451031  8400 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.451471  8400 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:06.366954  8378 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.357602  1665 rpcz_store.cc:276] 0709 13:04:04.934231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.148915 (+1214684us) service_pool.cc:224] Handling call
0709 13:04:06.180212 (+ 31297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.217959 (+ 37747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.218356 (+   397us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.456415  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2455) took 1479 ms. Trace:
I20260709 13:04:06.260205  8122 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.466076  8122 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.357613  1738 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:06.363503  1664 rpcz_store.cc:269] 0709 13:04:04.692116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.692240 (+   124us) service_pool.cc:224] Handling call
0709 13:04:06.275219 (+1582979us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:06.275323 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1580890880}
W20260709 13:04:06.362593  1666 rpcz_store.cc:269] 0709 13:04:04.879758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.879930 (+   172us) service_pool.cc:224] Handling call
0709 13:04:06.274328 (+1394398us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:06.274438 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1056430464}
I20260709 13:04:06.470585  1664 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:06.330744  8391 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:06.471447  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.471307  8391 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.331362  8339 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.472498  8339 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:06.280122  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:06.476370  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:06.375084  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3139) took 1440 ms. Trace:
W20260709 13:04:06.375871  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 152467us
I20260709 13:04:06.375972  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.478600  1737 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:06.379004  1742 rpcz_store.cc:276] 0709 13:04:05.278736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.279009 (+   273us) service_pool.cc:224] Handling call
0709 13:04:05.279479 (+   470us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:05.516784 (+237305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:05.516835 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:05.547314 (+ 30479us) log.cc:844] Serialized 26 byte log entry
0709 13:04:05.720881 (+173567us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:05.720926 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.134708 (+413782us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:06.134787 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.300873 (+166086us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:06.303369 (+  2496us) raft_consensus.cc:1685] finished
0709 13:04:06.303403 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.303606 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12452,"raft.queue_time_us":215203,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":66207,"spinlock_wait_cycles":547196032,"thread_start_us":114978,"threads_started":1,"wal-append.queue_time_us":223555}
I20260709 13:04:06.403932  1662 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:04:06.326336  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:06.342099  2316 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.018s
I20260709 13:04:06.456763  1665 rpcz_store.cc:276] 0709 13:04:04.976725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.455181 (+1478456us) service_pool.cc:224] Handling call
0709 13:04:06.455700 (+   519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.455985 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.456025 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.456083 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:06.456147 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.456224 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.456359 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:06.480052  2316 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.012s	sys 0.019s
I20260709 13:04:06.401886  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:06.423638  8318 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.425710  8085 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.427068  8349 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.429059  8401 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.432739  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:06.434449  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110682us
I20260709 13:04:06.505923  1668 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
W20260709 13:04:06.437146  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3142) took 1501 ms (client timeout 1653 ms). Trace:
I20260709 13:04:06.373484  8399 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:06.457101  2310 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:06.454291  2318 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:06.406227  8178 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.338976  8356 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.331979  8342 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.463539  8353 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.364715  8179 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.346072  8390 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.470822  1666 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:06.531011  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3150) took 1526 ms. Trace:
I20260709 13:04:06.531389  1666 rpcz_store.cc:276] 0709 13:04:05.004399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.470040 (+1465641us) service_pool.cc:224] Handling call
0709 13:04:06.470442 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.530709 (+ 60267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.530952 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37632}
I20260709 13:04:06.489929  8332 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.482185  1584 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:04:06.485424  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.485656  8339 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.486266  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:06.487142  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 777530us
I20260709 13:04:06.477068  1585 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:04:06.483011  1661 rpcz_store.cc:276] 0709 13:04:04.934770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.719159 (+784389us) service_pool.cc:224] Handling call
0709 13:04:06.375005 (+655846us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.498381  8122 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.545323  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1118) took 1288 ms. Trace:
I20260709 13:04:06.547516  1666 rpcz_store.cc:276] 0709 13:04:05.256847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.532476 (+1275629us) service_pool.cc:224] Handling call
0709 13:04:06.544817 (+ 12341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.545003 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.545027 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.545074 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:06.545120 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.545170 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.545263 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.550666  1668 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:06.551786  1737 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:06.553583  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.501012  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:06.551366  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2459) took 1320 ms. Trace:
I20260709 13:04:06.555148  1586 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:04:06.436004  1663 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:06.556592  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2454) took 1584 ms. Trace:
I20260709 13:04:06.557044  1663 rpcz_store.cc:276] 0709 13:04:04.972009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.435262 (+1463253us) service_pool.cc:224] Handling call
0709 13:04:06.435735 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.556254 (+120519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.556523 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.508510  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3140) took 1573 ms. Trace:
I20260709 13:04:06.558192  1659 rpcz_store.cc:276] 0709 13:04:04.935023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.259786 (+324763us) service_pool.cc:224] Handling call
0709 13:04:06.508309 (+1248523us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54001a72d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:06.508451 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1786946944}
W20260709 13:04:06.507194  1667 rpcz_store.cc:269] 0709 13:04:04.935537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.131635 (+196098us) service_pool.cc:224] Handling call
0709 13:04:06.436884 (+1305249us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:06.437059 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2360773376}
W20260709 13:04:06.526393  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 124543us
I20260709 13:04:06.555169  1668 rpcz_store.cc:276] 0709 13:04:05.231294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.531180 (+1299886us) service_pool.cc:224] Handling call
0709 13:04:06.550314 (+ 19134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.551071 (+   757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.551303 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:06.486845  8387 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.528695  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:06.482105  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:06.536067  2309 pending_rounds.cc:85] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 12
I20260709 13:04:06.497028  1738 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:06.571010  2309 pending_rounds.cc:107] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:04:06.566464  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.540212  2311 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:06.571386  2309 raft_consensus.cc:2889] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:06.583568  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.584657  1738 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:04:06.588049  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.592278  1738 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:06.594741  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112692us
I20260709 13:04:06.596228  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.596663  1738 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:06.526810  8398 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.601905  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:06.542953  8107 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:04:06.607687  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.605435  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:06.610286  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 104083us
I20260709 13:04:06.582019  8252 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:06.603026  8339 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.607342  8064 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 13:04:06.524699  8349 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.526546  8401 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.605729  8261 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:06.573911  8122 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.546195  8332 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.625236  1584 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:06.609048  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:06.624403  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.595353  8387 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.617477  8260 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:06.650588  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.651705  1664 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:06.652133  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:06.652297  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1117) took 1535 ms. Trace:
I20260709 13:04:06.652521  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3151) took 1591 ms. Trace:
I20260709 13:04:06.652971  1664 rpcz_store.cc:276] 0709 13:04:05.060800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.470061 (+1409261us) service_pool.cc:224] Handling call
0709 13:04:06.470442 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.652156 (+181714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.652463 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:06.673393  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 139350us
I20260709 13:04:06.674149  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:06.640439  7945 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:06.677875  1663 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:06.678901  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:06.679968  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68692.9us
I20260709 13:04:06.681011  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.688937  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:06.619915  8400 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.676683  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.671016  8339 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:06.654529  7946 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:06.691396  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:06.697912  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47430.9us
I20260709 13:04:06.672307  8182 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:06.714653  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3225) took 1022 ms. Trace:
I20260709 13:04:06.715227  1736 rpcz_store.cc:276] 0709 13:04:05.691874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.733365 (+ 41491us) service_pool.cc:224] Handling call
0709 13:04:06.714414 (+981049us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:06.714573 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":540596608}
I20260709 13:04:06.679415  8260 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:06.721740  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.682919  8338 raft_consensus.cc:2764] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:06.724922  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1119) took 1005 ms. Trace:
I20260709 13:04:06.733217  1666 rpcz_store.cc:276] 0709 13:04:05.719232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.548443 (+829211us) service_pool.cc:224] Handling call
0709 13:04:06.594680 (+ 46237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.594910 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:06.622310 (+ 27400us) log.cc:844] Serialized 14 byte log entry
0709 13:04:06.724513 (+102203us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.724571 (+    58us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:06.724630 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.724715 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.724859 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17638,"prepare.queue_time_us":46381,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":14464,"thread_start_us":96090,"threads_started":1}
I20260709 13:04:06.698799  8320 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:06.697711  8350 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.697751  8396 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.708776  8380 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.697331  8339 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:06.725693  8064 raft_consensus.cc:2764] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 13:04:06.764534  1741 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:06.697640  8417 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.144s
I20260709 13:04:06.748235  7945 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:06.718211  8410 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.223s
I20260709 13:04:06.763191  8339 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:06.757488  8252 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:06.771323  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.761600  8342 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:06.772498  8418 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 13:04:06.771934  8361 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:06.768396  8178 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.766242  8179 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:06.797672  8419 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 13:04:06.799248  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.801723  8179 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.802345  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.802260  8179 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.758395  8396 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.804195  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:06.783160  8390 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:06.792224  8416 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
W20260709 13:04:06.808493  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 127531us
I20260709 13:04:06.807822  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1150) took 1508 ms. Trace:
I20260709 13:04:06.812199  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:06.810316  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:06.782582  8356 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.762506  8380 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.778463  7946 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 13:04:06.814049  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:06.794415  7945 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:06.818751  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:06.812575  1735 rpcz_store.cc:276] 0709 13:04:05.299370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.300544 (+  1174us) service_pool.cc:224] Handling call
0709 13:04:05.339067 (+ 38523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.235865 (+896798us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.235898 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:06.236964 (+  1066us) log.cc:844] Serialized 501 byte log entry
0709 13:04:06.389644 (+152680us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:06.389714 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.439972 (+ 50258us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b5400404a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:06.440058 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.803310 (+363252us) raft_consensus.cc:1685] finished
0709 13:04:06.803333 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.807726 (+  4393us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060304640,"thread_start_us":152106,"threads_started":1,"wal-append.queue_time_us":383143}
I20260709 13:04:06.823483  1663 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:06.806443  8194 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:06.808394  8350 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:06.830860  8349 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.833810  8122 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:06.822711  7945 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:06.835945  8380 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:06.838037  8179 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.839689  8396 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:06.840723  8422 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 13:04:06.841997  8424 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 13:04:06.843436  8420 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:06.844230  8332 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.844065  8252 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.877316  8365 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:06.878118  8365 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:06.861824  7945 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:06.846144  1705 leader_election.cc:400] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:06.883088  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.885062  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.847273  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:06.850608  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:06.851236  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.895295  2315 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:06.851938  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.811848  8107 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 13:04:06.896639  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.881419  8365 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:06.859859  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54617.7us
I20260709 13:04:06.861896  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:06.863003  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 171612us
I20260709 13:04:06.824712  8417 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:06.886533  8372 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:06.898669  2317 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:06.886996  2316 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 13:04:06.898881  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:06.890367  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3230) took 1064 ms. Trace:
I20260709 13:04:06.899899  8179 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:06.900413  8179 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.900877  8179 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:06.901023  8122 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:06.901135  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:06.901861  8122 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.899993  1739 rpcz_store.cc:276] 0709 13:04:05.826073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.473285 (+647212us) service_pool.cc:224] Handling call
0709 13:04:06.473643 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.473870 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:06.473906 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:06.484395 (+ 10489us) log.cc:844] Serialized 26 byte log entry
0709 13:04:06.602790 (+118395us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:06.603407 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.603497 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:06.890079 (+286582us) raft_consensus.cc:1685] finished
0709 13:04:06.890107 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.890292 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":116701,"threads_started":1,"wal-append.queue_time_us":198089}
I20260709 13:04:06.902098  8064 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:06.902890  8064 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:06.903290  8064 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:06.906710  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.907599  1739 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.907944  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.908555  1738 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:06.908625  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.909143  8212 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:06.909873  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.910439  1736 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:06.911830  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3217) took 1257 ms. Trace:
I20260709 13:04:06.898343  8339 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.915023  8339 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.912225  1741 rpcz_store.cc:276] 0709 13:04:05.654917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.655570 (+   653us) service_pool.cc:224] Handling call
0709 13:04:05.655827 (+   257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.774778 (+1118951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.911079 (+136301us) spinlock_profiling.cc:243] Waited 3.05 ms on lock 0x7b5400333c20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0a02 00007fcdedfdcdd6 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:06.911591 (+   512us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:06.911768 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2547168512}
I20260709 13:04:06.919173  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.926018  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.917177  8339 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.936349  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.938874  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.952368  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.929803  8332 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.958021  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.958721  1741 raft_consensus.cc:2376] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19. Re-sending same reply.
I20260709 13:04:06.958802  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.959580  1738 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.960820  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.929602  8122 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:06.929378  8410 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:06.978125  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.961520  1739 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:06.955889  8212 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.955865  8397 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:06.963594  8433 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.982318  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.966615  8338 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.967459  8361 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:06.983569  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.983871  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.983570  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.984722  1742 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:06.933032  8417 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:06.986821  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:06.987228  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.987735  2318 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:06.973821  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.977205  1736 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:06.978873  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2461) took 1645 ms. Trace:
I20260709 13:04:06.982996  1741 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:06.971419  8320 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:06.971843  8407 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:06.939920  8380 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:06.988392  1738 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:06.989287  1737 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:06.989918  1661 rpcz_store.cc:276] 0709 13:04:05.333462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.542226 (+1208764us) service_pool.cc:224] Handling call
0709 13:04:06.582709 (+ 40483us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.582938 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:06.911730 (+328792us) log.cc:844] Serialized 14 byte log entry
0709 13:04:06.978487 (+ 66757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.978550 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:06.978610 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.978677 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.978808 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29476,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":111,"thread_start_us":372927,"threads_started":2}
I20260709 13:04:06.999490  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.000250  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:06.992871  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.981735  8212 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.993933  8431 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 13:04:06.994676  8432 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
W20260709 13:04:07.007089  1557 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.624s (SENT)
I20260709 13:04:07.004959  1741 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:06.997428  8320 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:06.977926  8332 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.980950  8390 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:07.001618  1737 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.001972  1736 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:07.006757  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:06.960340  8339 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:07.011487  1557 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.660s (SENT)
I20260709 13:04:07.011961  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:06.987417  8433 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.009013  7945 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.022723  8433 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.997686  8417 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:07.017716  8318 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.509s	user 0.013s	sys 0.002s
I20260709 13:04:07.010098  8320 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:06.986006  8064 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.019982  8332 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.014412  8212 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.026660  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.024458  7945 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:06.990119  8338 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.048024  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:07.006927  8399 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.036540  8365 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:07.047780  8261 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 13:04:07.052714  8261 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:07.053099  8261 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:07.038854  1741 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:07.038312  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.053892  1736 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.054242  8261 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.038398  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.058274  8445 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:07.058995  8445 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.059363  8445 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.060359  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.046972  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.061100  1735 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:07.061795  8338 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.062129  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.062194  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.063081  1742 raft_consensus.cc:2368] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:07.063035  1735 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:07.064055  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.064169  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.064687  1742 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:07.064843  1735 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
W20260709 13:04:07.040124  1557 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.700s (SENT) [suppressed 22 similar messages]
I20260709 13:04:07.066761  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:07.067418  2318 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.067378  2310 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:07.033149  8320 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.070418  1737 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:07.071877  8399 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.073175  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.073975  1742 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:07.076524  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.077374  1742 raft_consensus.cc:2393] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:07.078017  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.033187  8064 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:07.079201  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.080403  1737 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:07.082643  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.082878  1735 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:07.084645  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.087666  1736 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:07.095543  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.088868  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.089257  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.090479  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.085428  1735 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:07.100010  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:07.069411  8261 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.087313  1742 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:07.091992  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.106618  1584 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:04:07.100457  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.101176  1659 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:07.101783  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.113402  1742 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:07.078421  8401 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:07.104022  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.119431  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.122336  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.114950  1591 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:07.116106  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.125002  1741 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:07.102665  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.207602  1667 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:07.214860  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:07.220999  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90599.6us
I20260709 13:04:07.229784  1659 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:07.126471  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.212749  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.231181  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.128041  1591 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 13:04:07.138898  2318 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:07.142506  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.230968  1737 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
W20260709 13:04:07.183781  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 143597us
I20260709 13:04:07.134502  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.185882  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.143285  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.232034  1659 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:07.212569  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.219954  1736 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [17.8-19.9]
I20260709 13:04:07.207063  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.242782  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.200652  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.243237  1666 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 13:04:07.205041  8426 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.382s	user 0.001s	sys 0.000s
W20260709 13:04:07.237713  1705 connection.cc:256] RPC call timeout handler was delayed by -1.1566943540000238s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
I20260709 13:04:07.127214  8338 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:07.247684  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.253443  1737 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:07.126871  8320 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:07.221385  8025 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.256830  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:07.183693  8397 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:07.216549  8339 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.244077  1592 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:07.263690  1667 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:04:07.266325  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.264348  8338 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:07.252025  8433 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 13:04:07.128862  8445 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.242383  8391 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:07.305938  1591 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
W20260709 13:04:07.281131  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.308523  1664 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:07.261873  8064 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:07.314522  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.315390  1592 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:07.316979  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.318190  8445 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:07.317425  1667 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 13:04:07.319466  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:07.320294  1737 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.265774  8397 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:07.335341  1737 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:07.324334  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.337977  1586 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:07.327258  8378 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:07.330060  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:07.342072  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:07.266168  8025 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.305238  8433 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:07.345402  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.347913  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:07.322921  8064 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.349300  1666 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:07.355744  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.358428  1660 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 13:04:07.356531  1591 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:07.307164  8179 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.363214  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:07.364135  1667 raft_consensus.cc:2376] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16. Re-sending same reply.
I20260709 13:04:07.344867  8433 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:07.370507  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.360019  8064 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:07.371389  1591 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:07.362521  8391 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:07.373054  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 474551us
I20260709 13:04:07.366536  8443 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:07.377985  8400 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.379505  8348 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:07.382421  8448 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 13:04:07.418392  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.418882  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.419092  1591 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:07.374306  8391 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:07.386027  7939 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:07.387905  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:07.388326  7430 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.388705  8018 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:07.389744  8025 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.369441  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:07.393375  8179 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:07.394178  8429 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.556s	user 0.004s	sys 0.000s
W20260709 13:04:07.398944  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 165391us
I20260709 13:04:07.398720  1586 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:04:07.399933  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 192231us
W20260709 13:04:07.427342  1557 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.529s (SENT)
I20260709 13:04:07.428068  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.370069  8338 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.403534  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.404312  8417 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:07.371796  8397 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.408398  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.409798  8356 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Committing config change with OpId 15.10: config changed from index 4 to 10, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:07.411777  8387 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:07.374086  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:07.414096  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.417073  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.368048  8445 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:07.419569  1668 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:07.420044  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.420477  2309 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.449874  8445 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:07.451185  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:07.368163  8320 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:07.423837  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:07.452898  1667 raft_consensus.cc:2368] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:07.420717  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:07.431540  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.430820  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.435390  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.449275  1666 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:07.452637  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:06.671375  1665 rpcz_store.cc:276] 0709 13:04:05.117373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.480417 (+1363044us) service_pool.cc:224] Handling call
0709 13:04:06.480743 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:06.480980 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:06.530015 (+ 49035us) log.cc:844] Serialized 14 byte log entry
0709 13:04:06.651463 (+121448us) log.cc:844] Serialized 14 byte log entry
0709 13:04:06.651786 (+   323us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:06.651848 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:06.651930 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:06.652027 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:06.652210 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16728,"prepare.queue_time_us":52240,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":7296,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":14544,"thread_start_us":113828,"threads_started":1}
I20260709 13:04:07.454501  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.423372  8391 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:07.432539  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66890.5us
I20260709 13:04:07.455737  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.419086  8411 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 13:04:07.450995  8445 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.457342  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3272) took 2155 ms. Trace:
W20260709 13:04:07.473793  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46464us
I20260709 13:04:07.473801  1593 rpcz_store.cc:276] 0709 13:04:05.292130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:05.292376 (+   246us) service_pool.cc:224] Handling call
0709 13:04:07.442890 (+2150514us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:07.447265 (+  4375us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4448359168}
W20260709 13:04:07.474769  1557 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.585s (SENT)
I20260709 13:04:07.457760  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:07.460831  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86808.9us
W20260709 13:04:07.461771  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.485450  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.438987  7945 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:07.461400  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:07.490345  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:07.470402  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3147) took 2533 ms. Trace:
I20260709 13:04:07.490909  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:07.471119  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.436113  8025 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:07.462271  1665 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Committing config change with OpId 15.10: config changed from index 4 to 10, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:07.491258  1662 rpcz_store.cc:276] 0709 13:04:04.936809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.259223 (+1322414us) service_pool.cc:224] Handling call
0709 13:04:06.259705 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:06.479466 (+219761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.470126 (+990660us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400118354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:07.470335 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":330707200}
I20260709 13:04:07.471730  8179 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 13:04:07.478437  8318 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:07.438567  8417 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.437243  8350 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:07.477571  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.486948  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.478230  7748 raft_consensus.cc:2764] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:07.465484  8365 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:07.462328  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.466270  8212 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:07.494647  8085 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:07.498759  8320 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.494400  8398 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:07.518103  8398 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.518237  1486 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 18, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 18 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:07.520318  8398 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.519932  1486 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 20, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 20 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:07.498876  8260 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:07.465495  8410 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:07.521981  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52912.7us
I20260709 13:04:07.522847  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.523770  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:07.524159  8454 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:07.525041  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.488843  8378 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:07.498363  2309 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:07.473297  8387 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:07.521454  1486 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:07.503235  1590 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:07.485188  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.504463  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.551143  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:07.582446  1664 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 15.
I20260709 13:04:07.558996  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:07.499806  8397 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:07.478705  8443 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.065s
I20260709 13:04:07.500360  8318 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.559348  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:07.503304  7945 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:07.541898  7430 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:07.499622  8348 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:07.500893  8432 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:07.578094  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118110us
W20260709 13:04:07.581281  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56309.9us
I20260709 13:04:07.658004  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.508273  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:07.658680  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 229262us
I20260709 13:04:07.659471  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.583874  2318 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:07.506851  8212 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:07.599258  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 114110us
I20260709 13:04:07.599018  8400 raft_consensus.cc:2764] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 13:04:07.504112  8365 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:07.604982  1591 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:07.645030  8318 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:07.541582  8260 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:07.511704  8390 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:07.653803  8348 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.501118  8332 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.097s
I20260709 13:04:07.505322  8407 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.098s
I20260709 13:04:07.641407  2312 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:07.506474  8064 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.503657  8367 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:07.558751  8433 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.671499  8025 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:07.602918  8387 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:07.641803  8261 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:07.636667  8343 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:07.671608  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:07.672057  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 163824us
I20260709 13:04:07.677296  8260 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.670711  1590 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:07.653743  8389 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:07.675488  8450 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:07.653983  7430 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:07.640103  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 135659us
I20260709 13:04:07.719846  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.661266  8432 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.726074  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.662032  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.698307  1739 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:07.702399  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:07.583845  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.709213  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:07.687196  8348 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.715318  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:07.669062  8212 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:07.673956  8457 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 13:04:07.698832  8025 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.693377  8349 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:07.726460  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.696467  8261 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.697379  8387 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.770443  8378 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:07.771152  8378 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.730450  8431 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:07.728694  8432 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.732817  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:07.733906  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71945.3us
I20260709 13:04:07.724385  7945 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.734589  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:07.734356  8365 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:07.735095  8450 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:07.782471  8450 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:07.740326  8351 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 13:04:07.735337  8320 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:07.735589  8339 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:07.787581  8064 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:07.790311  8433 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:07.796099  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61579.1us
W20260709 13:04:07.738214  8437 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.892s	user 0.006s	sys 0.000s
I20260709 13:04:07.717226  8260 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:07.723874  8456 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
W20260709 13:04:07.740617  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 156996us
I20260709 13:04:07.768359  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:07.714046  8389 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:07.773851  1592 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:07.775717  8025 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:07.776728  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50303.6us
I20260709 13:04:07.723181  7430 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:07.727306  8343 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:07.766184  8399 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:07.798103  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:07.777112  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:07.740124  8367 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.162s
I20260709 13:04:07.739681  8411 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.163s
I20260709 13:04:07.965296  1586 raft_consensus.cc:2368] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:07.770454  8261 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.773481  8378 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:07.939940  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 265943us
I20260709 13:04:07.981652  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:07.953315  1705 leader_election.cc:400] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:07.982565  1705 leader_election.cc:403] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:07.777905  8317 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:07.906510  8443 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:07.794350  8431 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:07.966555  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 168476us
W20260709 13:04:07.977360  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202405us
I20260709 13:04:07.977763  1588 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:07.743150  8085 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.183s
I20260709 13:04:08.002434  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:08.006248  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63646us
I20260709 13:04:08.006893  2281 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:08.003391  1742 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:07.940197  8389 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:07.912021  8320 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:08.014834  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:08.010605  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:08.018975  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:08.012046  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:08.027360  1741 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:08.023263  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:08.024992  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:08.032155  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:08.033149  1737 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:07.976464  8399 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:08.005739  8338 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:04:08.021800  8437 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 13:04:08.008001  8431 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:08.046721  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 143900us
I20260709 13:04:08.018599  8085 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.002476  8317 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:08.042791  8466 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 13:04:07.901546  8468 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.212s
W20260709 13:04:08.042026  8460 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.249s	user 0.000s	sys 0.002s
I20260709 13:04:08.031243  8469 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:08.065972  8474 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:08.069260  1742 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:08.071724  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:08.073984  1586 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:08.096863  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.097723  1586 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:08.051326  8399 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:08.036464  8407 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 13:04:08.056140  8317 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.057027  8332 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:08.053565  8471 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.058609  8431 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:08.059535  8338 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:08.093376  8469 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:08.159502  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.169745  1586 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:08.074646  8396 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:08.096004  8474 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:08.100610  8465 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:08.159720  8343 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:08.218725  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:08.171216  8464 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 13:04:08.189648  8445 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:08.172210  8470 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:04:08.211418  8463 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.482s	user 0.000s	sys 0.002s
I20260709 13:04:08.207243  8466 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.209753  8474 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:08.268090  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.233906  8343 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:08.169130  8471 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.259788  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.287446  1586 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:04:08.254524  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 182835us
I20260709 13:04:08.253418  8470 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:08.257827  8445 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:08.182325  8338 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.215147  8465 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.168408  8320 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.302136  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:08.298339  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.002s	sys 0.003s
W20260709 13:04:08.298177  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 283334us
W20260709 13:04:08.297516  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 289292us
I20260709 13:04:08.363696  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:08.291627  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 523092us
W20260709 13:04:08.361279  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.002s	sys 0.007s
W20260709 13:04:08.366581  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 640286us
I20260709 13:04:08.228336  8396 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:08.367781  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:08.352689  8397 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.712s	user 0.014s	sys 0.001s
I20260709 13:04:08.318344  8443 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:08.365281  8064 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:08.352236  8468 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:08.376686  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.235080  8182 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:08.384814  1591 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 6. Preceding OpId from leader: term: 17 index: 8. (index mismatch)
I20260709 13:04:08.373716  1664 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:08.265748  8466 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:08.395176  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.378402  8318 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.001s
I20260709 13:04:08.370375  8478 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:08.370447  8473 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.205s	user 0.003s	sys 0.003s
W20260709 13:04:08.404338  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.003s	sys 0.010s
I20260709 13:04:08.399844  1662 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 13:04:08.400357  1668 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 13:04:08.398418  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 388308us
I20260709 13:04:08.301335  8343 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.411144  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:08.411885  1593 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:08.417659  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:08.421559  1664 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:08.421746  1591 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:08.423297  1591 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:08.426342  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.380708  8411 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:08.371683  8396 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.384999  8443 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:08.401039  8387 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.630s	user 0.013s	sys 0.000s
I20260709 13:04:08.440320  1587 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 13:04:08.400110  8348 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.634s	user 0.012s	sys 0.001s
I20260709 13:04:08.441609  1589 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
W20260709 13:04:08.421312  8391 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.945s	user 0.010s	sys 0.005s
I20260709 13:04:08.389407  8085 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
W20260709 13:04:08.404781  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.003s	sys 0.010s
I20260709 13:04:08.284008  8469 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.466717  8469 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:08.438849  8396 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:08.445585  8458 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.374456  8445 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.392938  8468 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:08.448457  1586 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:08.470001  8458 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.423089  8450 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.401314  8064 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.489642  8450 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.463224  8401 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.491360  8401 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.403945  8182 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:08.494128  8482 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:08.494271  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:08.485999  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:08.495832  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.472496  8391 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.497056  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.430301  8493 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:08.448678  8443 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:08.497210  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2513) took 1522 ms (client timeout 1624 ms). Trace:
I20260709 13:04:08.501029  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:08.507889  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 139952us
W20260709 13:04:08.514158  1739 rpcz_store.cc:269] 0709 13:04:06.974626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:06.979448 (+  4822us) service_pool.cc:224] Handling call
0709 13:04:08.497007 (+1517559us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b5400338954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:08.497127 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1641260928}
I20260709 13:04:08.515676  1557 leader_election.cc:400] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:08.516088  1557 leader_election.cc:403] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:08.400455  8318 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.519539  1593 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:08.486660  8432 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.714s	user 0.013s	sys 0.001s
I20260709 13:04:08.549279  1733 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:08.519606  8212 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.737s	user 0.014s	sys 0.000s
I20260709 13:04:08.554869  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:08.457083  8348 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.400846  8478 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.512813  8465 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:08.454643  8387 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:08.574031  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 130901us
I20260709 13:04:08.536765  8182 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:08.536576  8332 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:08.461230  8350 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 13:04:08.589905  8432 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.647760  8432 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:08.623785  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.658437  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:08.584964  8476 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.687307  8349 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:08.687943  8349 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:08.674579  8468 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.680001  8350 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:08.690966  8350 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:08.691325  8350 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:08.705864  8318 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.710950  8391 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.712069  8391 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.712622  8391 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:08.718513  8431 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.536s	user 0.005s	sys 0.007s
I20260709 13:04:08.718909  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.719291  8431 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.719955  1665 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:08.720991  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.681571  8064 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:08.706023  8462 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.071s	user 0.007s	sys 0.000s
I20260709 13:04:08.724937  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.729137  1661 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 13:04:08.722910  8317 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.004s
I20260709 13:04:08.594264  8212 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:08.780797  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45719.1us
W20260709 13:04:08.752961  8261 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.736s	user 0.016s	sys 0.000s
W20260709 13:04:08.729981  8389 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.690s	user 0.015s	sys 0.003s
W20260709 13:04:08.724062  8483 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.205s	user 0.000s	sys 0.000s
I20260709 13:04:08.845883  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:08.788537  1662 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 13:04:08.850687  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:08.843223  7748 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 13:04:08.736447  1665 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:08.750001  8025 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.772s	user 0.013s	sys 0.001s
I20260709 13:04:08.814859  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.893167  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:08.854456  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:08.818405  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:08.729362  8356 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.159s	user 0.013s	sys 0.004s
I20260709 13:04:08.487918  8470 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.831267  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:08.789012  8342 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.834810  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:08.918664  1593 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:08.728009  8387 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:08.721473  8431 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:08.844892  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:08.845862  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:08.728684  8458 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.856240  1587 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:08.858037  8182 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:08.858976  8389 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.823168  8317 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.777953  8474 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:08.862929  8496 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 13:04:08.682523  8476 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.995569  8474 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.842023  8261 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.998289  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3292) took 1284 ms. Trace:
I20260709 13:04:08.999012  8261 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.998767  1590 rpcz_store.cc:276] 0709 13:04:07.713437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.713648 (+   211us) service_pool.cc:224] Handling call
0709 13:04:07.718618 (+  4970us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:07.951750 (+233132us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.951786 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:07.952493 (+   707us) log.cc:844] Serialized 52 byte log entry
0709 13:04:08.332038 (+379545us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b18000cbf20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:08.476288 (+144250us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:08.562363 (+ 86075us) log.cc:844] Serialized 14 byte log entry
0709 13:04:08.753459 (+191096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.753580 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.997841 (+244261us) raft_consensus.cc:1685] finished
0709 13:04:08.997987 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:08.998184 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67452,"prepare.queue_time_us":86413,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":432182,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":13078,"spinlock_wait_cycles":13997696,"thread_start_us":567341,"threads_started":3,"wal-append.queue_time_us":565747}
I20260709 13:04:08.922533  1662 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:08.926513  8025 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:08.923959  1665 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:08.925861  1585 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:09.008554  8478 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:09.011766  8417 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.018060  8348 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.022090  8396 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.002s
I20260709 13:04:09.022778  8396 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.025825  8502 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:09.026536  8502 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:09.026981  8502 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:08.933758  8342 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:09.032538  8342 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:08.925596  2312 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:08.822856  8349 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:08.893262  8401 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.078681  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 523844us
W20260709 13:04:08.991024  8399 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.834s	user 0.012s	sys 0.001s
I20260709 13:04:09.079849  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:09.083737  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 484235us
I20260709 13:04:09.088403  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:09.089373  1590 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:09.097059  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:09.112962  1588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.040s	sys 0.056s
W20260709 13:04:09.113629  1588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.041s	sys 0.056s
W20260709 13:04:09.145287  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.012s	sys 0.006s
W20260709 13:04:09.145915  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.012s	sys 0.006s
I20260709 13:04:09.067929  8458 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:09.149586  1663 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.655s	user 0.034s	sys 0.050s
W20260709 13:04:09.177366  1663 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.683s	user 0.034s	sys 0.050s
W20260709 13:04:09.150729  1659 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.000s	sys 0.019s
W20260709 13:04:09.180176  1659 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.000s	sys 0.019s
W20260709 13:04:09.150104  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.007s	sys 0.011s
W20260709 13:04:09.180919  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.007s	sys 0.012s
W20260709 13:04:09.151019  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.010s	sys 0.008s
W20260709 13:04:09.183048  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.010s	sys 0.008s
I20260709 13:04:09.084824  8317 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.111438  8469 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.632s	user 0.007s	sys 0.008s
I20260709 13:04:09.246371  1590 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:09.245905  8458 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:09.200899  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.005s	sys 0.015s
W20260709 13:04:09.337620  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.005s	sys 0.015s
W20260709 13:04:09.186580  8493 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.674s	user 0.004s	sys 0.007s
W20260709 13:04:09.291540  7430 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.311s	user 0.015s	sys 0.000s
I20260709 13:04:09.091972  8399 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.712109  8396 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.451738  7430 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:09.634685  8350 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.943s	user 0.006s	sys 0.006s
I20260709 13:04:09.790318  8350 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.791556  8396 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:08.545463   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:04:09.716922  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:09.633759  8260 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.683s	user 0.015s	sys 0.002s
I20260709 13:04:09.795717  8260 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:09.720934  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 869924us
I20260709 13:04:09.798460  8260 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.728377  8365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.557s	user 0.000s	sys 0.012s
I20260709 13:04:09.717491  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:09.799990  8365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.628s	user 0.000s	sys 0.014s
I20260709 13:04:09.711923  8476 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.802763  8389 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:09.803411  1486 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 14 to 17, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 17 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:09.714781  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:09.728283  8400 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:09.725894  1660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.003s	sys 0.002s
W20260709 13:04:09.806025  1666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.103s	user 0.012s	sys 0.010s
W20260709 13:04:09.806110  1660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.003s	sys 0.004s
I20260709 13:04:09.806110  8400 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.805153  1486 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:09.713655  8212 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.834824  8342 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.791924  8490 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.722884  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3310) took 1337 ms. Trace:
I20260709 13:04:09.793679  2313 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
W20260709 13:04:09.475558  8356 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.000s	sys 0.009s
W20260709 13:04:09.462281  8343 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.008s	sys 0.002s
I20260709 13:04:09.794757  8396 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.805546  8470 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:09.360384  8493 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:09.816586  1486 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 20. New cstate: current_term: 20 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:09.819666  8318 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:09.896304  8318 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.897363  8396 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.898049  8396 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:09.898675  8318 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.898545  8396 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.822273  8317 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:09.899528  8317 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.900687  8396 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:09.901630  8317 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.886615  8343 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.008s	sys 0.003s
I20260709 13:04:09.838222  8490 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:09.875979  8342 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:09.713644  8399 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:09.376293  8445 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.000s	sys 0.008s
I20260709 13:04:09.800520  1589 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:09.916493  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3313) took 1101 ms. Trace:
I20260709 13:04:09.917258  1589 rpcz_store.cc:276] 0709 13:04:08.814710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.894280 (+ 79570us) service_pool.cc:224] Handling call
0709 13:04:09.916389 (+1022109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:09.806353  1666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.103s	user 0.013s	sys 0.010s
I20260709 13:04:09.736346  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:09.832844  8517 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:09.832948  8260 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:09.833072  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:09.833820  8476 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:09.825131  8179 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:09.836149  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:09.837128  8513 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:09.843277  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3178) took 1466 ms. Trace:
I20260709 13:04:09.715193  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:09.841423  1584 rpcz_store.cc:276] 0709 13:04:08.385814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.386035 (+   221us) service_pool.cc:224] Handling call
0709 13:04:08.386509 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:08.428257 (+ 41748us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:08.428299 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:08.455305 (+ 27006us) log.cc:844] Serialized 26 byte log entry
0709 13:04:08.620684 (+165379us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:08.620752 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.620867 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:09.722581 (+1101714us) raft_consensus.cc:1685] finished
0709 13:04:09.722605 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:09.722793 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25873,"raft.queue_time_us":26809,"spinlock_wait_cycles":14071168,"thread_start_us":88091,"threads_started":1,"wal-append.queue_time_us":614648}
I20260709 13:04:09.849848  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:09.791514  7430 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:09.856907  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3299) took 1829 ms. Trace:
W20260709 13:04:09.886503  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 806373us
I20260709 13:04:09.850741  8400 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:09.855425  8320 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.124s	user 0.008s	sys 0.015s
W20260709 13:04:09.929380  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 96316.1us
I20260709 13:04:09.675630  8458 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:09.856298  8025 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:10.400646  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.389389  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.388964  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.400779  1742 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:09.961247  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3175) took 1585 ms. Trace:
I20260709 13:04:10.281189  1708 leader_election.cc:400] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:10.409368  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:10.305169  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2482) took 1554 ms. Trace:
W20260709 13:04:10.221037  1663 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.002s	sys 0.047s
I20260709 13:04:10.370136  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3255) took 2383 ms. Trace:
W20260709 13:04:10.422001  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.252337  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3304) took 1842 ms. Trace:
I20260709 13:04:10.368992  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2528) took 3288 ms. Trace:
W20260709 13:04:10.110450  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3299) took 2640 ms (client timeout 1942 ms). Trace:
I20260709 13:04:10.279946  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:10.424499  1585 rpcz_store.cc:269] 0709 13:04:07.470439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.470832 (+   393us) service_pool.cc:224] Handling call
0709 13:04:10.110105 (+2639273us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540025e754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:10.110324 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3293478272}
W20260709 13:04:10.417557  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 162568us
W20260709 13:04:09.947459  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 111418us
W20260709 13:04:09.962229  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 149158us
I20260709 13:04:10.328606  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3298) took 2366 ms. Trace:
W20260709 13:04:10.422863  1663 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.004s	sys 0.063s
I20260709 13:04:10.434571  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:09.976311  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2478) took 1529 ms. Trace:
I20260709 13:04:10.252406  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3256) took 2265 ms. Trace:
I20260709 13:04:10.445077  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3181) took 2054 ms. Trace:
I20260709 13:04:10.415834  1741 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:10.429090  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:09.931387  8445 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.054s	user 0.000s	sys 0.010s
I20260709 13:04:10.442461  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:09.904480  7945 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:10.194025  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2480) took 1687 ms. Trace:
I20260709 13:04:10.307952  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3180) took 1918 ms. Trace:
I20260709 13:04:10.442160  8458 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:10.321310  8410 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:10.367295  8018 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:10.416134  1666 rpcz_store.cc:276] 0709 13:04:08.375724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.376982 (+  1258us) service_pool.cc:224] Handling call
0709 13:04:08.381036 (+  4054us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:08.536796 (+155760us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:08.537036 (+   240us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:08.701691 (+164655us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b2c0000bff8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:08.702511 (+   820us) log.cc:844] Serialized 52 byte log entry
0709 13:04:09.920958 (+1218447us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:09.921019 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.921122 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:09.921174 (+    52us) raft_consensus.cc:1685] finished
0709 13:04:09.921184 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:09.961137 (+ 39953us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":151636,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":24170,"spinlock_wait_cycles":33087232,"thread_start_us":1344496,"threads_started":2,"wal-append.queue_time_us":1014294}
I20260709 13:04:10.449361  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.445859  1663 rpcz_store.cc:276] 0709 13:04:08.391067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.391258 (+   191us) service_pool.cc:224] Handling call
0709 13:04:08.420138 (+ 28880us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.179511 (+759373us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.179547 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:09.713090 (+533543us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.444642 (+731552us) raft_consensus.cc:1645] Marking committed up to 6
0709 13:04:10.444710 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.444793 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.444851 (+    58us) raft_consensus.cc:1685] finished
0709 13:04:10.444863 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.445016 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87849,"raft.queue_time_us":214483,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":14517,"thread_start_us":1415655,"threads_started":2,"wal-append.queue_time_us":586083}
I20260709 13:04:10.426865  1592 rpcz_store.cc:276] 0709 13:04:07.962896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.963137 (+   241us) service_pool.cc:224] Handling call
0709 13:04:07.963952 (+   815us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:08.353948 (+389996us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:08.353983 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:08.362646 (+  8663us) log.cc:844] Serialized 26 byte log entry
0709 13:04:09.338805 (+976159us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:09.338871 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.339600 (+   729us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.328291 (+988691us) raft_consensus.cc:1685] finished
0709 13:04:10.328319 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.328520 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60264,"raft.queue_time_us":89979,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16666,"thread_start_us":1254536,"threads_started":2,"wal-append.queue_time_us":1671217}
I20260709 13:04:10.417346  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1106) took 1255 ms. Trace:
I20260709 13:04:10.416301  1588 rpcz_store.cc:276] 0709 13:04:08.027337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.028666 (+  1329us) service_pool.cc:224] Handling call
0709 13:04:08.028987 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.029208 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:08.299735 (+270527us) log.cc:844] Serialized 14 byte log entry
0709 13:04:09.114164 (+814429us) log.cc:844] Serialized 14 byte log entry
0709 13:04:09.856134 (+741970us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:09.856213 (+    79us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:09.856302 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.856567 (+   265us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:09.856800 (+   233us) inbound_call.cc:177] Queueing success response
0709 13:04:09.979102 (+122302us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b180033f6a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
Metrics: {"mutex_wait_us":1138686,"prepare.queue_time_us":198650,"prepare.run_cpu_time_us":120,"prepare.run_wall_time_us":126,"raft.queue_time_us":891365,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":14813,"spinlock_wait_cycles":79394560,"thread_start_us":1038288,"threads_started":3,"wal-append.queue_time_us":398215}
I20260709 13:04:10.423098  1591 rpcz_store.cc:276] 0709 13:04:08.410281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.437192 (+ 26911us) service_pool.cc:224] Handling call
0709 13:04:08.437666 (+   474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:08.454552 (+ 16886us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:04:08.487486 (+ 32934us) log.cc:844] Serialized 14 byte log entry
0709 13:04:09.962859 (+1475373us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.250497 (+287638us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.251953 (+  1456us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:10.251998 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.252072 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.252116 (+    44us) raft_consensus.cc:1685] finished
0709 13:04:10.252126 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.252270 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":782552,"prepare.queue_time_us":65037,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":47,"raft.queue_time_us":783283,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":17631,"thread_start_us":630077,"threads_started":1,"wal-append.queue_time_us":1185716}
W20260709 13:04:10.044947  8479 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.133s	user 0.001s	sys 0.000s
I20260709 13:04:10.226783  8350 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.084591  8179 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:10.455263  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.452529  1659 rpcz_store.cc:276] 0709 13:04:08.389362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.389523 (+   161us) service_pool.cc:224] Handling call
0709 13:04:08.389883 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.180659 (+790776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.180693 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:09.807010 (+626317us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.160180 (+353170us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:10.160240 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.160348 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.307605 (+147257us) raft_consensus.cc:1685] finished
0709 13:04:10.307639 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.307858 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22043,"raft.queue_time_us":184898,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6803,"thread_start_us":1127453,"threads_started":2,"wal-append.queue_time_us":498054}
I20260709 13:04:10.421401  1665 rpcz_store.cc:276] 0709 13:04:08.750568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.003513 (+252945us) service_pool.cc:224] Handling call
0709 13:04:09.714915 (+711402us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:09.830619 (+115704us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.830654 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:09.831356 (+   702us) log.cc:844] Serialized 52 byte log entry
0709 13:04:10.176588 (+345232us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:10.176648 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.176772 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.304780 (+128008us) raft_consensus.cc:1685] finished
0709 13:04:10.304825 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.305060 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115655,"raft.queue_time_us":117156,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":344410,"threads_started":1,"wal-append.queue_time_us":470125}
I20260709 13:04:10.444850  1661 rpcz_store.cc:276] 0709 13:04:08.446443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.934648 (+488205us) service_pool.cc:224] Handling call
0709 13:04:08.935073 (+   425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:09.839598 (+904525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:09.839656 (+    58us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:09.839737 (+    81us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:09.839986 (+   249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:09.975930 (+135944us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b54001a6dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:09.976008 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:09.976224 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43284,"raft.queue_time_us":59177,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":15201,"spinlock_wait_cycles":225005824}
I20260709 13:04:10.481774  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.417847  1667 rpcz_store.cc:276] 0709 13:04:08.376791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.376953 (+   162us) service_pool.cc:224] Handling call
0709 13:04:08.403717 (+ 26764us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:08.448693 (+ 44976us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:08.448733 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:08.475806 (+ 27073us) log.cc:844] Serialized 52 byte log entry
0709 13:04:08.749911 (+274105us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:08.749978 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:08.750095 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:09.230568 (+480473us) raft_consensus.cc:1685] finished
0709 13:04:09.230599 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:09.843175 (+612576us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44527,"raft.queue_time_us":196723,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":46684,"thread_start_us":199422,"threads_started":1,"wal-append.queue_time_us":665870}
I20260709 13:04:10.280472  8486 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.445294  8342 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:10.458292  8431 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.149654  8520 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.093s	user 0.000s	sys 0.004s
I20260709 13:04:10.465831  8400 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:10.488624  8431 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.397945  8387 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.458855  8509 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.461301  8410 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.489789  8387 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.489917  8509 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.334259  8397 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.491887  8410 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.492448  8509 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:09.957002  8443 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.406s	user 0.015s	sys 0.001s
I20260709 13:04:10.391659  8411 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.493093  8398 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.493295  8411 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.493736  8398 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.494621  8504 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:10.495041  8504 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:10.495757  8411 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.465389  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:10.460718  1590 rpcz_store.cc:276] 0709 13:04:09.162131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.336978 (+174847us) service_pool.cc:224] Handling call
0709 13:04:10.417272 (+1080294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:10.452152  1662 rpcz_store.cc:276] 0709 13:04:08.506310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.002242 (+495932us) service_pool.cc:224] Handling call
0709 13:04:09.718418 (+716176us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.193400 (+474982us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.193451 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:10.193549 (+    98us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:10.193613 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.193720 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.193933 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":101631,"raft.queue_time_us":108954,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":372513,"threads_started":1}
W20260709 13:04:10.466602  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57323.1us
I20260709 13:04:10.506390  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.445426  1735 rpcz_store.cc:276] 0709 13:04:07.986866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.993620 (+  6754us) service_pool.cc:224] Handling call
0709 13:04:08.023623 (+ 30003us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:08.024784 (+  1161us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:08.209333 (+184549us) log.cc:844] Serialized 14 byte log entry
0709 13:04:10.251957 (+2042624us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:10.252024 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:10.252091 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.252180 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.252310 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1724565,"prepare.queue_time_us":71916,"prepare.run_cpu_time_us":6872,"prepare.run_wall_time_us":45095,"raft.queue_time_us":1727383,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":23024,"thread_start_us":487228,"threads_started":2,"wal-append.queue_time_us":504547,"wal-append.run_cpu_time_us":1486,"wal-append.run_wall_time_us":1001506}
W20260709 13:04:10.507447  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3254) took 2521 ms (client timeout 2817 ms). Trace:
I20260709 13:04:10.309307  8417 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:09.935731  8339 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.972s	user 0.013s	sys 0.003s
I20260709 13:04:10.204792  8475 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.315986  7939 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.229423  8505 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:10.222568  8524 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.560447  8417 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.429461  8407 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.334558  8390 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.389549  8432 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:10.444152  8470 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.548s	user 0.016s	sys 0.000s
I20260709 13:04:10.321166  8494 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:10.471187  8332 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.367880  8528 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.455624  7945 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.208384  8516 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.337260  8471 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.101864  8351 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.160913  8512 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.428555  8389 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:10.423485  1738 rpcz_store.cc:276] 0709 13:04:07.081031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.222153 (+141122us) service_pool.cc:224] Handling call
0709 13:04:07.222622 (+   469us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:07.250319 (+ 27697us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:07.250356 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:07.251118 (+   762us) log.cc:844] Serialized 52 byte log entry
0709 13:04:07.416535 (+165417us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:07.430732 (+ 14197us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.435935 (+  5203us) log.cc:844] Serialized 14 byte log entry
0709 13:04:07.449383 (+ 13448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:07.449495 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:08.694993 (+1245498us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b18001c1260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:08.861382 (+166389us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b18001c1260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:10.044765 (+1183383us) spinlock_profiling.cc:243] Waited 6.71 ms on lock 0x7b18001c1260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:10.368689 (+323924us) raft_consensus.cc:1685] finished
0709 13:04:10.368721 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.368916 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3061618,"prepare.queue_time_us":57539,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":4243,"raft.queue_time_us":23586,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":10179,"spinlock_wait_cycles":34090240,"thread_start_us":140621,"threads_started":1,"wal-append.queue_time_us":196347}
I20260709 13:04:10.585620  8389 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.586277  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.587090  1738 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:10.588179  8389 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:09.946981  8517 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.590346  2312 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:10.591786  8517 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.422760  1740 rpcz_store.cc:276] 0709 13:04:07.986605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.986775 (+   170us) service_pool.cc:224] Handling call
0709 13:04:07.987224 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:07.987700 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:08.041655 (+ 53955us) log.cc:844] Serialized 14 byte log entry
0709 13:04:08.386016 (+344361us) log.cc:844] Serialized 14 byte log entry
0709 13:04:08.392839 (+  6823us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:09.346645 (+953806us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x7b2c00077ef8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:09.361695 (+ 15050us) log.cc:844] Serialized 26 byte log entry
0709 13:04:09.362031 (+   336us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:09.722577 (+360546us) log.cc:844] Serialized 14 byte log entry
0709 13:04:10.101596 (+379019us) spinlock_profiling.cc:243] Waited 6.48 ms on lock 0x7b1800416260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdf1eb158f 00007fcdeaef0571 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd
0709 13:04:10.332313 (+230717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.332423 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:10.369838 (+ 37415us) raft_consensus.cc:1685] finished
0709 13:04:10.369866 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.370054 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2322832,"prepare.queue_time_us":415274,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":3261,"raft.queue_time_us":638696,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":31641,"spinlock_wait_cycles":35163136,"thread_start_us":645605,"threads_started":2,"wal-append.queue_time_us":491463}
I20260709 13:04:10.593320  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.594964  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.596303  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.597251  1740 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:10.250203  8490 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:10.598140  8522 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.208s	user 0.000s	sys 0.001s
I20260709 13:04:10.561887  8486 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.601121  8486 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.043576  8356 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45372 (request call id 20) took 9158 ms. Trace:
I20260709 13:04:10.252615  8481 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:10.478080  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.428745  8085 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.451634  8367 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:10.483208  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.484727  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:10.481894  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.486788  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:10.495962  8504 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.487520  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.492223  8397 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.655750  1659 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:10.421326  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:10.492614  8410 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:09.961124  8476 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.498203  1742 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:09.945865  8348 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:10.501556  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.251660  8425 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:10.506428  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:09.962536  8212 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.524s
I20260709 13:04:10.521986  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.578878  8332 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.473615  8466 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:10.547765  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.542622  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:10.674331  1664 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:10.675222  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.577142  8432 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.445580  8513 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.491911  8387 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.496008  8398 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.520944  8400 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:10.702724  8350 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:10.703277  8400 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.467855  8018 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.569442  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.584419  8351 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:10.407872  8320 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.677s	user 0.011s	sys 0.040s
I20260709 13:04:10.601842  8179 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.602842  8512 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.564849  8475 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.566742  8524 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.567370  8407 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.606464  8431 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.606379  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.567898  7939 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.543732  8411 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 13:04:10.368196  8487 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.573555  8458 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:10.561223  8339 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.609s	user 0.027s	sys 0.008s
I20260709 13:04:10.620332  1738 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:10.727576  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.727876  1663 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:10.728219  1738 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:10.578106  7945 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:10.618445  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:10.622704  1661 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:04:10.511811  8443 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.660049  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.733115  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1131) took 1540 ms. Trace:
I20260709 13:04:10.572510  8390 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:10.665690  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 244342us
I20260709 13:04:10.667517  1662 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:10.737398  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:10.507911  1733 rpcz_store.cc:269] 0709 13:04:07.986243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:07.993092 (+  6849us) service_pool.cc:224] Handling call
0709 13:04:10.507264 (+2514172us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:10.507384 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1033144192}
I20260709 13:04:10.620975  8367 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:10.672428  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:10.742865  1737 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:10.675936  1664 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:10.542703  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:10.676416  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 396396us
I20260709 13:04:10.661010  8410 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.684813  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.756815  1666 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:10.686205  8389 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.609055  8516 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:10.676345  8502 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.649s	user 0.016s	sys 0.002s
I20260709 13:04:10.576691  8494 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.731858  8458 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.583297  8528 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.731024  8339 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:10.663193  8425 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:10.780306  8425 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.675130  8471 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:10.781791  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:10.782066  8471 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:10.726259  8342 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.782493  1590 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:10.782575  8471 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:10.783286  8342 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.622712  8085 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.772565  8536 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.785290  8536 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.785775  8535 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:10.786466  8085 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.675684  8486 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.787235  8486 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.787585  8536 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.675297  8348 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:10.718171  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 270373us
I20260709 13:04:10.817138  1590 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:10.793896  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:10.818485  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:10.717659  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 291275us
I20260709 13:04:10.708971  8445 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.720398  1739 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:10.830796  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:10.713683  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 185478us
W20260709 13:04:10.833192  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:10.754065  8398 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:10.738616  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.671696  1736 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:10.739883  1659 rpcz_store.cc:276] 0709 13:04:09.193369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.732274 (+1538905us) service_pool.cc:224] Handling call
0709 13:04:10.732590 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.732775 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.732807 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:10.732862 (+    55us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:10.732905 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.732958 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.733061 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:10.751040  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:10.845427  1591 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:10.751837  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:10.846534  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94796.4us
I20260709 13:04:10.758343  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.847662  1737 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:10.758992  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:10.764125  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.849085  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:04:10.769204  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:10.850014  1738 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:10.772804  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:10.850888  1586 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:10.773747  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:10.851884  1585 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:10.775290  1665 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16.
I20260709 13:04:10.776937  8502 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.779299  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:10.854372  1587 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:10.786721  8535 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Stepping down as leader of term 14
I20260709 13:04:10.623997  8481 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.790148  8339 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.790755  1735 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:10.791419  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:10.764743  8378 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.756253  8417 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:10.794170  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:10.869989  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 75882.1us
I20260709 13:04:10.781783  8425 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:10.766669  8025 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 13:04:10.737534  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1132) took 1543 ms. Trace:
I20260709 13:04:10.789373  8486 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.763764  8410 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:10.860189  1737 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:10.854460  8398 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:10.841872  8534 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:10.857314  8517 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:10.865442  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.792524  8494 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.832078  8411 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 13:04:10.774176  8389 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.923085  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:10.873342  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:10.882807  8528 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.863528  1733 raft_consensus.cc:2955] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Committing config change with OpId 15.10: config changed from index 4 to 10, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:10.931874  1585 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:10.891088  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:10.862604  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 356160us
W20260709 13:04:10.893478  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64263.7us
I20260709 13:04:10.934242  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:10.782716  8387 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260709 13:04:10.842181  8490 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:10.756024  8391 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:10.954432  8391 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:10.954742  8432 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.905128  8398 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:10.902832  8348 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.861388  8535 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:10.909719  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.914407  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:10.920634  1587 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:10.921682  8509 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.886246  1662 rpcz_store.cc:276] 0709 13:04:09.193879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.736694 (+1542815us) service_pool.cc:224] Handling call
0709 13:04:10.737009 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.737203 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.737228 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:10.737270 (+    42us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:10.737314 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.737367 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:10.737459 (+    92us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:10.925331  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132088us
W20260709 13:04:10.991614  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57475us
I20260709 13:04:10.927937  2313 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:10.928239  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:10.903849  8351 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.935905  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.003721  1738 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:11.005096  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:10.936551  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:10.913290  8486 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 13:04:10.942273  8541 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:10.943333  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:10.947464  8546 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.854005  8445 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.948172  8476 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:10.934897  8531 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.903870  8212 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:10.903066  8542 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:10.741199  8487 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.617754  8356 rpcz_store.cc:276] 0709 13:04:00.782152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:00.782286 (+   134us) service_pool.cc:224] Handling call
0709 13:04:02.137891 (+1355605us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b58005336d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd870a 00007fcdedfd976a 00007fcdedfd891b 00007fcdedfe403c 00007fcdedfe36b3 00007fcdedfdb9fa 00007fcdf2fa34c9 00007fcdec51fc54
0709 13:04:02.300427 (+162536us) log.cc:844] Serialized 501 byte log entry
0709 13:04:03.289590 (+989163us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b54007f0e20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf56142 00007fcdedf36eaa 00007fcdedf3dd53 00007fcdedf3dcd9 00007fcdedf3dc69 00007fcdedf3dc31 00007fcdedf3cdfd 000000000038f354 000000000038f199 00007fcdeaef0475
0709 13:04:09.940655 (+6651065us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3215,"raft.run_cpu_time_us":5183,"raft.run_wall_time_us":861502,"spinlock_wait_cycles":248353024,"thread_start_us":233440,"threads_started":1,"wal-append.queue_time_us":367313,"wal-append.run_cpu_time_us":2726,"wal-append.run_wall_time_us":1023554}
I20260709 13:04:10.957135  2318 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:10.855422  8537 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20260709 13:04:10.912011  8517 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:10.984683  1735 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:10.891552  8378 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:10.992697  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:10.994674  2316 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:11.007306  2309 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:10.939349  8397 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:10.949414  8475 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:11.013347  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77529.1us
I20260709 13:04:10.955009  8391 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:10.895195  8367 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.013576  8417 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:11.029336  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:11.075573  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46312.9us
I20260709 13:04:11.036932  8407 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:11.081609  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 89003.8us
I20260709 13:04:11.010375  8509 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.016594  8332 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:11.068205  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:11.064311  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.021126  8546 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.082973  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.083243  8550 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.085615  8475 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:11.134791  8475 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:11.134522  8550 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.135821  8475 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.136265  8487 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:11.136857  8550 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:11.088732  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:11.077332  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:11.138123  1738 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:04:11.138002  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60743.6us
I20260709 13:04:11.059852  8517 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:11.124532  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:11.139987  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.138782  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:11.145246  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 77284.2us
I20260709 13:04:11.077044  7945 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.149763  1584 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:11.151407  8391 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.154328  8556 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:11.027961  8531 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.155826  8550 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.155225  8556 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.073026  8513 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:11.157795  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74893.6us
I20260709 13:04:11.162628  1659 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:11.033712  8212 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.070456  8557 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:11.080056  8390 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:11.130944  8367 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:11.168836  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.170992  1584 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:11.171113  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:11.123179  8432 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:11.133164  7939 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:11.144572  8494 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.165565  8391 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:11.033087  8542 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:11.164711  8397 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:11.212138  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.165907  8557 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.219991  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:11.163944  8556 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:11.173357  8332 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:11.224661  1738 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:11.202678  8387 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:11.202811  8512 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.173413  8486 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 13:04:11.207346  8481 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:11.165050  8550 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.208832  8432 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:11.247586  8432 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.255832  8543 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:11.262915  8509 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.264643  8342 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.183169  8487 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 13:04:11.220762  8391 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.209080  8516 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.226105  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.227931  7945 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.226032  8502 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.280349  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1135) took 1307 ms. Trace:
I20260709 13:04:11.281633  1661 rpcz_store.cc:276] 0709 13:04:09.973303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.749128 (+775825us) service_pool.cc:224] Handling call
0709 13:04:10.749491 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:10.892800 (+143309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.892829 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:10.892872 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:10.892912 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.280003 (+387091us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400117954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:11.280070 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.280264 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":322147712}
I20260709 13:04:11.228327  8557 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:11.285284  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:11.287343  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61380.4us
I20260709 13:04:11.288430  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:11.223649  8537 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 13:04:11.224244  8494 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:11.297575  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:11.297407  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:11.265316  8542 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:11.245076  8389 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:11.272631  8391 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:11.270764  8332 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 13:04:11.292022  8541 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:11.347596  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.351379  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.358909  2317 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:11.361964  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.367611  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.328890  8502 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:11.368634  2311 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:11.389173  8398 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:11.390174  8212 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:11.394265  8539 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.344s	user 0.001s	sys 0.001s
W20260709 13:04:11.395808  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.380679  8552 raft_consensus.cc:2764] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 13:04:11.355474  8471 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.572s	user 0.015s	sys 0.000s
I20260709 13:04:11.294162  8417 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 13:04:11.419183  8085 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.005s	sys 0.060s
W20260709 13:04:11.513870  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 347927us
I20260709 13:04:11.538559  8378 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:11.562898  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3182) took 3170 ms. Trace:
W20260709 13:04:11.543609  8561 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 13:04:11.584965  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:11.571127  1660 rpcz_store.cc:276] 0709 13:04:08.393611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.393748 (+   137us) service_pool.cc:224] Handling call
0709 13:04:08.394135 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:09.050893 (+656758us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b18004181e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfd5bfb 00007fcdedfe064b
0709 13:04:09.051175 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:10.342070 (+1290895us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b18004181e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf
0709 13:04:10.342325 (+   255us) log.cc:844] Serialized 14 byte log entry
0709 13:04:10.724323 (+381998us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:10.725307 (+   984us) log.cc:844] Serialized 26 byte log entry
0709 13:04:10.725544 (+   237us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:10.725595 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:10.725705 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:11.562624 (+836919us) raft_consensus.cc:1685] finished
0709 13:04:11.562649 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.562797 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3220,"prepare.queue_time_us":674437,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":139303,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":51329,"spinlock_wait_cycles":75966720,"thread_start_us":1092303,"threads_started":3,"wal-append.queue_time_us":322984}
W20260709 13:04:11.612154  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72818.2us
I20260709 13:04:11.446424  8541 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:11.533329  8085 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.005s	sys 0.060s
I20260709 13:04:11.522374  8417 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:11.746776  1660 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:04:11.747670  1734 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:11.750547  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:11.752687  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:11.752017  1557 connection.cc:256] RPC call timeout handler was delayed by -2.9944054110000025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15360000000000001s for a response
W20260709 13:04:11.794076  1557 connection.cc:256] RPC call timeout handler was delayed by -2.9314054110000143s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16060000000000002s for a response
W20260709 13:04:11.794509  1557 connection.cc:256] RPC call timeout handler was delayed by -2.6770831629999634s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
I20260709 13:04:11.763108  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:11.794885  1557 connection.cc:256] RPC call timeout handler was delayed by -2.5348831629999609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21660000000000001s for a response
W20260709 13:04:11.795367  1557 connection.cc:256] RPC call timeout handler was delayed by -2.3717154960000357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
W20260709 13:04:11.795746  1557 connection.cc:256] RPC call timeout handler was delayed by -2.1874831629999676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25520000000000004s for a response
I20260709 13:04:11.795825  1589 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:11.551450  8546 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.801662  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:11.558666  8553 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 13:04:11.804919  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:11.807675  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:11.796180  1557 connection.cc:256] RPC call timeout handler was delayed by -1.6726831629999879s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3124s for a response
I20260709 13:04:11.792626  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3188) took 1312 ms. Trace:
I20260709 13:04:11.812603  1660 rpcz_store.cc:276] 0709 13:04:10.480087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.745763 (+1265676us) service_pool.cc:224] Handling call
0709 13:04:11.746345 (+   582us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:11.792274 (+ 45929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.792563 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:11.828322  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:11.829710  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:11.830842  1591 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:11.841465  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:11.751582  8556 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.514s	user 0.011s	sys 0.000s
I20260709 13:04:11.829625  1592 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:04:11.894378  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 82577.2us
I20260709 13:04:11.896260  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:11.902798  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 85204.2us
I20260709 13:04:11.908761  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:11.908769  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.909970  2310 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:11.904601  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:11.913053  1735 raft_consensus.cc:2376] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17. Re-sending same reply.
I20260709 13:04:11.558976  8398 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:11.918511  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:11.919287  2316 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:11.523685  8471 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:11.726006  8550 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:11.797753  8212 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:11.893208  8509 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.803278  8417 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:11.931829  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:11.956295  1593 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:11.956413  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:11.946136  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:11.945698  1661 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.014s	sys 0.014s
W20260709 13:04:11.958072  1661 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.015s	sys 0.014s
I20260709 13:04:11.951769  2310 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:11.797397  8502 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.563750  8468 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:11.959138  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3184) took 1491 ms. Trace:
I20260709 13:04:11.892740  8552 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:11.791865  8541 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:11.933892  7430 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.018s	sys 0.049s
I20260709 13:04:11.913296  8543 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:12.030144  1662 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:12.029764  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.060899  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.062045  1593 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
W20260709 13:04:12.054306  7430 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.018s	sys 0.049s
I20260709 13:04:12.066722  2310 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 13:04:12.072695  1634 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.005s	sys 0.029s
W20260709 13:04:12.073688  1634 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.006s	sys 0.029s
W20260709 13:04:12.074225  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 935101us
W20260709 13:04:12.075212  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:11.912673  8556 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:12.081712  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 125346us
I20260709 13:04:11.743983  8432 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.095387  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.096381  1733 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:11.972781  8378 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:12.065330  8410 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.154s	user 0.010s	sys 0.005s
I20260709 13:04:12.114125  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.120086  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.111122  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2483) took 1903 ms. Trace:
I20260709 13:04:12.150487  1733 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:12.175756  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.087961  8550 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.086880  8085 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.185297  8549 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.185885  8567 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.029735  8502 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:12.189762  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 267819us
I20260709 13:04:12.030117  8552 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.183230  1587 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:11.562909  8535 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.212694  8566 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:12.234529  8535 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:11.986969  8509 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.190611  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.171087  1593 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:04:12.260591  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1171) took 1855 ms (client timeout 1524 ms). Trace:
I20260709 13:04:12.277379  1733 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:12.262861  1557 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.700s (SENT)
W20260709 13:04:12.260586  1632 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.821s (SENT) [suppressed 11 similar messages]
W20260709 13:04:12.277657  1742 rpcz_store.cc:269] 0709 13:04:10.405268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.434045 (+ 28777us) service_pool.cc:224] Handling call
0709 13:04:12.260354 (+1826309us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.260501 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122758016}
I20260709 13:04:12.172809  8378 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.199074  8494 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:12.214396  8554 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.005s	user 0.038s	sys 0.157s
I20260709 13:04:12.276939  1664 rpcz_store.cc:276] 0709 13:04:10.207815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.777159 (+569344us) service_pool.cc:224] Handling call
0709 13:04:10.777578 (+   419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:10.853613 (+ 76035us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:10.887455 (+ 33842us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:10.907600 (+ 20145us) log.cc:844] Serialized 52 byte log entry
0709 13:04:11.079538 (+171938us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:11.079597 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.079695 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.110769 (+1031074us) raft_consensus.cc:1685] finished
0709 13:04:12.110803 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.111027 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12344,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":215238,"threads_started":2,"wal-append.queue_time_us":284639}
I20260709 13:04:12.234841  8566 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:12.289675  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.355037  1587 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:12.198450  8450 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.283618  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:12.308660  8554 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.110s	user 0.042s	sys 0.158s
I20260709 13:04:12.234961  8350 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:12.307430  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.363580  1733 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:12.202339  8085 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.030467  8212 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.350870  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.088755  8332 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.236s
I20260709 13:04:12.281917  8546 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:12.291675  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.199245  8550 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:12.361799  1734 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.567s	user 0.008s	sys 0.004s
I20260709 13:04:12.389951  1734 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
W20260709 13:04:12.356320  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.390909  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.356177  1742 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:12.390974  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.392496  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.392930  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 12 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.393154  1742 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:12.394793  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.395792  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:12.202878  8549 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.211854  8567 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.356098  8577 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:12.363051  8425 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.479s	user 0.009s	sys 0.019s
I20260709 13:04:12.353332  8348 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:12.411159  8348 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.389305  8546 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.353977  8349 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.398233  8471 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.390055  2316 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:12.374653  1593 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:12.290103  8410 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.363229  8350 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:12.403440  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 111851us
I20260709 13:04:12.404755  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.432709  1738 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.434991  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.418390  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3323) took 1628 ms. Trace:
I20260709 13:04:12.435753  1584 rpcz_store.cc:276] 0709 13:04:10.790493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.791049 (+   556us) service_pool.cc:224] Handling call
0709 13:04:12.418199 (+1627150us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b540033aed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.418320 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":648585600}
I20260709 13:04:12.436488  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:12.413447  8559 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.186s	user 0.000s	sys 0.011s
W20260709 13:04:12.427167  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.420885  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.442628  1589 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:12.419976  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.428395  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.444607  1586 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:12.437911  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.435014  1742 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:12.447136  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.426565  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:12.445641  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.449627  1586 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:12.450197  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:12.452036  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.456664  1740 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:12.457051  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:12.437060  8517 consensus_meta.cc:287] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.268s	user 0.015s	sys 0.000s
I20260709 13:04:12.413575  8465 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.455423  8500 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:12.420161  8432 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.423257  8502 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.419049  8567 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.422601  8378 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:12.454274  8557 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.157s	user 0.013s	sys 0.001s
W20260709 13:04:12.453797  8391 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.109s	user 0.000s	sys 0.026s
I20260709 13:04:12.474893  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.488726  1593 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:12.429613  8443 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.422973  8431 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:12.414813  8509 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.463644  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.457458  8398 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:12.518697  8398 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.519375  8398 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:12.518570  8417 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.458194  8179 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:12.491416  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.523281  8179 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:12.524250  8179 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.525357  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.526015  1587 raft_consensus.cc:2368] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16.
I20260709 13:04:12.494833  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.531530  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.538115  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.546795  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.547955  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.491498  8554 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:12.460520  8425 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.457209  8348 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.490494  8085 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.477811  8465 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.487020  8500 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:12.515829  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.609771  1593 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:12.491451  8557 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.517530  8431 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.553200  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.616792  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.617990  1742 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:12.619205  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.549352  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.623288  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:12.623499  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74211.6us
I20260709 13:04:12.557996  8580 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:12.558622  8567 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:12.559522  8573 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
W20260709 13:04:12.560482  8572 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
W20260709 13:04:12.561182  8575 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.195s	user 0.000s	sys 0.001s
I20260709 13:04:12.563380  8537 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:12.528542  8570 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 13:04:12.496054  1738 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:12.359206  8566 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.423746  8577 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.533139  8502 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:12.476572  8517 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.543340  8432 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:12.628656  8567 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.550691  8509 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:12.497648  8391 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.574498  8425 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.630004  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.624694  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.608390  8552 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.638147  1735 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:12.638863  1734 raft_consensus.cc:2410] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 15 because replica has last-logged OpId of term: 14 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 13.
I20260709 13:04:12.638926  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.493150  8350 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:12.640677  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 379983us
I20260709 13:04:12.657060  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.637583  8470 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.644163  8502 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.629304  8580 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.633152  8085 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:12.634105  8537 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:12.626809  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 190376us
I20260709 13:04:12.717408  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:12.658735  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74574.5us
I20260709 13:04:12.659489  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:12.720079  1742 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:12.640353  8417 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:12.665433  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.624723  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.710235  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2491) took 1802 ms. Trace:
W20260709 13:04:12.712081  2281 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.524s (SENT)
I20260709 13:04:12.543591  8378 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:12.650449  8567 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:12.717157  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.731907  1593 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:12.730131  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:12.725322  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:12.734449  1632 connection.cc:256] RPC call timeout handler was delayed by -2.2469733899999937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2763s for a response
W20260709 13:04:12.735074  1632 connection.cc:256] RPC call timeout handler was delayed by -2.198373389999972s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2817s for a response
W20260709 13:04:12.735474  1632 connection.cc:256] RPC call timeout handler was delayed by -2.1503316700000141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1985s for a response
I20260709 13:04:12.712738  8387 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.735394  1664 rpcz_store.cc:276] 0709 13:04:10.908375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.295694 (+1387319us) service_pool.cc:224] Handling call
0709 13:04:12.353588 (+ 57894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.353878 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:12.357484 (+  3606us) log.cc:844] Serialized 14 byte log entry
0709 13:04:12.653876 (+296392us) log.cc:844] Serialized 14 byte log entry
0709 13:04:12.709592 (+ 55716us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:12.709706 (+   114us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:12.709796 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.709912 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.710135 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7924,"prepare.queue_time_us":10264,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":2748,"raft.queue_time_us":64030,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":5077,"spinlock_wait_cycles":43988736,"thread_start_us":293615,"threads_started":1}
I20260709 13:04:12.712355  8212 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.712818  8535 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.714428  8585 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:12.721473  8350 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:12.715492  8582 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:12.719321  8502 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:12.748255  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.720033  8580 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:12.723758  8537 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:12.716776  8085 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.737392  8349 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:12.728237  8550 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.761469  1593 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
W20260709 13:04:12.778627  8581 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 13:04:12.781994  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2488) took 1946 ms. Trace:
I20260709 13:04:12.782527  8465 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.783159  8557 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:12.755899  1632 connection.cc:256] RPC call timeout handler was delayed by -1.9568316699999855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
I20260709 13:04:12.782390  1660 rpcz_store.cc:276] 0709 13:04:10.835104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.910931 (+1075827us) service_pool.cc:224] Handling call
0709 13:04:11.911402 (+   471us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:11.911727 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:11.911766 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:11.922440 (+ 10674us) log.cc:844] Serialized 26 byte log entry
0709 13:04:12.528444 (+606004us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:12.528487 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:12.528699 (+   212us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:12.781594 (+252895us) raft_consensus.cc:1685] finished
0709 13:04:12.781617 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:12.781923 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":438519,"threads_started":1,"wal-append.queue_time_us":604343}
I20260709 13:04:12.714859  8500 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.766556  8582 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:12.762584  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45243.2us
I20260709 13:04:12.717485  8470 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.740698  8410 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.775584  1742 raft_consensus.cc:2368] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:12.772854  8085 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:12.772791  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91599us
I20260709 13:04:12.800079  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.759465  8585 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.770753  8417 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:12.794692  8431 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.801136  1593 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:12.756816  8387 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.807154  8470 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.813651  8212 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:12.762001  8535 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:12.820180  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.819640  8212 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:12.829161  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:11.973716  1661 rpcz_store.cc:276] 0709 13:04:10.467630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.284091 (+816461us) service_pool.cc:224] Handling call
0709 13:04:11.284527 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.284815 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:11.370949 (+ 86134us) log.cc:844] Serialized 14 byte log entry
0709 13:04:11.958658 (+587709us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:11.958733 (+    75us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:11.958806 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:11.958904 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:11.959066 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":102475,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":14582,"thread_start_us":598052,"threads_started":2}
I20260709 13:04:12.829910  1590 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:12.829753  8212 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.830873  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.831297  8212 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:12.831507  1590 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:12.833158  8445 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:12.834159  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2485) took 2379 ms (client timeout 2051 ms). Trace:
W20260709 13:04:12.834612  1665 rpcz_store.cc:269] 0709 13:04:10.454759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.484507 (+ 29748us) service_pool.cc:224] Handling call
0709 13:04:12.833897 (+2349390us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.834054 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352958336}
I20260709 13:04:12.835500  8517 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.836364  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.836948  8566 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.837200  1591 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 13:04:12.815805  8458 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.838158  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.838884  1591 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:12.840580  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.811995  8585 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:12.841778  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.842283  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.842657  1591 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:04:12.842962  1741 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:12.843989  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.774477  8549 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:12.848030  8425 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:12.849910  8505 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:12.850411  8505 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.858547  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:12.812330  8417 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:12.859544  8417 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.860836  8349 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:12.862104  8592 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:12.845294  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.863739  1585 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:12.831353  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2536) took 2463 ms (client timeout 1700 ms). Trace:
W20260709 13:04:12.865165  1736 rpcz_store.cc:269] 0709 13:04:10.368298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.368494 (+   196us) service_pool.cc:224] Handling call
0709 13:04:12.831131 (+2462637us) spinlock_profiling.cc:243] Waited 258 ms on lock 0x7b5400352fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.831275 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":593727872}
W20260709 13:04:12.864615  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1189) took 1974 ms (client timeout 2152 ms). Trace:
I20260709 13:04:12.798640  8582 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:12.866703  2318 rpcz_store.cc:269] 0709 13:04:10.890577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.890796 (+   219us) service_pool.cc:224] Handling call
0709 13:04:12.864398 (+1973602us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b5400570654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.864545 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":53884800}
I20260709 13:04:12.870980  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.891692  1585 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:12.868093  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:12.871773  1590 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:12.873564  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:12.877396  8524 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:12.891737  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.881265  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.883057  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.897022  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:12.895072  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:12.897964  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.890363  1591 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:12.899791  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.879391  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.900767  1588 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:12.899938  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.903995  1591 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:12.897830  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.904228  1584 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:12.908015  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.908965  1584 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:12.909694  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.905452  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:12.911530  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.915757  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.916293  1585 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:04:12.921288  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.921465  1588 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:04:12.916872  1589 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 17 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 13:04:12.916646  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.915867  8367 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:12.863808  8387 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:12.903682  8468 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.873s	user 0.002s	sys 0.012s
I20260709 13:04:12.927464  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.928825  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:12.927857  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:12.900823  1593 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:12.932083  1590 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:12.919800  8465 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:12.935261  8465 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.936853  8589 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:12.926630  8367 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.937484  8589 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.938138  8367 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:12.926811  8332 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:12.945736  8387 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:12.946321  8387 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.925283  8579 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.947296  8579 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:12.948323  2312 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 991) took 3213 ms (client timeout 2041 ms). Trace:
I20260709 13:04:12.948712  8597 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:04:12.948912  2312 rpcz_store.cc:269] 0709 13:04:09.735240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:09.735452 (+   212us) service_pool.cc:224] Handling call
0709 13:04:12.948077 (+3212625us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54006a0ed4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:12.948224 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1536747904}
I20260709 13:04:12.903115  7945 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:12.946767  8387 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:12.915521  8458 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:12.950039  8597 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:12.953737  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:12.920356  8425 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:12.957614  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 99127.9us
I20260709 13:04:12.959643  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.960091  1593 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:04:12.964221  1590 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:12.932175  8566 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:12.963310  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:12.960783  8425 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:12.968593  8349 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:12.968914  8425 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:12.964007  8468 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:12.971140  8468 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:12.971524  8468 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:12.963820  8458 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:12.974411  8505 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.974714  8458 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.924391  8398 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.966065  1584 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:04:12.978174  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.977550  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:12.960404  8597 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:12.919602  8542 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:04:12.955298  8540 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:12.954766  7945 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:13.010934  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:12.969172  8349 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:13.014812  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:13.013552  7945 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.023518  8505 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.024978  8398 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:13.028199  8387 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:12.959367  8085 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:13.029053  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.011229  8580 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.041127  8350 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:13.047768  8350 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:13.048470  8085 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.049078  8580 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:13.031700  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.018174  8540 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.034886  8417 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.036051  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2486) took 2569 ms. Trace:
I20260709 13:04:13.022264  8349 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.037892  8541 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.038033  8597 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.038492  8367 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:12.988046  8566 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.007810  8577 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.011911  8592 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:12.967374  8332 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:13.043066  8398 raft_consensus.cc:2764] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:04:13.045609  8535 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.042872  8517 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:13.078168  8517 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:13.079243  8517 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:13.081099  8594 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 13:04:13.049237  8350 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.013051  8554 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:13.081929  8554 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:13.082405  8554 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:13.082685  8350 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:13.072753  8566 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.052538  8502 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:13.079253  8555 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.863s	user 0.004s	sys 0.013s
I20260709 13:04:13.089112  8387 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.095103  8585 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.076735  8597 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.100361  8425 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.102368  8417 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.102910  8502 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.103547  8504 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:13.102922  8367 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.104669  8589 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.105134  8531 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.107188  8537 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.122093  8537 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.124912  8567 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.125782  8585 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.126255  8546 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:13.127533  8560 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.842s	user 0.025s	sys 0.015s
I20260709 13:04:13.127640  8567 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.129021  8475 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.128422  8468 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.124357  8579 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.124054  8540 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.136600  8556 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.129319  8582 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.136075  8475 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.138937  8475 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.124847  8554 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.139271  8582 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.141404  8554 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.139029  8566 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.144482  8566 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.191833  1585 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:13.197825  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:13.194535  1632 connection.cc:256] RPC call timeout handler was delayed by -1.7891070360000185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21400000000000002s for a response
W20260709 13:04:13.199864  1632 connection.cc:256] RPC call timeout handler was delayed by -1.7471316699999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2433s for a response
W20260709 13:04:13.200371  1632 connection.cc:256] RPC call timeout handler was delayed by -1.5830516029999444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
W20260709 13:04:13.200829  1632 connection.cc:256] RPC call timeout handler was delayed by -1.1681516029999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2041s for a response
I20260709 13:04:13.202371  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.204710  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.060388  1659 rpcz_store.cc:276] 0709 13:04:10.467030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.017892 (+550862us) service_pool.cc:224] Handling call
0709 13:04:11.018355 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:11.167067 (+148712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.035756 (+1868689us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b540013ddd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:13.035981 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":316861696}
W20260709 13:04:13.208833  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3259) took 2809 ms (client timeout 1727 ms). Trace:
I20260709 13:04:13.219957  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:13.219980  1739 rpcz_store.cc:269] 0709 13:04:10.399172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.399440 (+   268us) service_pool.cc:224] Handling call
0709 13:04:13.208182 (+2808742us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b540012d9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.208479 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":629854080}
I20260709 13:04:13.221912  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3270) took 1144 ms. Trace:
I20260709 13:04:13.224079  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:13.225085  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.222621  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:13.039644  1667 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 13:04:13.222493  1735 rpcz_store.cc:276] 0709 13:04:12.077420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.077734 (+   314us) service_pool.cc:224] Handling call
0709 13:04:12.078147 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.655671 (+577524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.221152 (+565481us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400284454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:13.221668 (+   516us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1286895872}
I20260709 13:04:13.230548  1739 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:13.232223  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:13.232962  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3183) took 2765 ms. Trace:
I20260709 13:04:13.210273  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.214570  1733 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:13.201349  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0439516029999822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21789999999999998s for a response
I20260709 13:04:13.216886  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1143) took 3007 ms. Trace:
I20260709 13:04:13.218510  1741 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:13.219323  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3271) took 1125 ms. Trace:
W20260709 13:04:13.052997  1740 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1175) took 2610 ms (client timeout 1997 ms). Trace:
I20260709 13:04:13.220930  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:13.038688  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1138) took 3064 ms (client timeout 1929 ms). Trace:
I20260709 13:04:13.049293  1736 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:13.107723  7945 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.259047  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.258124  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:13.269132  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.233476  1735 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.255497  1734 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:13.264812  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.274983  2318 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:13.261276  1665 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260709 13:04:13.263378  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.259291  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.263185  1661 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:13.254251  1742 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:13.241542  1667 rpcz_store.cc:276] 0709 13:04:10.467382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:11.252219 (+784837us) service_pool.cc:224] Handling call
0709 13:04:13.039378 (+1787159us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:13.230367 (+190989us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.232519 (+  2152us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b540013ecd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.232730 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4107624704}
I20260709 13:04:13.233848  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1170) took 2812 ms. Trace:
I20260709 13:04:13.272866  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.254698  1738 rpcz_store.cc:276] 0709 13:04:12.094028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.094287 (+   259us) service_pool.cc:224] Handling call
0709 13:04:13.218711 (+1124424us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400404f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.219049 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":587874688}
I20260709 13:04:13.238283  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1107) took 1147 ms. Trace:
I20260709 13:04:13.276336  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:13.278048  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:13.273150  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.255755  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:13.277135  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.275803  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.266669  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.255904  1663 rpcz_store.cc:269] 0709 13:04:09.974175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.460508 (+486333us) service_pool.cc:224] Handling call
0709 13:04:13.038443 (+2577935us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.038606 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242009216}
I20260709 13:04:13.278929  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.248420  1557 leader_election.cc:400] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:13.382298  1557 leader_election.cc:403] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 13:04:13.247292  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:13.259250  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.264781  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.254455  1662 rpcz_store.cc:276] 0709 13:04:10.208938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.991739 (+782801us) service_pool.cc:224] Handling call
0709 13:04:10.992211 (+   472us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.053835 (+1061624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.216164 (+1162329us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000aa0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:13.216605 (+   441us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2385572224}
I20260709 13:04:13.278072  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.283756  8339 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:13.283980  8348 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:13.376619  1737 rpcz_store.cc:276] 0709 13:04:10.404971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.902938 (+497967us) service_pool.cc:224] Handling call
0709 13:04:13.215145 (+2312207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.215809 (+   664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.215837 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.215909 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:13.215961 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.216070 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.217074 (+  1004us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540012d9d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.217219 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5314386304}
I20260709 13:04:13.386359  8339 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:13.386579  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.282109  8179 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:13.281880  8182 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.277948  7945 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.284991  8398 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.260074  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:13.255275  1740 rpcz_store.cc:269] 0709 13:04:10.443029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:10.606046 (+163017us) service_pool.cc:224] Handling call
0709 13:04:13.052726 (+2446680us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54002030d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.052897 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3991687552}
I20260709 13:04:13.245127  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:13.378237  1587 rpcz_store.cc:276] 0709 13:04:12.090698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.386318 (+295620us) service_pool.cc:224] Handling call
0709 13:04:13.237707 (+851389us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5400431a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.238051 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308375936}
I20260709 13:04:13.269471  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet dfe67343bf1842888d6176ed2ffe1419 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 12)
I20260709 13:04:13.397907  1585 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:13.403800  1659 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.396498  8339 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:13.390090  8348 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:13.415295  2310 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 13:04:13.412781  2316 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:13.403484  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.409631  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.420446  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:13.420732  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.421547  2310 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:13.421985  8018 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.423354  2310 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 13:04:13.424612  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.425784  8182 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.426316  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.427296  1660 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:13.427937  8391 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.428272  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.428722  8182 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.421430  1668 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:13.430449  1667 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.431797  2317 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:13.433405  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:13.432200  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.436612  1737 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:13.413852  8348 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.455729  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2547) took 1050 ms. Trace:
I20260709 13:04:13.456234  1741 rpcz_store.cc:276] 0709 13:04:12.404931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.843717 (+438786us) service_pool.cc:224] Handling call
0709 13:04:13.455369 (+611652us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.455633 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":857538304}
I20260709 13:04:13.240623  1486 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 8 to 17, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 17 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:13.458230  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.460002  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.438127  1664 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:13.287566  8407 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.143s
I20260709 13:04:13.444710  8425 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:13.429777  8398 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.459987  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:13.448835  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.420503  8592 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.452374  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.437966  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:13.442765  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:13.437314  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.465116  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.465418  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 210060us
I20260709 13:04:13.466564  8556 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:13.468477  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.437148  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.469786  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:13.453116  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.482882  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:13.484287  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:13.485249  1586 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
W20260709 13:04:13.478121  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 231302us
I20260709 13:04:13.488703  1741 raft_consensus.cc:2368] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:13.489158  8367 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:13.489800  8367 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:13.490300  8367 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.490219  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 6 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.491987  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.496834  8556 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:13.480823  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:13.495448  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 154126us
I20260709 13:04:13.490309  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.447602  8465 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:13.492887  1741 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:13.506095  2316 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:13.509106  1663 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 17 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 6.
I20260709 13:04:13.509168  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.507546  1592 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:13.514143  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2470) took 5089 ms. Trace:
I20260709 13:04:13.504786  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.470834  8179 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:13.515762  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 267962us
I20260709 13:04:13.516594  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2557) took 1104 ms. Trace:
I20260709 13:04:13.523862  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.456290  8349 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.514707  1668 rpcz_store.cc:276] 0709 13:04:08.424457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:08.726513 (+302056us) service_pool.cc:224] Handling call
0709 13:04:08.726852 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.463805 (+4736953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.494106 (+ 30301us) spinlock_profiling.cc:243] Waited 4.56 s on lock 0x7b54002f3354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:13.514012 (+ 19906us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10485960448}
I20260709 13:04:13.522485  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.521622  1733 rpcz_store.cc:276] 0709 13:04:12.412613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.624471 (+211858us) service_pool.cc:224] Handling call
0709 13:04:13.503148 (+878677us) spinlock_profiling.cc:243] Waited 487 ms on lock 0x7b5400284954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.516512 (+ 13364us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1119965952}
I20260709 13:04:13.500342  8556 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:13.524169  1737 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:13.531755  1665 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.537968  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.536170  1742 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:04:13.539582  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2551) took 1131 ms. Trace:
I20260709 13:04:13.539840  1742 rpcz_store.cc:276] 0709 13:04:12.407899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.398554 (+990655us) service_pool.cc:224] Handling call
0709 13:04:13.539527 (+140973us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.541998  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.547611  8398 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.549207  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.550796  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.551514  1668 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:13.552502  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.557693  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.557732  8475 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.549356  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.558607  1587 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.559597  1586 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:13.560778  8411 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:13.545068  1738 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:13.562085  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.561357  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.560796  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.540855  1486 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 17, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 17 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:13.573230  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.573601  1588 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:13.575786  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.575559  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.532605  8556 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:13.558653  8475 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:13.582006  2280 connection.cc:256] RPC call timeout handler was delayed by -1.5205256479999889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
I20260709 13:04:13.584386  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.585417  1741 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:13.574203  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 304885us
I20260709 13:04:13.579413  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.561420  8411 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:13.586694  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57034.6us
I20260709 13:04:13.592029  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.594435  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 349657us
I20260709 13:04:13.588171  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2552) took 1177 ms. Trace:
I20260709 13:04:13.597211  1741 rpcz_store.cc:276] 0709 13:04:12.411038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.533127 (+1122089us) service_pool.cc:224] Handling call
0709 13:04:13.588090 (+ 54963us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.604468  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:13.604835  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.605758  8616 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:13.606807  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.608273  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.609364  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.610318  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112511us
I20260709 13:04:13.523808  8592 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.618114  8504 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:13.621982  1592 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:13.622164  1737 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 13:04:13.622987  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:13.623664  8616 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:13.624863  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.624703  8179 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.626039  8378 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:13.625929  8320 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:04:13.628360  1733 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 13:04:13.628436  8470 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.629819  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:13.630869  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.633481  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3340) took 1255 ms. Trace:
I20260709 13:04:13.658200  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3279) took 1289 ms. Trace:
W20260709 13:04:13.646190  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73504.3us
I20260709 13:04:13.638327  8540 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18.
I20260709 13:04:13.638942  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.638485  8411 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.640741  1741 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:13.641566  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.611187  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.642264  1586 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:13.594308  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.644708  1667 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:13.648464  8349 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.649585  8387 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:13.600625  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2561) took 1066 ms. Trace:
I20260709 13:04:13.658699  1737 rpcz_store.cc:276] 0709 13:04:12.369228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.560848 (+1191620us) service_pool.cc:224] Handling call
0709 13:04:13.658123 (+ 97275us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.662442  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2542) took 1268 ms. Trace:
W20260709 13:04:13.665709  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 405941us
I20260709 13:04:13.667212  1733 rpcz_store.cc:276] 0709 13:04:12.389577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.627167 (+1237590us) service_pool.cc:224] Handling call
0709 13:04:13.627737 (+   570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.655922 (+ 28185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.657857 (+  1935us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.672741  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.673564  1661 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16.
I20260709 13:04:13.666100  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.674782  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2540) took 1281 ms. Trace:
I20260709 13:04:13.676373  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.678809  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.666590  1736 rpcz_store.cc:276] 0709 13:04:12.432321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.866297 (+433976us) service_pool.cc:224] Handling call
0709 13:04:13.489344 (+623047us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b5400436054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:13.498615 (+  9271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":355962112}
I20260709 13:04:13.681772  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:13.684154  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53462.4us
I20260709 13:04:13.685175  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.684337  8504 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:13.687067  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.675868  1741 rpcz_store.cc:276] 0709 13:04:12.388782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.639647 (+1250865us) service_pool.cc:224] Handling call
0709 13:04:13.640286 (+   639us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.669407 (+ 29121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.669690 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.686321  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.674782  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3332) took 1487 ms. Trace:
I20260709 13:04:13.690426  8025 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:13.693404  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.694247  1661 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:13.695935  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.691385  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49226.7us
I20260709 13:04:13.690789  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.704668  1667 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:13.704963  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.705785  1592 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.706166  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.707581  1588 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:13.708859  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.710005  8517 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:13.711872  8468 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.712715  8182 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:13.672573  8470 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.713390  8182 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:13.714641  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.714856  8182 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.715755  8470 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:13.706247  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 691115us
I20260709 13:04:13.718238  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.718503  1733 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:13.719755  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.702477  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.717710  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 94899.5us
I20260709 13:04:13.717650  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:13.719429  1663 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:13.724045  1739 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:13.724612  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.728772  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.724700  1740 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:13.692581  8025 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:13.692646  8616 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:13.724176  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2545) took 1319 ms. Trace:
I20260709 13:04:13.734277  1733 rpcz_store.cc:276] 0709 13:04:12.404376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.674457 (+1270081us) service_pool.cc:224] Handling call
0709 13:04:13.695315 (+ 20858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.723874 (+ 28559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.724113 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.730970  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.728317  1588 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.723500  8411 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:13.703802  8504 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:13.732339  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3338) took 1414 ms. Trace:
I20260709 13:04:13.741019  1665 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:13.745976  8407 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:13.746521  8407 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:13.747360  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:13.745986  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116290us
I20260709 13:04:13.702134  8475 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.742558  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.689754  1591 rpcz_store.cc:276] 0709 13:04:12.034640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.912603 (+877963us) service_pool.cc:224] Handling call
0709 13:04:12.913023 (+   420us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:12.921085 (+  8062us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.921132 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:12.952040 (+ 30908us) log.cc:844] Serialized 52 byte log entry
0709 13:04:13.190522 (+238482us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:13.401093 (+210571us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.478677 (+ 77584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.503194 (+ 24517us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.503259 (+    65us) raft_consensus.cc:1685] finished
0709 13:04:13.503276 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.521967 (+ 18691us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100083,"prepare.queue_time_us":283766,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":118187,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":52092,"thread_start_us":417703,"threads_started":2,"wal-append.queue_time_us":173169}
W20260709 13:04:13.754616  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64021.6us
I20260709 13:04:13.733819  8025 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:13.755458  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.755124  1661 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:04:13.658579  1584 rpcz_store.cc:276] 0709 13:04:12.378244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.008218 (+629974us) service_pool.cc:224] Handling call
0709 13:04:13.008612 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.008880 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:13.432544 (+423664us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.632397 (+199853us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.632667 (+   270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.632743 (+    76us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:13.632861 (+   118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.632999 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.633210 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62762,"prepare.queue_time_us":467657,"prepare.run_cpu_time_us":161,"prepare.run_wall_time_us":8679,"raft.queue_time_us":110777,"raft.run_cpu_time_us":296,"raft.run_wall_time_us":294,"thread_start_us":261533,"threads_started":2,"wal-append.queue_time_us":183098}
I20260709 13:04:13.728845  8470 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:13.767143  1667 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:13.731647  1486 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 9 to 15, now has a pending config: VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194), VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195), VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:13.752799  1662 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:13.770721  1667 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:13.737061  8616 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:13.768606  1668 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:13.774881  8476 raft_consensus.cc:2764] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:13.779211  8602 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:13.779688  8602 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:13.782075  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.776460  1663 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:13.784483  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.785288  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.786710  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.769809  8604 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:04:13.743484  1593 rpcz_store.cc:276] 0709 13:04:12.174707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.961721 (+787014us) service_pool.cc:224] Handling call
0709 13:04:12.962040 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.962267 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:13.011170 (+ 48903us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.522787 (+511617us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.532215 (+  9428us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:13.565468 (+ 33253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.577940 (+ 12472us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.588808 (+ 10868us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":406520,"prepare.queue_time_us":48322,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":104866,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":20575,"thread_start_us":397739,"threads_started":1,"wal-append.queue_time_us":462132}
I20260709 13:04:13.783388  1660 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:13.786559  1737 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:13.775774  8616 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.790699  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.796187  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 133829us
I20260709 13:04:13.797843  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:13.798691  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.781599  8475 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:13.747810  8407 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.801687  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.797711  1738 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:13.806043  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:13.804023  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48544.8us
I20260709 13:04:13.805156  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2544) took 1407 ms. Trace:
I20260709 13:04:13.807930  1737 rpcz_store.cc:276] 0709 13:04:12.390186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.673113 (+1282927us) service_pool.cc:224] Handling call
0709 13:04:13.730117 (+ 57004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.792166 (+ 62049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.797417 (+  5251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.808898  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.806416  1741 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:13.811985  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3335) took 1584 ms. Trace:
I20260709 13:04:13.813215  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3277) took 1697 ms. Trace:
I20260709 13:04:13.813737  1738 rpcz_store.cc:276] 0709 13:04:12.110905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.602943 (+1492038us) service_pool.cc:224] Handling call
0709 13:04:13.725063 (+122120us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.807811 (+ 82748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.808028 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.816630  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2558) took 1403 ms. Trace:
I20260709 13:04:13.816955  1741 rpcz_store.cc:276] 0709 13:04:12.412958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.689608 (+1276650us) service_pool.cc:224] Handling call
0709 13:04:13.757084 (+ 67476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.816428 (+ 59344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.816584 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.815121  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:13.819713  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.820550  1740 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:13.814756  8476 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 13:04:13.806281  8475 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:13.826550  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 109036us
I20260709 13:04:13.826592  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.824972  1738 raft_consensus.cc:2368] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:13.821331  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2559) took 1408 ms. Trace:
I20260709 13:04:13.830629  1740 rpcz_store.cc:276] 0709 13:04:12.413251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.734724 (+1321473us) service_pool.cc:224] Handling call
0709 13:04:13.800279 (+ 65555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:13.820991 (+ 20712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.821252 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.828837  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.823786  1486 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 16 to 19, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 19 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:13.829592  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.804841  8592 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:13.835167  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3288) took 1272 ms. Trace:
I20260709 13:04:13.780890  8602 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:13.827680  1668 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 11.
I20260709 13:04:13.838330  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.839051  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:13.840456  8577 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:13.840899  8577 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:13.841276  8577 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:13.844551  8621 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:13.845840  1486 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:13.847899  1486 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:13.840451  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62773.5us
I20260709 13:04:13.852690  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:13.853622  8621 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:13.853951  8504 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.857067  8621 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:13.864022  1587 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:13.858846  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:13.864696  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.865025  1741 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:13.837953  1739 rpcz_store.cc:276] 0709 13:04:12.562851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.753086 (+1190235us) service_pool.cc:224] Handling call
0709 13:04:13.792632 (+ 39546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.834738 (+ 42106us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.834780 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.834834 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:13.834885 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.834960 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.835107 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13937,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8341}
W20260709 13:04:13.868655  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46781.9us
W20260709 13:04:13.869455  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 174738us
I20260709 13:04:13.870416  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:13.871181  1668 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:13.835831  8475 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:13.874966  1586 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:13.809597  8470 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:13.873931  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3339) took 1385 ms. Trace:
I20260709 13:04:13.877430  2314 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:13.882012  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.883020  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:13.873044  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:13.813642  1589 rpcz_store.cc:276] 0709 13:04:12.109207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.932829 (+823622us) service_pool.cc:224] Handling call
0709 13:04:12.933402 (+   573us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:12.938147 (+  4745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:12.938184 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:12.938890 (+   706us) log.cc:844] Serialized 52 byte log entry
0709 13:04:13.021474 (+ 82584us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:13.412395 (+390921us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.578116 (+165721us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.671396 (+ 93280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.675621 (+  4225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:13.689061 (+ 13440us) raft_consensus.cc:1685] finished
0709 13:04:13.689081 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.693951 (+  4870us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":318311,"prepare.queue_time_us":532592,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":5284,"raft.queue_time_us":199339,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":5017,"thread_start_us":280061,"threads_started":2,"wal-append.queue_time_us":133954}
I20260709 13:04:13.891323  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.892486  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3285) took 1330 ms. Trace:
I20260709 13:04:13.890630  1660 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:13.893047  1733 rpcz_store.cc:276] 0709 13:04:12.561799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.735239 (+1173440us) service_pool.cc:224] Handling call
0709 13:04:13.812893 (+ 77654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.882905 (+ 70012us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:13.882955 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.883019 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:13.892149 (+  9130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.892247 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.892410 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:13.876734  1557 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.393s (SENT)
I20260709 13:04:13.906490  1591 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:13.899722  1593 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:13.910625  1662 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 13:04:13.909260  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:13.920080  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:13.920389  1586 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:13.909619  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:13.883697  1486 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:13.927585  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107156us
I20260709 13:04:13.932158  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3331) took 1284 ms. Trace:
I20260709 13:04:13.932660  1586 rpcz_store.cc:276] 0709 13:04:12.636738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.878311 (+1241573us) service_pool.cc:224] Handling call
0709 13:04:13.920192 (+ 41881us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.920848 (+   656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.921107 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.918457  1737 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:13.946900  1738 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:13.944859  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:13.948215  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:13.951411  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:13.972383  1738 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 13:04:13.976003  8618 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 13:04:13.980283  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 183114us
W20260709 13:04:13.980537  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 112734us
I20260709 13:04:13.981228  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:13.985374  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.990782  1739 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:13.879108  1590 rpcz_store.cc:276] 0709 13:04:12.369949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:12.965092 (+595143us) service_pool.cc:224] Handling call
0709 13:04:12.965416 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:12.965680 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:13.476085 (+510405us) log.cc:844] Serialized 14 byte log entry
0709 13:04:13.686875 (+210790us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:13.695653 (+  8778us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:13.720559 (+ 24906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:13.733141 (+ 12582us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:13.755543 (+ 22402us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80672,"prepare.queue_time_us":431247,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":16513,"raft.queue_time_us":13419,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":138,"thread_start_us":532601,"threads_started":2,"wal-append.queue_time_us":245857}
I20260709 13:04:13.999460  8391 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:14.000177  8391 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:14.001284  8391 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.005090  1592 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:14.005677  1586 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:14.006047  8592 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:14.007166  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2571) took 1079 ms. Trace:
I20260709 13:04:14.008864  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1192) took 1444 ms. Trace:
I20260709 13:04:14.012730  1739 rpcz_store.cc:276] 0709 13:04:12.564125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.883317 (+1319192us) service_pool.cc:224] Handling call
0709 13:04:13.942400 (+ 59083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.991372 (+ 48972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.008801 (+ 17429us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.017527  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3335) took 1370 ms. Trace:
I20260709 13:04:14.017998  1586 rpcz_store.cc:276] 0709 13:04:12.638613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.004939 (+1366326us) service_pool.cc:224] Handling call
0709 13:04:14.005451 (+   512us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.008784 (+  3333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.008993 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:13.970842  8407 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.011891  1740 rpcz_store.cc:276] 0709 13:04:12.927878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.934653 (+1006775us) service_pool.cc:224] Handling call
0709 13:04:14.006392 (+ 71739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.006780 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.006810 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:14.006857 (+    47us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:14.006899 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.006969 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.007100 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.019836  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:13.986518  8018 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:14.025687  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:14.028133  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.029999  1587 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 13:04:14.030668  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 191682us
I20260709 13:04:14.030867  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1118) took 1396 ms. Trace:
I20260709 13:04:14.031474  1587 rpcz_store.cc:276] 0709 13:04:12.634445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.875365 (+1240920us) service_pool.cc:224] Handling call
0709 13:04:14.016904 (+141539us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.030501 (+ 13597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.030791 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.032480  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:14.033377  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:14.034026  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 157311us
I20260709 13:04:14.036718  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:14.037776  1486 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 12 to 14, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:14.040073  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:14.042057  1589 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:14.042922  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3332) took 1405 ms. Trace:
I20260709 13:04:14.043336  1589 rpcz_store.cc:276] 0709 13:04:12.637198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.904716 (+1267518us) service_pool.cc:224] Handling call
0709 13:04:14.010709 (+105993us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.042556 (+ 31847us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.042844 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.045037  1589 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:14.052825  8494 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:14.053946  8494 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:14.054311  8494 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.054677  8494 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:14.059729  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.061338  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:14.063373  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3344) took 1335 ms. Trace:
I20260709 13:04:14.066366  1589 rpcz_store.cc:276] 0709 13:04:12.728049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.044335 (+1316286us) service_pool.cc:224] Handling call
0709 13:04:14.044809 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.045459 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.063294 (+ 17835us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.007545  8349 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:14.094357  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.096046  1668 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:14.114995  1489 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 254) took 99810 ms (client timeout 15000 ms). Trace:
I20260709 13:04:14.082013  8476 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:14.116226  8476 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:14.116845  8476 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:14.122180  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88677.6us
I20260709 13:04:14.122918  8320 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:14.123406  8320 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:14.123870  8320 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.135906  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:14.169716  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.149039  1557 leader_election.cc:400] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:14.204408  1557 leader_election.cc:403] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:14.211428  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:14.211819  1733 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:14.214041  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:14.233614  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 134472us
W20260709 13:04:14.177886  8367 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.687s	user 0.010s	sys 0.006s
I20260709 13:04:14.236100  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2564) took 1303 ms. Trace:
I20260709 13:04:14.237942  1740 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 13:04:14.234920  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3290) took 1467 ms. Trace:
I20260709 13:04:14.239328  1733 rpcz_store.cc:276] 0709 13:04:12.909142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.946784 (+1037642us) service_pool.cc:224] Handling call
0709 13:04:14.099748 (+152964us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.212655 (+112907us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.212916 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.236461  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:14.276302  8543 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.277474  8543 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.235684  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.253649  1739 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:14.284224  1663 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
W20260709 13:04:14.288479  1659 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.850s	user 0.010s	sys 0.014s
W20260709 13:04:14.291034  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 188362us
I20260709 13:04:14.291841  1659 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:14.299028  1660 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:14.229218  8621 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:14.300117  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.300632  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2492) took 1035 ms. Trace:
I20260709 13:04:14.301044  1659 rpcz_store.cc:276] 0709 13:04:13.257862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.258164 (+   302us) service_pool.cc:224] Handling call
0709 13:04:14.293218 (+1035054us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:14.301631  1585 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.882s	user 0.016s	sys 0.000s
I20260709 13:04:14.307049  1586 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:14.307927  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:14.309140  1585 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:14.309787  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3342) took 1594 ms. Trace:
I20260709 13:04:14.334223  1586 rpcz_store.cc:276] 0709 13:04:12.714747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.019046 (+1304299us) service_pool.cc:224] Handling call
0709 13:04:14.241300 (+222254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.308877 (+ 67577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.309126 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.301955  8504 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:14.337934  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:14.338863  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3350) took 1133 ms. Trace:
I20260709 13:04:14.302677  8621 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:14.340519  1584 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [14.9-16.11]
W20260709 13:04:14.321991  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 307299us
I20260709 13:04:14.310227  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.311678  1587 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 22
W20260709 13:04:14.332872  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50821.7us
I20260709 13:04:14.257920  8367 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.342443  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:14.345817  1659 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:14.347559  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:14.115574  1489 rpcz_store.cc:269] 0709 13:02:34.304455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:34.317642 (+ 13187us) service_pool.cc:224] Handling call
0709 13:04:14.114692 (+99797050us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:04.491380 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:04.505677 (+ 14297us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:04.505734 (+    57us) write_op.cc:435] Acquired schema lock
0709 13:04:04.505753 (+    19us) tablet.cc:662] Decoding operations
0709 13:04:04.506503 (+   750us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:04.506614 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:04.506642 (+    28us) tablet.cc:685] Acquiring locks for 3 operations
0709 13:04:04.507151 (+   509us) tablet.cc:701] Row locks acquired
0709 13:04:04.507209 (+    58us) write_op.cc:260] PREPARE: finished
0709 13:04:04.507542 (+   333us) write_op.cc:270] Start()
0709 13:04:04.507761 (+   219us) write_op.cc:276] Timestamp: P: 1783602244507436 usec, L: 0
0709 13:04:04.507803 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 13:04:04.508700 (+   897us) log.cc:844] Serialized 1317 byte log entry
0709 13:04:05.469341 (+960641us) op_driver.cc:464] REPLICATION: finished
0709 13:04:05.828732 (+359391us) write_op.cc:301] APPLY: starting
0709 13:04:05.890958 (+ 62226us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:06.041129 (+150171us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:06.041263 (+   134us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:06.137874 (+ 96611us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:06.233749 (+ 95875us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=6,mrs_lookups=0
0709 13:04:06.233822 (+    73us) write_op.cc:312] APPLY: finished
0709 13:04:06.234167 (+   345us) log.cc:844] Serialized 41 byte log entry
0709 13:04:06.234375 (+   208us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:06.234497 (+   122us) write_op.cc:454] Released schema lock
0709 13:04:06.234562 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:06.235168 (+   606us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:06.235200 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":58966,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":294733,"apply.run_cpu_time_us":9505,"apply.run_wall_time_us":443756,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16673,"num_ops":3,"prepare.queue_time_us":225748,"prepare.run_cpu_time_us":42631,"prepare.run_wall_time_us":295946,"raft.queue_time_us":318341,"raft.run_cpu_time_us":27386,"raft.run_wall_time_us":157280,"replication_time_us":961307,"thread_start_us":622371,"threads_started":4,"wal-append.queue_time_us":332319,"wal-append.run_cpu_time_us":27194,"wal-append.run_wall_time_us":2392260}]]}
I20260709 13:04:14.363842  1586 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [16.9-18.10]
I20260709 13:04:14.365185  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3351) took 1164 ms. Trace:
I20260709 13:04:14.365746  1586 rpcz_store.cc:276] 0709 13:04:13.200915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.339148 (+1138233us) service_pool.cc:224] Handling call
0709 13:04:14.363472 (+ 24324us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:14.364760 (+  1288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.365102 (+   342us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.379079  1662 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 13:04:14.340296  8504 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.343804  8621 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:14.344513  8543 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.397173  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2514) took 1026 ms. Trace:
I20260709 13:04:14.399428  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.341207  1585 rpcz_store.cc:276] 0709 13:04:13.200398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.218167 (+ 17769us) service_pool.cc:224] Handling call
0709 13:04:14.334063 (+1115896us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:14.400734  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 456122us
I20260709 13:04:14.399524  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:14.397698  1660 rpcz_store.cc:276] 0709 13:04:13.359858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.987513 (+627655us) service_pool.cc:224] Handling call
0709 13:04:14.289869 (+302356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.304918 (+ 15049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.386341 (+ 81423us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b5400075dd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.386571 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":614537472}
W20260709 13:04:14.406699  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.390233  8504 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.415097  1659 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 13:04:14.415343  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43558.3us
I20260709 13:04:14.423098  1663 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:04:14.420465  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:14.433904  8624 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 13:04:14.435166  8367 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:14.438426  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:14.439247  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.439375  1739 raft_consensus.cc:2393] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:14.440339  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.440554  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.441155  2311 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:14.441363  2315 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:14.410696  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:14.433177  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.442129  2313 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 14. Current term is 15. Ops: []
I20260709 13:04:14.442528  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.443265  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.443511  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.443914  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.444283  2313 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:14.444545  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.445183  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.446003  2317 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:14.446165  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.446764  2318 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:14.446933  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.443305  2315 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:14.450060  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2493) took 1191 ms. Trace:
I20260709 13:04:14.450452  1666 rpcz_store.cc:276] 0709 13:04:13.258346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.258613 (+   267us) service_pool.cc:224] Handling call
0709 13:04:14.449865 (+1191252us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.449980 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763776384}
I20260709 13:04:14.451994  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2553) took 2040 ms. Trace:
I20260709 13:04:14.452384  1736 rpcz_store.cc:276] 0709 13:04:12.411382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.688175 (+1276793us) service_pool.cc:224] Handling call
0709 13:04:13.775644 (+ 87469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:13.856146 (+ 80502us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:13.856531 (+   385us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.011211 (+154680us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.099320 (+ 88109us) log.cc:844] Serialized 26 byte log entry
0709 13:04:14.099520 (+   200us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:14.099849 (+   329us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.100155 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.100250 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.451753 (+351503us) raft_consensus.cc:1685] finished
0709 13:04:14.451774 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.451911 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55377,"prepare.queue_time_us":393,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":117,"raft.queue_time_us":1160,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":218,"spinlock_wait_cycles":1145728,"thread_start_us":154239,"threads_started":1,"wal-append.queue_time_us":448444}
I20260709 13:04:14.432737  2283 leader_election.cc:400] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16.
W20260709 13:04:14.456333  1591 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.522s	user 0.013s	sys 0.003s
I20260709 13:04:14.456739  1591 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:14.457376  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3355) took 1049 ms. Trace:
I20260709 13:04:14.457741  1591 rpcz_store.cc:276] 0709 13:04:13.412070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.816051 (+403981us) service_pool.cc:224] Handling call
0709 13:04:14.457315 (+641264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.419870  8182 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:14.461427  1587 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:14.458686  1738 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:14.424324  8540 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:14.465862  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.470151  8349 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.470554  8182 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:14.474102  8411 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.737s	user 0.011s	sys 0.015s
I20260709 13:04:14.497330  2312 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:14.476471  1735 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.072s	user 0.003s	sys 0.016s
I20260709 13:04:14.497879  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:14.498476  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2565) took 1587 ms (client timeout 1692 ms). Trace:
W20260709 13:04:14.498890  1735 rpcz_store.cc:269] 0709 13:04:12.911420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.231647 (+320227us) service_pool.cc:224] Handling call
0709 13:04:14.498412 (+1266765us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.497218  8182 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:14.477454  8476 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:14.483282  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.413062  8543 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.073s
W20260709 13:04:14.485773  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:14.493098  1734 consensus_meta.cc:287] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.089s	user 0.008s	sys 0.011s
I20260709 13:04:14.513237  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:14.513962  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2548) took 2108 ms (client timeout 2154 ms). Trace:
I20260709 13:04:14.494951  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:14.495687  2313 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:14.508765  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:14.483916  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:14.494138  8540 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.517890  1737 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:14.518278  2316 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:14.514395  1734 rpcz_store.cc:269] 0709 13:04:12.405172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.220315 (+815143us) service_pool.cc:224] Handling call
0709 13:04:14.513878 (+1293563us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.521513  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.516978  8540 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:14.525668  8432 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.526221  8432 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.485783  8468 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:14.240007  1741 rpcz_store.cc:276] 0709 13:04:12.743932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.885430 (+1141498us) service_pool.cc:224] Handling call
0709 13:04:14.019001 (+133571us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.176445 (+157444us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.176494 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:14.210712 (+ 34218us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:14.210812 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.210915 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.211105 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5287,"raft.queue_time_us":83087,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":21901}
I20260709 13:04:14.536509  8494 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:14.537788  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 123325us
I20260709 13:04:14.538130  1740 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:14.545125  8025 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.758s	user 0.006s	sys 0.006s
I20260709 13:04:14.547866  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:14.550987  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3198) took 1125 ms. Trace:
I20260709 13:04:14.551146  8494 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.552011  8494 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:14.552462  8494 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:14.554951  8411 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:14.556573  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:14.525595  8517 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.815s	user 0.026s	sys 0.012s
W20260709 13:04:14.558209  8556 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.905s	user 0.012s	sys 0.002s
I20260709 13:04:14.558499  8517 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:14.558869  8556 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.559656  2309 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
W20260709 13:04:14.560266  8623 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 13:04:14.508165  8320 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.562070  8602 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:14.562460  8577 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.721s	user 0.013s	sys 0.000s
I20260709 13:04:14.563344  8320 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.563026  8577 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.564289  8411 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:14.566299  8339 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.136s	user 0.009s	sys 0.013s
I20260709 13:04:14.566407  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3341) took 2040 ms. Trace:
I20260709 13:04:14.567022  1584 rpcz_store.cc:276] 0709 13:04:12.526546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.858468 (+1331922us) service_pool.cc:224] Handling call
0709 13:04:14.334063 (+475595us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:14.340994 (+  6931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.566178 (+225184us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54002d19d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.566332 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":937150720}
W20260709 13:04:14.567659  8616 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.755s	user 0.010s	sys 0.010s
I20260709 13:04:14.568377  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:14.568240  8616 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.569080  1584 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:14.571177  8476 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:14.571673  8470 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.571852  1591 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:14.572234  1591 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:14.572270  8470 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:14.551596  1663 rpcz_store.cc:276] 0709 13:04:13.425538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.348948 (+923410us) service_pool.cc:224] Handling call
0709 13:04:14.379108 (+ 30160us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.423736 (+ 44628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.550679 (+126943us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:14.550905 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100152576}
I20260709 13:04:14.573096  1591 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.8, Last appended by leader: 8, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.520826  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:14.577596  2318 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:14.588502  1591 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:14.589988  1661 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:14.591573  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78887.7us
I20260709 13:04:14.593604  1663 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 13:04:14.595091  1593 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.687s	user 0.018s	sys 0.002s
I20260709 13:04:14.595512  1593 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:14.596002  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3359) took 1005 ms. Trace:
I20260709 13:04:14.596340  1593 rpcz_store.cc:276] 0709 13:04:13.608880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.801358 (+192478us) service_pool.cc:224] Handling call
0709 13:04:14.595938 (+794580us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.573001  8470 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.614024  1587 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:14.610348  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:14.616086  1588 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.842s	user 0.013s	sys 0.013s
I20260709 13:04:14.616482  1588 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:14.616997  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3361) took 1007 ms. Trace:
I20260709 13:04:14.617106  1586 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:14.617338  1588 rpcz_store.cc:276] 0709 13:04:13.609803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.708586 (+ 98783us) service_pool.cc:224] Handling call
0709 13:04:14.616934 (+908348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:14.504484  8349 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.545715  8025 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.567041  8339 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:14.658116  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59538.2us
I20260709 13:04:14.659785  8339 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.640836  8517 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:14.662413  8517 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:14.662848  8349 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.553447  8494 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.663496  8517 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.666024  8025 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.640187  8592 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 13:04:14.672855  8621 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.675937  1664 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:14.676156  8540 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:14.677831  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2499) took 1410 ms (client timeout 1866 ms). Trace:
W20260709 13:04:14.678277  1664 rpcz_store.cc:269] 0709 13:04:13.267697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.477624 (+209927us) service_pool.cc:224] Handling call
0709 13:04:14.677565 (+1199941us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540030b454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.677753 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2680067968}
I20260709 13:04:14.682018  1667 raft_consensus.cc:2393] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:14.682938  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.684965  1666 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:14.661211  1584 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:14.694881  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3353) took 1433 ms. Trace:
I20260709 13:04:14.695464  1586 rpcz_store.cc:276] 0709 13:04:13.261445 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.368033 (+1106588us) service_pool.cc:224] Handling call
0709 13:04:14.616934 (+248901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.617522 (+   588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.694575 (+ 77053us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54001def50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.694793 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":499819392}
I20260709 13:04:14.636363  8577 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:14.700989  8407 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.642s	user 0.011s	sys 0.000s
I20260709 13:04:14.642935  1593 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:14.704532  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3275) took 2594 ms. Trace:
I20260709 13:04:14.711928  1740 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:14.661214  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:14.724524  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:14.717548  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 513793us
W20260709 13:04:14.743235  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82132.8us
I20260709 13:04:14.743821  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:14.744120  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:14.744527  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:14.744901  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:14.745787  8407 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:14.745975  8367 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:14.746372  8407 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.746821  8407 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:14.749780  8411 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:14.750949  8411 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:14.751611  8411 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:14.756253  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 236537us
I20260709 13:04:14.757579  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:14.759099  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:14.766877  8629 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.112s	user 0.001s	sys 0.004s
I20260709 13:04:14.768958  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:14.769210  8540 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:14.770110  8540 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:14.770527  8540 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:14.770933  8540 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.705142  1742 rpcz_store.cc:276] 0709 13:04:12.110270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.576692 (+1466422us) service_pool.cc:224] Handling call
0709 13:04:13.730976 (+154284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:13.767444 (+ 36468us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:13.793957 (+ 26513us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.210594 (+416637us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:14.210664 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:14.210728 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.210811 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.704308 (+493497us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b54000e2c50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.704451 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12724,"prepare.queue_time_us":17409,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":25239,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":22476,"spinlock_wait_cycles":129648000,"thread_start_us":363419,"threads_started":1,"wal-append.queue_time_us":581796}
I20260709 13:04:14.716183  8556 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:14.645350  8616 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:14.645350  8470 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:14.793632  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1148) took 1509 ms. Trace:
I20260709 13:04:14.794137  1661 rpcz_store.cc:276] 0709 13:04:13.270743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.823432 (+552689us) service_pool.cc:224] Handling call
0709 13:04:13.825805 (+  2373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.590729 (+764924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.779891 (+189162us) spinlock_profiling.cc:243] Waited 749 ms on lock 0x7b5400014fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:14.780167 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721795456}
I20260709 13:04:14.649262  8182 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:14.800612  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 261800us
I20260709 13:04:14.801067  8182 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:14.793362  8628 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.092s	user 0.001s	sys 0.001s
I20260709 13:04:14.805478  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:14.811829  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3227) took 1019 ms. Trace:
I20260709 13:04:14.812413  1667 rpcz_store.cc:276] 0709 13:04:13.792360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.798682 (+  6322us) service_pool.cc:224] Handling call
0709 13:04:14.811547 (+1012865us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54001a6dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.811749 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1954402432}
I20260709 13:04:14.753048  8411 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:14.815573  8630 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.092s	user 0.000s	sys 0.002s
I20260709 13:04:14.816370  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3212) took 1182 ms. Trace:
I20260709 13:04:14.818085  8616 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:14.818986  8616 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:14.816870  1666 rpcz_store.cc:276] 0709 13:04:13.634159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.451286 (+817127us) service_pool.cc:224] Handling call
0709 13:04:14.684593 (+233307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.714837 (+ 30244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.816074 (+101237us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b540030b450. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:14.816294 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":535486208}
I20260709 13:04:14.826180  8018 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.826750  8018 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.756954  8504 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.830758  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:14.806409  8475 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.900s	user 0.010s	sys 0.009s
I20260709 13:04:14.831995  2318 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:14.831710  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 476468us
I20260709 13:04:14.803527  8621 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:14.835095  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 524191us
I20260709 13:04:14.854267  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:14.854434  8627 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 13:04:14.855866  2313 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:14.856019  1586 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:14.857621  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.857949  8025 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:14.858458  1586 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:14.858556  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3340) took 2203 ms. Trace:
I20260709 13:04:14.859215  8556 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:14.858036  8622 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.456s	user 0.002s	sys 0.000s
I20260709 13:04:14.859629  8556 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:14.860647  8556 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.861979  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.863348  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.863994  1587 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:14.864917  8432 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:14.861241  8494 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:14.862670  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.864059  1584 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:14.867396  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.859037  1590 rpcz_store.cc:276] 0709 13:04:12.656899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.011969 (+1355070us) service_pool.cc:224] Handling call
0709 13:04:14.120500 (+108531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.120835 (+   335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:14.120867 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.121382 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 13:04:14.613540 (+492158us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:14.613586 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.613714 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:14.858272 (+244558us) raft_consensus.cc:1685] finished
0709 13:04:14.858309 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.858493 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":491703,"threads_started":1,"wal-append.queue_time_us":681811}
I20260709 13:04:14.804740  8470 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
W20260709 13:04:14.890508  2281 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.005s	sys 0.006s
W20260709 13:04:14.893142  2281 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.006s	sys 0.006s
W20260709 13:04:14.894066  1706 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.012s	sys 0.015s
W20260709 13:04:14.894423  1706 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.012s	sys 0.015s
I20260709 13:04:14.896041  1666 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:14.894761  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 872938us
W20260709 13:04:14.913954  1592 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.001s	sys 0.014s
W20260709 13:04:14.914740  1592 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.001s	sys 0.014s
W20260709 13:04:14.915098  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61364.8us
I20260709 13:04:14.916026  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3348) took 2096 ms. Trace:
I20260709 13:04:14.933146  1586 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:14.935580  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.943764  1587 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:14.850185  8475 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.947474  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:14.954092  1706 connection.cc:256] RPC call timeout handler was delayed by -1.0505814060000489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3245s for a response
W20260709 13:04:14.954886  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:14.920099  8543 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:14.957598  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.004s	sys 0.006s
W20260709 13:04:14.958039  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.004s	sys 0.007s
I20260709 13:04:14.962128  8475 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.962471  1740 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:14.927469  8182 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:04:14.934446  8432 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:14.963479  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:14.964493  1590 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:14.963505  8182 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:14.966218  8182 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:14.966754  1660 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:14.962504  8504 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:14.971843  8398 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:14.972716  8398 consensus_queue.cc:1243] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:14.973743  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:14.973994  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:14.974740  1590 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:14.974978  1742 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 13:04:14.975155  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58935.8us
I20260709 13:04:14.975955  1590 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:04:14.978145  1586 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 13:04:14.967303  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:14.979936  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:14.980159  1588 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:04:14.981395  8025 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:14.973780  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:14.982976  8577 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:14.983395  8577 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:14.983930  8349 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:14.984365  8349 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:14.984846  8349 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:14.984431  8577 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:14.925575  8348 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.471s	user 0.011s	sys 0.015s
I20260709 13:04:14.982039  8182 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.985832  8627 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:14.986639  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:14.986819  8182 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:14.983403  8592 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:14.987627  8592 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:14.992815  8018 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:14.981477  1706 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:14.993989  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:14.996058  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:14.916579  1592 rpcz_store.cc:276] 0709 13:04:12.819959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.096397 (+1276438us) service_pool.cc:224] Handling call
0709 13:04:14.250868 (+154471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.311440 (+ 60572us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:14.344355 (+ 32915us) log.cc:844] Serialized 14 byte log entry
0709 13:04:14.915488 (+571133us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:14.915564 (+    76us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:14.915639 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.915728 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:14.915955 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32710,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"thread_start_us":513352,"threads_started":1}
W20260709 13:04:15.006532  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 260080us
I20260709 13:04:15.007074  1584 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:15.005357  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82172.6us
W20260709 13:04:15.008265  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 239293us
I20260709 13:04:15.008906  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:15.010154  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.012315  1588 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:15.015834  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:15.008221  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:15.020032  1661 raft_consensus.cc:2376] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18. Re-sending same reply.
I20260709 13:04:15.022914  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:15.028640  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:15.030297  1663 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:15.030310  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:15.033584  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:15.024801  8475 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.012s
W20260709 13:04:15.039932  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 88560.8us
I20260709 13:04:14.999845  8348 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.047112  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:15.045245  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:15.050679  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:15.053529  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.049175  1666 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:15.071770  2310 raft_consensus.cc:606] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:15.072319  2310 raft_consensus.cc:614] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:15.073062  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3353) took 1692 ms. Trace:
I20260709 13:04:15.073585  1585 rpcz_store.cc:276] 0709 13:04:13.381604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.422351 (+1040747us) service_pool.cc:224] Handling call
0709 13:04:14.462700 (+ 40349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.593332 (+130632us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:14.638391 (+ 45059us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.072460 (+434069us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.072633 (+   173us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:15.072681 (+    48us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:15.072741 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.072849 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.072993 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1185,"prepare.queue_time_us":42700,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":373858,"threads_started":1}
I20260709 13:04:15.079262  1666 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:15.033736  8367 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:15.075968  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.034507  8592 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.085371  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:15.092197  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44757.8us
I20260709 13:04:15.093918  1665 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:15.092074  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95955us
I20260709 13:04:15.099092  2309 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:15.100613  1663 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:15.099532  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 114989us
I20260709 13:04:15.106437  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.093346  8592 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:15.076002  8475 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 13:04:15.114574  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.113797  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.099216  8636 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:15.098878  8633 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:15.107795  8387 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.121042  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 361897us
W20260709 13:04:15.121349  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80647.9us
I20260709 13:04:15.122886  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.124313  8643 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 13:04:15.127350  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.121892  8636 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.124394  8387 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.130146  8636 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:15.111960  8320 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 13:04:15.132019  8387 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:15.132656  8387 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.129683  8635 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:15.134038  8635 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.135157  8635 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.137794  8018 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:15.138569  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.138773  8018 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:15.133327  8387 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:15.139424  8018 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:15.140429  8387 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.122614  8182 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
I20260709 13:04:15.116492  8407 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.141607  8320 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:15.141988  8320 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.142833  8320 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:15.143503  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43635.3us
I20260709 13:04:15.144502  8320 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.140630  8018 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:15.147060  8639 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:04:15.150445  8411 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.151460  8367 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.151986  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.152519  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3209) took 1653 ms. Trace:
I20260709 13:04:15.153008  8643 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.154443  8475 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:15.154884  8475 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.155258  8475 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:15.158154  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 176738us
I20260709 13:04:15.159046  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.159130  8398 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:15.159868  8398 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:15.160342  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:15.129990  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3347) took 2332 ms. Trace:
I20260709 13:04:15.161958  1589 rpcz_store.cc:276] 0709 13:04:12.797192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.217704 (+1420512us) service_pool.cc:224] Handling call
0709 13:04:14.336418 (+118714us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:14.362434 (+ 26016us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:14.362476 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:14.426063 (+ 63587us) log.cc:844] Serialized 26 byte log entry
0709 13:04:14.891501 (+465438us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:14.891552 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:14.891630 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:15.129669 (+238039us) raft_consensus.cc:1685] finished
0709 13:04:15.129695 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.129886 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":423707,"threads_started":1,"wal-append.queue_time_us":632092}
I20260709 13:04:15.163056  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.163981  8411 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.164824  8411 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:04:15.166026  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52573.1us
I20260709 13:04:15.124972  1557 leader_election.cc:400] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:15.166635  1557 leader_election.cc:403] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:15.152945  1660 rpcz_store.cc:276] 0709 13:04:13.507907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.425213 (+917306us) service_pool.cc:224] Handling call
0709 13:04:14.441940 (+ 16727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.967174 (+525234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.152241 (+185067us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540013e2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:15.152443 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205974912}
I20260709 13:04:15.172766  8637 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 13:04:15.174033  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49213.8us
I20260709 13:04:15.175976  8636 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.177109  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:15.122781  8633 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:15.177640  1706 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:15.177984  8633 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.178287  8643 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:15.179334  8643 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:15.179373  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:15.179932  8643 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.180401  8643 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.180447  8577 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.181686  8517 raft_consensus.cc:2764] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:15.182014  8407 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:15.182252  8556 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:15.166270  8650 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.182777  8407 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:15.183252  8650 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.183818  8407 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:15.184427  8407 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:15.184576  8475 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.174398  8432 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:15.174640  8494 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
W20260709 13:04:15.190613  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 183157us
W20260709 13:04:15.198009  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:15.200078  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 70568.4us
I20260709 13:04:15.121949  8627 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 13:04:15.208391  8348 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.209427  8636 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.211508  8475 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.165387  8411 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:15.222101  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.223060  2309 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:15.141264  8387 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.229014  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 66078.3us
I20260709 13:04:15.230481  8647 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.233106  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 109194us
I20260709 13:04:15.234584  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.122929  8592 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
W20260709 13:04:15.241014  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102852us
I20260709 13:04:15.161007  8398 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.246294  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.204382  8182 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 13:04:15.248152  1588 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:15.253553  1588 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:15.185557  8407 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.264954  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.265950  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.238619  8650 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:15.272307  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:15.275494  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.290733  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3206) took 1862 ms. Trace:
I20260709 13:04:15.229230  8387 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:15.291201  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44571.9us
W20260709 13:04:15.292555  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115525us
I20260709 13:04:15.269726  8656 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:15.280637  1588 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.237299  8475 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:04:15.241568  8517 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:04:15.291260  1662 rpcz_store.cc:276] 0709 13:04:13.428669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.393313 (+964644us) service_pool.cc:224] Handling call
0709 13:04:14.419951 (+ 26638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.574626 (+154675us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:14.946879 (+372253us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.290318 (+343439us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:15.290391 (+    73us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:15.290446 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.290520 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.290661 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5014,"prepare.queue_time_us":480250,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"thread_start_us":593908,"threads_started":2}
I20260709 13:04:15.274048  8592 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:15.299909  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.299619  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3208) took 1870 ms. Trace:
W20260709 13:04:15.307152  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:15.294394  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 104482us
I20260709 13:04:15.240516  8647 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.290820  8621 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:15.312538  1588 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:15.304687  1659 rpcz_store.cc:276] 0709 13:04:13.431213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.422898 (+991685us) service_pool.cc:224] Handling call
0709 13:04:14.436379 (+ 13481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:14.436766 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:14.885935 (+449169us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.156820 (+270885us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18003b13a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a
0709 13:04:15.169724 (+ 12904us) log.cc:844] Serialized 14 byte log entry
0709 13:04:15.299209 (+129485us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:15.299268 (+    59us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:15.299324 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.299435 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:15.299560 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29526,"prepare.queue_time_us":596449,"prepare.run_cpu_time_us":150,"prepare.run_wall_time_us":5397,"raft.queue_time_us":83805,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":13397,"spinlock_wait_cycles":15166208,"thread_start_us":692245,"threads_started":3,"wal-append.queue_time_us":222995}
I20260709 13:04:15.299906  8182 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:15.317556  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:15.294858  8656 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:15.320719  2313 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:15.322530  2313 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:15.327255  8649 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.328176  8649 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.329742  8638 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:15.332701  8665 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:04:15.327425  8540 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.556s	user 0.013s	sys 0.002s
I20260709 13:04:15.315267  8621 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:15.345670  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.348207  8649 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:15.337213  2313 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.333906  8660 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:15.352679  2313 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:15.357405  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110144us
I20260709 13:04:15.358434  1556 leader_election.cc:400] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16.
I20260709 13:04:15.360904  1556 leader_election.cc:403] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:04:15.327489  8658 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:15.361456  8642 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 13:04:15.313360  8411 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 14.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.368531  1588 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:15.343000  8540 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.357190  8660 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:15.334015  8657 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.142s
I20260709 13:04:15.381551  1667 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:15.379175  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 132953us
I20260709 13:04:15.382114  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:15.357322  8543 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:15.394145  2313 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 13:04:15.394654  2313 pending_rounds.cc:85] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 12
I20260709 13:04:15.395068  2313 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 13:04:15.408416  2313 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:15.409590  8666 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 13:04:15.406729  8494 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 15. Current term is 16.
I20260709 13:04:15.411702  1737 pending_rounds.cc:85] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:04:15.384689  8592 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:15.413923  1737 pending_rounds.cc:107] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 13:04:15.390705  8660 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:15.414958  1737 raft_consensus.cc:2889] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:15.420066  2309 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:04:15.422024  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.422669  1740 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:04:15.394601  8543 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:15.409607  8663 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:04:15.427912  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.429092  8668 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:15.431836  8668 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:15.432248  8668 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:15.433679  8387 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:15.435355  8647 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:15.431149  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 72790.4us
I20260709 13:04:15.437315  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.438992  8633 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:15.416599  8602 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 13:04:15.444414  8540 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.385893  8650 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 13:04:15.412462  8018 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:15.447130  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.385558  8349 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.453183  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.414709  8592 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:15.453365  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:15.454981  2309 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:15.423601  8616 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:15.441447  8633 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:15.454785  8018 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:15.466868  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 99509.1us
W20260709 13:04:15.465852  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 128900us
I20260709 13:04:15.468772  8494 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:15.468967  8470 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.469805  8018 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:15.469722  8470 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.471707  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.475087  8636 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:15.476466  8636 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:15.476186  8592 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.476888  8636 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:15.397838  8656 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.489400  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.490525  8470 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:15.490895  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.491261  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.492179  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.492534  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.490928  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.494411  1738 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:15.494674  8339 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:15.495064  8339 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.495496  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.496269  1740 raft_consensus.cc:2368] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20.
I20260709 13:04:15.495951  8339 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.497231  8339 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:15.502871  1634 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.497925  8504 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:15.498306  1738 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:15.498812  8470 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.505205  8504 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.517159  8470 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.518028  8504 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.519276  8504 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:15.519608  8470 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.499500  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.520531  1740 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:15.521528  8667 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:15.522022  8667 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.522951  8667 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.500991  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2502) took 2231 ms. Trace:
I20260709 13:04:15.501528  1668 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:15.524176  8667 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:15.527176  8349 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:15.527506  8348 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 13:04:15.514258  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.523933  1665 rpcz_store.cc:276] 0709 13:04:13.269908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:13.815419 (+545511us) service_pool.cc:224] Handling call
0709 13:04:15.500805 (+1685386us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54000145d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:15.500936 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2823154048}
I20260709 13:04:15.532634  8348 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.539745  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.539556  8621 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.496997  8636 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.543310  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.544137  1739 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:15.544361  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.463995  8616 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.545236  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.545482  2310 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:15.545938  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.546586  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.547376  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.528573  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:15.548659  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.548720  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.549937  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:15.550544  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46321.4us
I20260709 13:04:15.546828  1736 raft_consensus.cc:2410] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:15.547464  8468 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:15.538251  1664 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:04:15.527765  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:15.552026  1667 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:15.582518  1664 raft_consensus.cc:606] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.527894  8391 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:04:15.582749  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 55046.9us
I20260709 13:04:15.528970  8649 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:15.583211  8391 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.583673  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.583873  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:15.584450  8649 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:15.584169  8391 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.585000  8649 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:15.553076  1659 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:15.585698  8647 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:15.586186  8647 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.562011  8636 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:15.565035  8540 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:15.565654  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.567044  1741 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:15.567461  8671 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.595059  1660 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:15.567704  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.567915  8656 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:15.568478  8602 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.569137  8616 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:15.570188  8592 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.541378  8621 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:15.504695  1634 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.549953  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.573004  2310 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:04:15.582970  1664 raft_consensus.cc:614] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:04:15.585157  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.554248  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3238) took 1507 ms. Trace:
I20260709 13:04:15.607568  1668 rpcz_store.cc:276] 0709 13:04:14.046686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.522689 (+476003us) service_pool.cc:224] Handling call
0709 13:04:15.501332 (+978643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:15.524422 (+ 23090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:15.554062 (+ 29640us) spinlock_profiling.cc:243] Waited 957 ms on lock 0x7b54000145d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:15.554185 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2202014976}
I20260709 13:04:15.477854  8661 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:15.597270  1738 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:15.598951  1659 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:15.611999  1632 connection.cc:256] RPC call timeout handler was delayed by -2.765877525999997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2548s for a response
I20260709 13:04:15.585381  8391 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:15.611050  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106417us
I20260709 13:04:15.618636  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:15.586714  8647 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.618714  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.516774  8633 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:15.610265  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 156920us
I20260709 13:04:15.627714  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:15.629323  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.594373  8540 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:15.636137  1664 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:15.637416  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 132855us
I20260709 13:04:15.638382  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.639122  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.639649  1738 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:15.642874  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.637993  1634 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:15.643867  1736 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.645744  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.649467  1740 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:15.655534  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.610991  8661 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:15.658056  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.627146  8468 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:15.660385  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.682148  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:15.687132  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:15.687312  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48276.3us
I20260709 13:04:15.570235  8348 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.610680  8602 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.646528  8665 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:15.697808  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:15.702271  1584 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
W20260709 13:04:15.686556  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56407us
I20260709 13:04:15.704068  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:15.697342  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 225617us
I20260709 13:04:15.586124  8649 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.659610  8637 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:15.698130  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:15.698312  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:15.710353  1738 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:15.711971  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:15.712471  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.697770  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:15.706072  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.611541  8671 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.707422  2310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
 from {username='slave'} at 127.0.0.1:59800
W20260709 13:04:15.707445  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98487.5us
W20260709 13:04:15.713771  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.721688  2310 raft_consensus.cc:606] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Received request to transfer leadership to TS 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:15.722114  2310 raft_consensus.cc:614] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:15.723381  1584 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:15.717938  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:15.646426  8672 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.059s
I20260709 13:04:15.685108  8650 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:15.685566  8468 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:15.814477  1740 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:15.688863  8661 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.641376  8627 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 13:04:15.726948  2317 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:15.643015  8643 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.730386  8637 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.690752  8411 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:15.751466  8683 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.752511  8677 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:15.769773  8540 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.729077  2312 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:15.728461  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:15.798359  1736 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:15.867924  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:15.871712  1587 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:15.822731  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:15.879351  1585 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:15.727644  8638 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:15.806320  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.808398  1738 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:15.809734  1741 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:15.889622  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:15.812256  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:15.813325  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 191039us
I20260709 13:04:15.820325  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:15.724181  8349 raft_consensus.cc:2764] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 13:04:15.828462  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.822041  1584 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:04:15.854627  1740 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 13:04:15.893079  1740 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:15.893954  1736 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
I20260709 13:04:15.890261  1593 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:15.888736  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73340.7us
I20260709 13:04:15.730204  8665 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:15.890379  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:15.899863  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.902427  2317 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:15.902671  1738 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:04:15.903065  1632 connection.cc:256] RPC call timeout handler was delayed by -2.1871775260000277s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3191s for a response
I20260709 13:04:15.903093  2317 raft_consensus.cc:606] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:15.903700  1632 connection.cc:256] RPC call timeout handler was delayed by -1.1993695180000259s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
W20260709 13:04:15.904048  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0607695180000292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260709 13:04:15.904790  1589 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:15.906630  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.907686  8517 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:15.908188  1587 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:15.905361  1738 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:15.909044  1740 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:15.922327  1591 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:15.894336  1736 pending_rounds.cc:85] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
W20260709 13:04:15.927259  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 205253us
I20260709 13:04:15.929062  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:15.930361  1736 pending_rounds.cc:107] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 13:04:15.930757  1736 raft_consensus.cc:2889] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:15.931541  1584 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:15.821532  8398 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.095s
I20260709 13:04:15.823669  8468 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:15.964056  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:15.950242  1593 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:15.930015  8476 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:15.910046  8677 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:15.975658  1591 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:15.867225  8411 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.885478  8683 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:15.891610  8660 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.031981  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.014983  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:04:15.984679  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 56307.5us
I20260709 13:04:15.981984  1593 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:04:15.995029  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:15.995983  8685 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
W20260709 13:04:16.011044  8679 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 13:04:15.939220  8638 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:15.980868  8650 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:16.014837  8018 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.001s
W20260709 13:04:16.015352  8680 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
W20260709 13:04:16.022572  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:15.938843  8320 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 13:04:16.084051  2310 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:04:16.035401  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:15.983424  8540 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:16.057351  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.120692  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.121608  8682 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:04:15.990324  8367 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:16.064477  1591 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:16.066460  2312 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:16.126204  8602 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:15.989259  8671 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.071578  1739 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:04:16.126919  8683 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:16.127533  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 89923us
I20260709 13:04:16.128326  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:16.129470  8543 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.675s	user 0.009s	sys 0.002s
I20260709 13:04:16.129992  8543 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.131094  8638 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.116138  8677 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.132210  8543 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.135238  8660 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:16.135457  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 80833.1us
I20260709 13:04:16.136008  1664 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:16.136548  1664 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:16.137719  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:16.141461  8411 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:16.143240  8668 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.710s	user 0.006s	sys 0.007s
I20260709 13:04:16.143736  8668 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.144969  1665 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:15.972872  8476 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.147387  8476 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.028149  8647 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.149837  8647 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:16.151598  1664 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.153430  1664 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:16.099828  8667 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.575s	user 0.008s	sys 0.005s
I20260709 13:04:16.102218  8643 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:16.105146  8636 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.501s	user 0.006s	sys 0.001s
I20260709 13:04:16.047346  8637 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.112654  8665 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.065551  8470 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:16.083374  8387 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:16.117038  1593 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:16.116051  8657 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 13:04:16.095518  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.165272  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 253522us
W20260709 13:04:16.232666  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 217903us
I20260709 13:04:16.199790  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:16.165328  8504 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.643s	user 0.013s	sys 0.001s
I20260709 13:04:16.227128  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:16.168471  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 356177us
W20260709 13:04:16.250584  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.129607  8650 raft_consensus.cc:2764] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 13:04:16.086793  8018 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:16.242121  8339 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.686s	user 0.002s	sys 0.010s
W20260709 13:04:16.227083  8592 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.614s	user 0.005s	sys 0.006s
I20260709 13:04:16.200412  8391 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:16.266464  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:16.267756  1593 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:16.231681  8671 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.245991  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:16.271538  1741 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:16.165041  8543 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.096s
W20260709 13:04:16.231972  8621 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.619s	user 0.008s	sys 0.004s
I20260709 13:04:16.248514  1742 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:16.248896  2318 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:16.294667  2318 raft_consensus.cc:606] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:16.295066  2318 raft_consensus.cc:614] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:16.233952  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:16.251819  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111297us
I20260709 13:04:16.200176  8411 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 13:04:16.256325  1591 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:16.166777  8627 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.082s
I20260709 13:04:16.273228  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:16.280833  8668 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:16.281970  1585 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:16.283479  2316 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:16.284435  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.274257  1664 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:16.231706  8657 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:16.295938  2310 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:16.203933  8656 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.001s
I20260709 13:04:16.297129  1735 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:16.185494  8647 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.085s
W20260709 13:04:16.270388  8689 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 13:04:16.315559  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:16.329749  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:16.334276  2311 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
W20260709 13:04:16.336069  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51749.1us
I20260709 13:04:16.254771  8387 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:16.341112  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67489.6us
I20260709 13:04:16.344651  2313 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:16.347532  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47954.5us
I20260709 13:04:16.349839  8387 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:16.350538  8643 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:16.350939  8643 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.351297  8643 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:16.342792  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.359149  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:16.359156  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:16.360249  2310 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:16.359481  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:16.368263  1557 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.097s (SENT)
I20260709 13:04:16.334291  8517 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:16.374825  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80998.8us
W20260709 13:04:16.375299  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.379688  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 251368us
I20260709 13:04:16.338117  8683 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:16.387013  1632 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:16.340553  8592 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:16.387125  8387 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:16.389562  8650 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.292109  8671 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.392444  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:16.356580  8636 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:16.395944  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53150.4us
I20260709 13:04:16.400902  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.397051  1585 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 13:04:16.453933  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.467744  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.479635  8636 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.387228  8683 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:16.486565  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118311us
I20260709 13:04:16.479231  8391 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 13:04:16.479231  8621 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:16.358487  8616 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 13:04:16.397261  8182 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 13:04:16.480649  8699 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.278s	user 0.000s	sys 0.002s
I20260709 13:04:16.496575  2313 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:16.416982  8543 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 13:04:16.480784  8650 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.491286  8387 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:16.498528  1664 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:16.499086  8517 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.412815  8668 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:16.500573  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:16.507472  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 106610us
I20260709 13:04:16.399514  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:16.740844  1585 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:16.510387  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:16.741595  2310 raft_consensus.cc:2376] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21. Re-sending same reply.
I20260709 13:04:16.505260  8339 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 13:04:16.519465  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:16.480058  1588 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:16.744590  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:16.526047  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:16.745344  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
W20260709 13:04:16.529736  1557 leader_election.cc:336] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.154s (SENT)
I20260709 13:04:16.658862  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:16.533460  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:16.592581  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:16.746846  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:16.532629  2311 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:16.532038  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:16.598417  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.661063  8621 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:16.531036  8683 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:16.528484  8540 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.531975  8348 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.512154  8475 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:16.690721  2316 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1008) took 1149 ms. Trace:
I20260709 13:04:16.691953  8650 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.479981  8657 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.693236  8642 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.704165  8018 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:16.708369  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:16.794981  1590 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:16.509966  8677 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.168s
I20260709 13:04:16.720554  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:16.724408  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.527767  8708 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:16.747913  1734 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 13:04:16.749053  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.661653  8661 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.000s
W20260709 13:04:16.758168  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 226141us
I20260709 13:04:16.761638  8621 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:16.762892  8504 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:16.763201  8671 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.769052  1659 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:16.769788  8540 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.770141  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:16.808524  1740 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 16: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:16.768340  2316 rpcz_store.cc:276] 0709 13:04:15.541489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.544128 (+  2639us) service_pool.cc:224] Handling call
0709 13:04:16.690499 (+1146371us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:16.690620 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1668093184}
W20260709 13:04:16.743323  8689 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.214s	user 0.001s	sys 0.000s
I20260709 13:04:16.774570  1588 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
W20260709 13:04:16.798218  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:16.801252  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 271458us
I20260709 13:04:16.530761  8635 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.240s
I20260709 13:04:16.809793  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:16.770562  8348 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.818811  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:16.792143  8713 raft_consensus.cc:2764] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 13:04:16.835083  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 435466us
I20260709 13:04:16.523907  8476 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.274s
W20260709 13:04:16.818542  1557 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.208s (SENT)
I20260709 13:04:16.800486  8668 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:16.742535  8697 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 13:04:16.817979  8320 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.796370  8475 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.800091  8602 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 13:04:16.813802  8504 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:16.855767  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3306) took 1311 ms. Trace:
I20260709 13:04:16.859529  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.848801  8695 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:16.844287  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:16.801865  8642 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:16.875770  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 365699us
I20260709 13:04:16.849259  8182 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:16.860769  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:16.809607  8661 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.858047  8018 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.820540  8708 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.869016  1733 rpcz_store.cc:276] 0709 13:04:15.544173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.553945 (+  9772us) service_pool.cc:224] Handling call
0709 13:04:15.554318 (+   373us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:15.555133 (+   815us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 13:04:15.685523 (+130390us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.126645 (+441122us) spinlock_profiling.cc:243] Waited 8.71 ms on lock 0x7b18002dafc0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d23a
0709 13:04:16.126867 (+   222us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:04:16.127856 (+   989us) log.cc:844] Serialized 78 byte log entry
0709 13:04:16.128129 (+   273us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:16.128412 (+   283us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.128808 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 13:04:16.129004 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:16.129225 (+   221us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:16.855376 (+726151us) raft_consensus.cc:1685] finished
0709 13:04:16.855419 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:16.855672 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":375634,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":4876,"raft.queue_time_us":18231,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":330,"spinlock_wait_cycles":20034304,"thread_start_us":385972,"threads_started":2,"wal-append.queue_time_us":388947}
I20260709 13:04:16.868654  8635 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:16.875358  8714 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.330s	user 0.002s	sys 0.000s
I20260709 13:04:16.873911  8543 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:16.888689  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44541us
I20260709 13:04:16.861073  8647 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 13:04:16.899722  2309 raft_consensus.cc:2368] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:04:16.901592  2313 raft_consensus.cc:606] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:16.902246  2313 raft_consensus.cc:614] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:16.865024  8320 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:16.910259  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50832.9us
I20260709 13:04:16.895738  8387 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:16.893931  8182 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:16.893648  8635 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:16.931226  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118542us
I20260709 13:04:16.931668  1667 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 15. Current term is 16. Ops: []
I20260709 13:04:16.894639  8668 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:16.926396  1664 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:16.927277  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:16.940769  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 231594us
I20260709 13:04:16.929165  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:16.935345  8540 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.934814  8504 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:16.935981  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:16.937759  8411 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:16.937839  2315 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:04:16.985131  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:16.938717  8712 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 13:04:16.930217  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:16.989120  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:16.976401  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:16.977586  1659 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:16.990098  1587 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:04:16.978929  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:16.930600  8468 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.953s	user 0.006s	sys 0.007s
I20260709 13:04:16.900192  8018 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:16.913424  8636 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:16.956063  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:16.997839  1667 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:16.987234  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:16.942469  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:16.993523  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:16.997534  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:16.994306  1587 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:17.002220  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.003590  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:16.910919  8339 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 13:04:17.006666  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:17.012405  1585 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:17.005010  1668 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:17.013931  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.015241  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:16.912427  8391 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 13:04:17.009855  1591 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:17.023736  2309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1004) took 1533 ms (client timeout 1612 ms). Trace:
I20260709 13:04:17.010586  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.010815  1584 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
W20260709 13:04:17.006599  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92543.5us
I20260709 13:04:17.025249  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:16.922883  8637 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 13:04:17.024057  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:17.024227  2309 rpcz_store.cc:269] 0709 13:04:15.489836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.587811 (+ 97975us) service_pool.cc:224] Handling call
0709 13:04:17.023530 (+1435719us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400724054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:17.023675 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2928341248}
I20260709 13:04:17.029773  1587 raft_consensus.cc:2376] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20. Re-sending same reply.
I20260709 13:04:17.032624  1592 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:17.039780  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110777us
I20260709 13:04:16.984297  8387 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:16.996405  8540 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.050865  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:16.945652  8677 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 13:04:17.055891  1585 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:17.057947  8671 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.020063  8713 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.027921  8543 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:16.946118  8716 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.115s
I20260709 13:04:17.063995  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.070187  1592 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:16.992751  8411 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:17.081696  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:17.064481  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 131280us
I20260709 13:04:17.055220  8647 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:17.017613  8468 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.092002  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.113339  1592 raft_consensus.cc:2376] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19. Re-sending same reply.
I20260709 13:04:17.041416  8717 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 13:04:17.095088  1585 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:17.066047  8543 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.122802  8650 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.068727  8671 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.123543  8650 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.126144  8671 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:17.085318  8348 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.060599  8540 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.138638  1593 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:17.096025  8647 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:17.142387  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.147626  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.114578  8636 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:17.142746  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:17.145982  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136802us
I20260709 13:04:17.069700  8713 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.119091  8543 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:17.148356  1592 raft_consensus.cc:2376] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13. Re-sending same reply.
I20260709 13:04:17.128970  8637 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:17.153872  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 128670us
I20260709 13:04:17.068115  8635 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.130869  8728 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:17.134197  8475 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:17.159926  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 233878us
I20260709 13:04:17.078087  8182 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:17.159452  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 118343us
I20260709 13:04:17.192374  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:17.145347  8642 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:17.189698  1632 leader_election.cc:400] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20.
I20260709 13:04:17.193915  1585 raft_consensus.cc:2376] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22. Re-sending same reply.
I20260709 13:04:17.132476  8668 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.200202  1739 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:17.206809  1592 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:17.155573  8661 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.155434  8636 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.222431  1592 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 13:04:17.225534  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 70166.6us
I20260709 13:04:17.220894  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.188989  8737 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:17.195179  8320 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.232033  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.232770  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:17.236177  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.236992  1590 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:17.157016  8387 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:17.243863  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 84656us
I20260709 13:04:17.244872  1592 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 13:04:17.161854  8649 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 13:04:17.249965  1591 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:17.211177  8708 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.250949  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.247470  1592 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.162082  8724 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 13:04:17.188650  8637 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.251680  1590 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:17.240317  8723 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 13:04:17.263026  8715 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 13:04:17.264412  8621 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:17.266178  8725 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 13:04:17.267583  2310 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 13:04:17.243201  8339 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:17.279276  1590 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:17.244098  8650 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.250017  8635 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:17.307909  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60223.1us
W20260709 13:04:17.291973  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 86017.9us
W20260709 13:04:17.322672  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.328902  1592 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 14 index: 14. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 13:04:17.331182  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.331888  1592 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:17.333660  1585 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:04:17.304167  8657 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:17.341308  1591 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:17.309108  8339 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.360023  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.360936  1585 raft_consensus.cc:2368] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16.
I20260709 13:04:17.281209  8713 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:17.342065  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 124070us
W20260709 13:04:17.363696  1708 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.793s (SENT) [suppressed 46 similar messages]
I20260709 13:04:17.282985  8018 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.374004  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.343273  8728 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
W20260709 13:04:17.383751  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.346434  8411 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:17.386004  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1130) took 1696 ms (client timeout 2250 ms). Trace:
W20260709 13:04:17.389353  1708 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.552s (SENT)
I20260709 13:04:17.365008  8339 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:17.389653  1588 rpcz_store.cc:269] 0709 13:04:15.689428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:15.697874 (+  8446us) service_pool.cc:224] Handling call
0709 13:04:17.385801 (+1687927us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:17.385934 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1769768320}
I20260709 13:04:17.378298  8713 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.405305  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:17.403631  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.733s (SENT)
I20260709 13:04:17.406066  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:17.405305  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.382390  8650 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:17.410965  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:17.411392  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:17.411698  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43664.7us
I20260709 13:04:17.413920  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:17.415578  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:17.408254  1585 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:17.421608   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:04:17.424515   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:17.425308  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.425865  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.429311  8411 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:17.429746  8650 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.430470  8411 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:17.430858  8411 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:17.431792  8411 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.433096  8411 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.439041  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:17.439837  1735 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:04:17.406256  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:17.390920  8468 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.438016  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.344548  8668 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.382889  8649 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.024s
W20260709 13:04:17.463927  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.404768  8656 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 13:04:17.480693  1735 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [17.12-19.13]
I20260709 13:04:17.498198  1585 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 13:04:17.497052  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.507815  1666 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:17.407255  8348 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.089s
I20260709 13:04:17.494753  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:17.513890  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98402.4us
I20260709 13:04:17.474777  8540 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.408586  8713 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.524736  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.525496  1661 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
W20260709 13:04:17.520605  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 117004us
I20260709 13:04:17.493103  8747 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.520674  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.509589  8705 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.522123  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.493388  8656 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:17.499244  8636 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.505705  8730 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:17.514973  1735 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:17.513931  8683 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.752s	user 0.011s	sys 0.003s
I20260709 13:04:17.513430  8739 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.499044  8468 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:17.553314  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 115398us
I20260709 13:04:17.534830  1668 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:17.528815  8692 raft_consensus.cc:993] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:04:17.532655  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:17.503234  8637 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.567068  8476 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:04:17.553574  8643 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.185s	user 0.012s	sys 0.004s
I20260709 13:04:17.525334  8504 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:17.525591  8621 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.528111  8540 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:17.585870  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53308.9us
I20260709 13:04:17.605820  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:17.582705  8476 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.572182  8748 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:17.582428  8739 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.633538  8540 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.634464  8739 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.636089  8540 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.604863  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:17.638553  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:17.638495  8756 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:17.558257  8747 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.640640  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:17.614656  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.641917  1735 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:17.642050  8747 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.642903  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.643534  1735 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:17.644572  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.568425  8667 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:17.613956  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:17.619161  8738 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.327s	user 0.002s	sys 0.000s
I20260709 13:04:17.619978  8476 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.621284  8504 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.621829  8736 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:17.622509  8743 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 13:04:17.581459  8744 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:17.618377  8636 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:17.682296  8744 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:17.682559  8636 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.683265  8744 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.684906  8744 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:17.562841  8705 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.646221  8737 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:17.658824  8728 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:17.690423  8705 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.690919  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:17.660507  8748 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 13:04:17.612382  8643 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.691601  1666 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 17
I20260709 13:04:17.498946  8733 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.294s
I20260709 13:04:17.625809  8470 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:17.577520  8730 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.625303  8713 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.615834  8637 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:17.697674  8018 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:17.698276  8637 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.698798  8637 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:17.699712  8730 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.577626  8683 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.635911  8739 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:17.701668  1738 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:17.636785  8656 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.677332  8476 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.702991  8656 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.577718  8746 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.614993  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:17.705122  8760 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.701663  8713 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.005s
I20260709 13:04:17.697757  8470 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:17.659540  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45673.4us
I20260709 13:04:17.707693  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:17.707419  8470 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.705150  8746 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.708660  8470 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:17.662602  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61100.9us
I20260709 13:04:17.709612  8649 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:17.710497  8746 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.677529  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.711257  8746 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:17.711604  1668 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:17.679878  8543 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.611943  8468 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.681618  8743 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:17.648245  8667 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.654842  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:17.718983  1735 raft_consensus.cc:2376] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16. Re-sending same reply.
I20260709 13:04:17.612167  8692 raft_consensus.cc:1081] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:04:17.705790  8760 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.710405  1634 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.680413  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:17.731491  1664 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:17.678426  8761 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:04:17.705871  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 90954.2us
I20260709 13:04:17.690287  8728 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.658047  8411 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:17.757114  8761 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.757265  8411 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:17.758304  8761 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.758453  8411 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.719311  8348 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 13:04:17.760679  8348 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:17.762216  8768 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:17.726419  1588 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:17.719722  1666 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:17.740830  1738 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:04:17.685313  8636 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.739554  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.697793  8733 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:17.742863  8643 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.738875  8754 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 13:04:17.743427  8468 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:17.744017  8695 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.859s	user 0.012s	sys 0.002s
W20260709 13:04:17.754129  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 43791us
I20260709 13:04:17.754808  8739 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:17.755125  8543 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:17.756476  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:17.701723  8752 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:17.707716  8713 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:17.761098  8348 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.714144  8746 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:17.742924  8683 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.808319  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.769459  8728 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.813650  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.784233  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.827121  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:17.727569  8504 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
W20260709 13:04:17.824716  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68296.1us
I20260709 13:04:17.803630  8540 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:17.832072  2317 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:17.834378  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:17.769220  8737 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 13:04:17.779536  8733 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:17.835064  1735 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:17.822480  8695 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:17.814339  8739 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.826130  8635 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 13:04:17.841698  8768 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.842410  8543 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:17.843194  8730 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:17.822356  8752 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.832741  8736 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:17.832813  8636 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:17.850589  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:17.828927  8765 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:17.853977  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:17.833637  8540 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.839010  8760 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:17.846376  8727 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.114s
I20260709 13:04:17.836233  8621 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:17.853183  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 177394us
I20260709 13:04:17.923009  1735 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:17.845219  8637 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.926155  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:17.927177  1735 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:17.875696  1587 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:17.933379  1708 leader_election.cc:400] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16.
I20260709 13:04:17.934036  1708 leader_election.cc:403] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 13:04:17.860188  8736 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.922555  8540 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:17.855816  8543 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.938628  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:17.853935  8739 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:17.858114  8752 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:17.860189  8636 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:17.931763  8621 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:17.975983  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125882us
I20260709 13:04:17.940478  8736 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:17.977515  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:17.941758  8387 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:17.945749  8543 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:17.987972  1557 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.570s (SENT)
W20260709 13:04:17.983959  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 156545us
I20260709 13:04:17.874099  8730 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:17.974973  8657 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.997161  1587 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:17.860105  8768 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:17.997753  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59221.2us
I20260709 13:04:17.999758  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:18.000039  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.922462  8348 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.009277  1591 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:04:18.009357  1735 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:17.975178  8756 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.967806  8636 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:18.000445  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0422186309999688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
I20260709 13:04:18.002363  1584 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:17.975284  8681 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:17.975674  8713 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:17.975441  8748 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:17.940147  8769 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 13:04:18.007009  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:17.942487  8018 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 13:04:18.017153  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:18.000310  8761 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.005798  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.033424  1592 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:04:18.020931  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.028615  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.016884  8681 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.035131  1591 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:18.035557  1735 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:17.939534  8746 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.040133  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.041438  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.027766  8748 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.050236  1632 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:18.002291  8657 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:17.968837  8733 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:18.060410  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 72440.5us
I20260709 13:04:18.019127  8411 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:04:18.049688  8348 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.017910  8756 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.053939  8748 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.070968  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:18.073555  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.078269  1735 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
W20260709 13:04:18.081063  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.040536  8761 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.091629  8470 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:18.092262  8470 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.064112  8643 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:18.094295  8772 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
W20260709 13:04:18.096458  8764 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 13:04:18.093899  8643 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.040025  8713 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:18.075946  8719 log.cc:927] Time spent T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.989s	user 0.000s	sys 0.008s
I20260709 13:04:18.064834  8778 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:18.126328  1590 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:18.099210  8643 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.154336  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.155166  1735 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:18.145535  8733 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:18.171833  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.172609  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:18.147971  8018 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:18.147780  8637 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:18.183548  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 112745us
I20260709 13:04:18.183887  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.184830  1735 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.185913  8787 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:18.193642  8770 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.205s	user 0.002s	sys 0.000s
I20260709 13:04:18.198845  8779 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.200060  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.200631  8766 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:18.200840  1741 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:18.202096  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.199721  8779 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.206311  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.182066  8761 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.216917  8782 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:18.210477  8730 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.217383  8637 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:18.218227  8681 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.222146  8540 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:18.224261  8339 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.821s	user 0.013s	sys 0.001s
I20260709 13:04:18.226913  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.227582  1734 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:18.228505  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.229245  8470 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.230391  8713 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:18.207056  8476 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.002s
I20260709 13:04:18.231664  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.235705  1589 raft_consensus.cc:1217] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Deduplicated request from leader. Original: 13.8->[14.9-16.11]   Dedup: 16.11->[]
I20260709 13:04:18.236231  1589 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [14.9-16.11]
I20260709 13:04:18.224797  8339 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:18.258056  8744 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.005s
I20260709 13:04:18.281636  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.282361  1739 raft_consensus.cc:2376] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21. Re-sending same reply.
I20260709 13:04:18.282620  8779 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.301880  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.302587  1589 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.301311  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:18.278447  8750 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.005s	sys 0.009s
I20260709 13:04:18.272954  8657 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:18.327244  8750 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.818s	user 0.006s	sys 0.009s
I20260709 13:04:18.267882  1735 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:18.346726  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.347723  1735 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:18.361889  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:18.362728  8784 log.cc:927] Time spent T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:04:18.363988  8636 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:18.365614  8728 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.514s	user 0.009s	sys 0.004s
I20260709 13:04:18.366110  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.366693  8739 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:18.368839  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 219335us
I20260709 13:04:18.371244  1738 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:18.369484  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 319277us
I20260709 13:04:18.373199  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:18.374116  8728 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.376659  8728 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.378921  1589 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:18.380698  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.381469  1589 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:18.383381  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.384259  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:18.386013  1632 leader_election.cc:400] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:04:18.368590  8657 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.386478  1632 leader_election.cc:403] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:18.390617  1738 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:18.392189  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:18.394451  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.400338  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.379992  8348 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 13:04:18.404035  8783 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.404628  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.405778  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 671) took 1472 ms. Trace:
I20260709 13:04:18.394181  8775 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:18.423723  2310 rpcz_store.cc:276] 0709 13:04:16.932937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.938263 (+  5326us) service_pool.cc:224] Handling call
0709 13:04:16.938605 (+   342us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:17.268061 (+329456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.405376 (+1137315us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54006e0fd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:18.405675 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":755651968}
I20260709 13:04:18.412418  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.423511  1736 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.411600  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.426602  2312 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:04:18.413425  2317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1015) took 1894 ms (client timeout 2515 ms). Trace:
I20260709 13:04:18.427253  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.427841  1591 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:18.413265  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.430150  2309 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:18.430219  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3388) took 2058 ms. Trace:
I20260709 13:04:18.430693  1593 rpcz_store.cc:276] 0709 13:04:16.371927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.372117 (+   190us) service_pool.cc:224] Handling call
0709 13:04:16.372453 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:17.139141 (+766688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.429952 (+1290811us) spinlock_profiling.cc:243] Waited 766 ms on lock 0x7b540025c454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:18.430154 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1761002880}
I20260709 13:04:18.416424  8667 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.432399  1593 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:18.417810  8796 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17.
I20260709 13:04:18.398800  8602 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:18.420877  1589 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [16.9-19.10]
I20260709 13:04:18.438432  1593 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:04:18.438907  8636 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.442273  8543 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.442624  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:18.383849  8752 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.432744  8667 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.434852  8602 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:18.442724  8768 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.443284  1739 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:18.443792  8756 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.444540  8543 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.444976  1592 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:18.447518  8739 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.392958  8657 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.418773  8339 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.448511  8783 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.412917  8736 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.423506  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.524820  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.524921  1735 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:18.525460  2309 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:18.427497  2317 rpcz_store.cc:269] 0709 13:04:16.519026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.519161 (+   135us) service_pool.cc:224] Handling call
0709 13:04:18.413221 (+1894060us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400573ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:18.413349 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2503826688}
I20260709 13:04:18.462031  8791 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:18.462153  8650 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.462276  8775 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:18.437397  8540 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.464730  8749 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:18.539422  8749 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:18.539608  8775 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.540408  8749 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.466697  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.542287  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.543417  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2583) took 1119 ms. Trace:
I20260709 13:04:18.468957  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.472681  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.545243  1586 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:18.545850  1589 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
W20260709 13:04:18.473762  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.545949  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.547170  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.543916  1733 rpcz_store.cc:276] 0709 13:04:17.423515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.427937 (+  4422us) service_pool.cc:224] Handling call
0709 13:04:18.542099 (+1114162us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:18.543326 (+  1227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26699008}
W20260709 13:04:18.475517  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 89793us
I20260709 13:04:18.461230  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.554014  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.554745  1659 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:18.500207  1591 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:18.500671  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.501284  8398 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.501950  8697 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:18.503276  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:18.504083  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:18.505601  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.473261  8746 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:18.518901  1734 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:18.519126  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:18.522668  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:18.523856  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.573041  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:18.573675  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 51097.3us
I20260709 13:04:18.574373  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.534126  8391 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:04:18.535590  1618 heartbeater.cc:646] Failed to heartbeat to 127.0.107.254:39071 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.107.254:39071 timed out after 15.000s (SENT)
I20260709 13:04:18.577832  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.542474  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.468123  8387 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 15. Current term is 16.
I20260709 13:04:18.506346  8018 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 13:04:18.563417  8783 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.516453  8411 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 13:04:18.555219  8791 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.564826  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.565044  1587 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:18.568028  2315 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.589138  2317 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:18.569018  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.571848  1591 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:18.543897  8752 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.593370  1735 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:18.594460  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.595288  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.568009  8650 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.527707  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:18.533430  8786 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 13:04:18.574661  1584 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 13:04:18.572664  8746 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:18.576625  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.538035  8602 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.578578  1733 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:18.579664  2316 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:18.581684  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.578550  1590 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:18.560879  1592 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:18.577756  8391 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:18.587590  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110637us
I20260709 13:04:18.596117  1735 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:18.573763  1666 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:18.596844  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.597023  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:18.599113  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71474.5us
I20260709 13:04:18.606218  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:18.600291  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.582422  8705 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:18.618170  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:18.609239  2281 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:18.575125  1664 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:18.604074  1736 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:18.605295  1659 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:18.606293  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.620167  1736 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:18.613905  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:18.621876  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.622552  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.622656  8320 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:18.622759  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:18.608148  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.615751  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.616356  1587 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:18.623134  1735 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:18.607824  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.624625  1660 raft_consensus.cc:2368] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:18.627233  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.624562  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.629453  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.604519  8658 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:18.623263  8320 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.604638  8650 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.633704  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.575470  8398 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.634393  8746 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:18.630390  8656 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:18.645466  1665 raft_consensus.cc:2368] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:18.609910  8387 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:18.616250  8796 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.617492  8733 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:18.632296  8751 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.034s	user 0.004s	sys 0.010s
I20260709 13:04:18.640141  1733 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:18.650810  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.625187  8705 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:18.579389  8697 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:18.730166  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79426.7us
I20260709 13:04:18.667277  1592 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:18.649258  8391 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.638695  8683 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:18.689854  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 83676.4us
I20260709 13:04:18.690577  8728 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.691864  1659 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:04:18.724524  8387 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.692035  8018 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:18.693647  8786 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:18.725287  8796 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.694936  1662 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:18.714130  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.696126  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.730230  8733 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.733708  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:18.738078  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.741232  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.741804  1664 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:18.695855  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.743489  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.755993  1592 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:18.689899  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:18.695725  1663 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 13:04:18.726593  8751 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.232s	user 0.004s	sys 0.011s
I20260709 13:04:18.742614  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:18.716579  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 93870.1us
I20260709 13:04:18.696216  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.757352  1662 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:18.649212  8792 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.649724  8658 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.659134  8320 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:18.726392  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.771993  1733 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:18.772267  1664 raft_consensus.cc:1217] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Deduplicated request from leader. Original: 15.10->[16.11-16.11]   Dedup: 16.11->[]
I20260709 13:04:18.772832  1664 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 13:04:18.728068  2315 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:18.729051  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.774188  1584 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 13:04:18.683173  8348 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 13:04:18.739060  8748 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.608s	user 0.007s	sys 0.004s
I20260709 13:04:18.739517  8736 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:18.739837  8796 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.748910  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:18.759305  1665 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:18.738695  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.763934  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.789610  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 11.
I20260709 13:04:18.740052  8387 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.767122  1663 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:04:18.768067  1667 raft_consensus.cc:2393] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:18.769317  2312 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.716274  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.748250  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2600) took 1120 ms. Trace:
I20260709 13:04:18.756996  8809 raft_consensus.cc:2764] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:04:18.730324  8705 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:18.724215  8656 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:18.774922  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.773015  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.772848  1634 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.740265  8657 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.733287  8683 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.771422  8792 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.776023  8391 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:18.790196  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2582) took 1367 ms. Trace:
I20260709 13:04:18.790251  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:18.766057  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:18.741297  8749 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:18.744156  8733 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.756266  8760 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:18.824123  8650 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:18.820390  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.737884  8728 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.935856  8728 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:18.809986  8798 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:18.908116  8540 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:18.824712  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.834762  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.771937  8658 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:18.859017  2316 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:18.860332  8672 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.865367  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:18.863969  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.775813  8348 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:18.741964  8182 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 13:04:18.744841  8727 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.114s
I20260709 13:04:18.777274  8339 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 13:04:18.828305  8398 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.836696  8744 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:18.932494  8504 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:18.872192  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:18.859921  8801 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:18.871858  1734 rpcz_store.cc:276] 0709 13:04:17.627825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.229287 (+601462us) service_pool.cc:224] Handling call
0709 13:04:18.747956 (+518669us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b5400338e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:18.748141 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":650897280}
I20260709 13:04:18.861617  8761 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.890482  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.890560  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:18.909101  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136270us
W20260709 13:04:18.910385  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.739207  8018 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:18.943078  8733 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:18.923801  1737 rpcz_store.cc:276] 0709 13:04:17.423185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.427505 (+  4320us) service_pool.cc:224] Handling call
0709 13:04:18.748699 (+1321194us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:18.790130 (+ 41431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26166656}
I20260709 13:04:18.970134  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.926137  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:18.924120  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 133924us
I20260709 13:04:18.839279  2280 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:18.913465  8683 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:18.939145  1589 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:18.941407  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:18.943704  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.944049  2313 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:18.914520  8736 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:04:18.943140  8665 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.504s	user 0.011s	sys 0.010s
I20260709 13:04:18.957356  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.943627  8798 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:18.791426  8748 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:18.870781  8791 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:18.983839  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:18.984032  1592 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:18.970927  1662 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:19.078871  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:18.949409  8760 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:18.970947  1735 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:18.958295  8801 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.009908  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:19.087859  1589 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:18.987102  8733 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:19.008876  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 169658us
I20260709 13:04:18.953352  8348 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.014941  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:18.984536  8665 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.550s	user 0.011s	sys 0.011s
I20260709 13:04:18.985548  8656 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.009546  8798 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.025079  8398 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:18.951184  8540 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.109340  1589 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:18.963792  8018 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:19.045158  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:18.962937  8745 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.264s	user 0.000s	sys 0.005s
I20260709 13:04:18.956638  8744 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:19.070065  1741 raft_consensus.cc:2368] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:18.985941  8765 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:18.956247  8672 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.016440  8697 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:18.982815  8761 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.012748  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.084721  8411 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.088927  8801 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.090785  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:19.096563  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.169217  1666 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:18.949575  8627 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:19.089098  8667 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.611s	user 0.012s	sys 0.001s
I20260709 13:04:19.081324  8182 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:19.103305  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.106686  1592 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:19.107852  1735 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:19.084738  8782 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 13:04:19.145758  1589 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:19.178525  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.099697  8504 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:19.145321  1668 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:19.108654  8821 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:19.172101  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 77580.8us
I20260709 13:04:19.201651  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:19.202610  1735 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:04:19.176306  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.180632  1660 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:19.183002  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.122910  8765 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:19.184079  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:19.185160  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.185595  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.187505  8672 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.141916  8398 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:19.194226  8504 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.183879  8760 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.195500  8681 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:19.196358  8813 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 13:04:19.214999  1661 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:19.170207  8411 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.198089  8821 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.230172  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:19.234396  8817 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:04:19.235132  8765 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.236090  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:19.205209  8339 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.181252  8182 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.216717  8736 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:19.223031  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1222) took 2385 ms. Trace:
I20260709 13:04:19.177982  8627 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.263069  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:19.256398  1735 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:19.246271  2311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1225) took 2318 ms. Trace:
I20260709 13:04:19.247787  2312 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:19.263306  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 171225us
I20260709 13:04:19.267448  1740 rpcz_store.cc:276] 0709 13:04:16.837175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.837806 (+   631us) service_pool.cc:224] Handling call
0709 13:04:16.838171 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:16.838414 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:16.991358 (+152944us) log.cc:844] Serialized 14 byte log entry
0709 13:04:17.156615 (+165257us) log.cc:844] Serialized 14 byte log entry
0709 13:04:17.159005 (+  2390us) log.cc:844] Serialized 14 byte log entry
0709 13:04:19.212569 (+2053564us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b18003e04a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287
0709 13:04:19.212672 (+   103us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:19.212723 (+    51us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:19.212779 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.212846 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:19.222968 (+ 10122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7510,"prepare.queue_time_us":281904,"prepare.run_cpu_time_us":209,"prepare.run_wall_time_us":4179,"raft.queue_time_us":326362,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":27151,"spinlock_wait_cycles":3657984,"thread_start_us":362990,"threads_started":3}
I20260709 13:04:19.275776  2313 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:19.279026  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.218009  8018 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.270287  2311 rpcz_store.cc:276] 0709 13:04:16.928322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:16.936423 (+  8101us) service_pool.cc:224] Handling call
0709 13:04:16.936751 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:17.199297 (+262546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:17.199345 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:18.815845 (+1616500us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b2c00070ac8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:18.821022 (+  5177us) log.cc:844] Serialized 26 byte log entry
0709 13:04:18.949372 (+128350us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:18.949409 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:18.949708 (+   299us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:19.245822 (+296114us) raft_consensus.cc:1685] finished
0709 13:04:19.245851 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:19.246139 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12868,"raft.queue_time_us":75212,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":18004,"spinlock_wait_cycles":8375168,"thread_start_us":349992,"threads_started":2,"wal-append.queue_time_us":276713}
I20260709 13:04:19.250434  8348 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.283113  8665 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:19.226003  8744 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.278887  8809 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.257753  8798 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.302591  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:19.269750  8339 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.295830  8765 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.295136  8748 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:19.332983  1661 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:19.335094  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:19.336140  1593 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:19.305183  8665 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:19.333081  8744 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:19.325533  8809 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.359338  8647 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.363013  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:19.364842  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:19.379302  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:19.382054  8818 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
W20260709 13:04:19.383991  8831 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 13:04:19.386495  2318 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 13:04:19.341701  8736 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.388361  1737 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 23. Ops: []
I20260709 13:04:19.389068  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2596) took 1805 ms. Trace:
I20260709 13:04:19.327338  8398 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.390419  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.606766  8647 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:19.456815  8411 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:19.427002  8793 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:19.607416  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.454545  8809 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:19.451701  8348 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.609443  8782 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:19.438598  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.610810  8761 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:19.567961  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.610073  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:19.610100  8705 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.698s	user 0.009s	sys 0.003s
I20260709 13:04:19.607528  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:19.612841  1664 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:19.458230  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2609) took 1780 ms. Trace:
I20260709 13:04:19.448788  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.618587  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:19.466265  8796 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.635s	user 0.006s	sys 0.007s
I20260709 13:04:19.619623  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.389389  1737 rpcz_store.cc:276] 0709 13:04:17.584019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.387849 (+1803830us) service_pool.cc:224] Handling call
0709 13:04:19.388167 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.388820 (+   653us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.389012 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:19.618577  1736 rpcz_store.cc:276] 0709 13:04:17.678069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.460852 (+782783us) service_pool.cc:224] Handling call
0709 13:04:19.457961 (+997109us) spinlock_profiling.cc:243] Waited 52.3 ms on lock 0x7b5400404a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:19.458156 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":120385920}
W20260709 13:04:19.444950  8749 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.512s	user 0.009s	sys 0.000s
I20260709 13:04:19.376189  8733 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:19.341917  8665 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:19.391127  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.454473  1735 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:19.681453  1664 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:19.685075  1666 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:19.685779  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 294653us
W20260709 13:04:19.537633  8391 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.580s	user 0.009s	sys 0.003s
I20260709 13:04:19.687868  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:19.688563  1668 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:19.621009  1660 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.623633  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:19.613871  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:19.695307  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81527.9us
W20260709 13:04:19.696282  8755 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.129s	user 0.000s	sys 0.003s
I20260709 13:04:19.696552  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.626504  2311 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:04:19.612051  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.698499  8744 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:19.698576  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.699025  8744 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.699436  8744 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.700012  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:19.700654  1667 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:19.701861  8833 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:19.703114  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.705214  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.446180  8540 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:18.462172   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:19.705789  8621 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.707803  8730 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:19.709970  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:19.764792  1661 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:19.765328  1661 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:19.711630  1667 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
W20260709 13:04:19.712293  8658 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.753s	user 0.010s	sys 0.003s
I20260709 13:04:19.713712  8752 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
W20260709 13:04:19.715406  8683 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.702s	user 0.009s	sys 0.001s
I20260709 13:04:19.716216  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.781061  1589 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:19.716596  8743 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:19.717821  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.685075  1593 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:19.783041  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:19.719849  8801 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.543s	user 0.007s	sys 0.003s
I20260709 13:04:19.720337  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:19.797256  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 76978.4us
I20260709 13:04:19.798339  2283 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.443948  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:19.463116  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2601) took 1835 ms (client timeout 2201 ms). Trace:
I20260709 13:04:19.742632  1668 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:19.799122  1741 rpcz_store.cc:269] 0709 13:04:17.628096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.203031 (+574935us) service_pool.cc:224] Handling call
0709 13:04:19.462852 (+1259821us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b5400108bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:19.463039 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1936470912}
I20260709 13:04:19.783735  1586 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:19.618901  8320 consensus_meta.cc:287] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.828s	user 0.032s	sys 0.000s
I20260709 13:04:19.434978  8798 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.442528  8821 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.758338  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1224) took 1951 ms. Trace:
I20260709 13:04:19.761006  1734 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:19.454071  8339 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:19.761811  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74871.5us
I20260709 13:04:19.465898  8182 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.763164  8540 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:19.809953  1736 rpcz_store.cc:276] 0709 13:04:17.806570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.757027 (+1950457us) service_pool.cc:224] Handling call
0709 13:04:19.757390 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.757671 (+   281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:19.757725 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:19.757792 (+    67us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:19.757849 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.758042 (+   193us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:19.758256 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:19.811578  1736 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:19.629248  8827 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.665052  8650 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.779309  1661 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:19.818867  1661 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:19.623250  8705 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.437067  8728 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:19.809036  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 103856us
I20260709 13:04:19.823992  1586 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:19.777081  8752 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:19.774856  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.814417  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:19.827597  1742 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.817314  1665 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:19.828583  1660 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:19.821863  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.830883  1589 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 13:04:19.817745  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.835484  8656 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:19.837203  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:19.823266  8827 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.843883  8827 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:19.847411  8792 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.808326  8602 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:19.852684  1739 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2607) took 2175 ms (client timeout 2600 ms). Trace:
I20260709 13:04:19.854278  1733 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:19.855242  8728 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:19.856158  1585 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
W20260709 13:04:19.857180  8814 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.774s	user 0.000s	sys 0.002s
I20260709 13:04:19.857388  8705 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.857963  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3403) took 1144 ms. Trace:
I20260709 13:04:19.763365  8733 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 13:04:19.806099  8657 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:19.812721  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3331) took 1833 ms. Trace:
I20260709 13:04:19.868532  1736 rpcz_store.cc:276] 0709 13:04:17.979340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.811174 (+1831834us) service_pool.cc:224] Handling call
0709 13:04:19.811405 (+   231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.812365 (+   960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.812657 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:19.814649  8809 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.756018  8796 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.858349  1585 rpcz_store.cc:276] 0709 13:04:18.714064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.714250 (+   186us) service_pool.cc:224] Handling call
0709 13:04:18.714684 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.856573 (+1141889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.857680 (+  1107us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540025add4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:19.857895 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2623921024}
W20260709 13:04:19.853196  1739 rpcz_store.cc:269] 0709 13:04:17.677424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.340795 (+663371us) service_pool.cc:224] Handling call
0709 13:04:19.852495 (+1511700us) spinlock_profiling.cc:243] Waited 80.5 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:19.852613 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185079168}
I20260709 13:04:19.860035  1705 leader_election.cc:400] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:19.824368  1586 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:19.889662  1705 leader_election.cc:403] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:19.816277  8540 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.861856  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:19.802183  1593 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [16.9-19.10]
I20260709 13:04:19.891970  1664 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:19.866135  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:19.876899  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 253227us
I20260709 13:04:19.893571  1589 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.892750  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.897037  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:19.892514  8724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.428s	user 0.016s	sys 0.004s
I20260709 13:04:19.780860  8658 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.894304  1593 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:19.901635  8339 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:19.899641  8798 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:19.802927  8683 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.904017  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:19.825753  1741 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
I20260709 13:04:19.812052  8320 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.865335  8824 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.915118  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.866986  8657 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:19.916349  1739 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:04:19.930647  2317 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:19.912981  8387 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 18. Current term is 19.
I20260709 13:04:19.914439  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:19.905879  8724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.450s	user 0.016s	sys 0.004s
I20260709 13:04:19.926319  1741 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:19.939139  1741 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:19.896843  8728 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:19.826797  8650 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.806320  8801 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:19.895975  8540 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:19.898609  8809 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.932514  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3333) took 1950 ms. Trace:
I20260709 13:04:19.961009  1739 rpcz_store.cc:276] 0709 13:04:17.981668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.889415 (+1907747us) service_pool.cc:224] Handling call
0709 13:04:19.915993 (+ 26578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.929647 (+ 13654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:19.932447 (+  2800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:19.921180  1589 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:19.930271  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.926870  8733 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:19.928580  8705 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:19.947249  8658 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:19.948776  8320 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:19.954936  1705 leader_election.cc:400] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:19.933431  8657 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:19.943068  8769 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:19.958259  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:19.963227  1592 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:19.933874  8621 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:19.933017  8798 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:19.934281  1662 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:19.928733  8752 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:19.992686  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64626.7us
I20260709 13:04:19.969585  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:19.989368  1585 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:04:19.932325  8824 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:19.995594  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:19.995208  1589 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:20.005358  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:19.953298  8683 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:19.999684  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:19.956723  8728 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.015525  8728 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.015242  8798 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.002060  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:20.021476  1742 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
I20260709 13:04:19.956974  8844 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
W20260709 13:04:20.022716  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2586) took 2598 ms (client timeout 1738 ms). Trace:
I20260709 13:04:20.011914  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:20.023169  1742 rpcz_store.cc:269] 0709 13:04:17.424367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.428884 (+  4517us) service_pool.cc:224] Handling call
0709 13:04:19.814244 (+2385360us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:20.022635 (+208391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23053568}
I20260709 13:04:19.965009  8744 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.032671  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:20.039613  8839 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 13:04:20.023511  8743 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:20.042265  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2615) took 1473 ms. Trace:
I20260709 13:04:19.995186  8621 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.046203  8665 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.042783  1734 rpcz_store.cc:276] 0709 13:04:18.568407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.013001 (+444594us) service_pool.cc:224] Handling call
0709 13:04:20.042065 (+1029064us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b54003534d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.042195 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1490766208}
I20260709 13:04:20.051447  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.052408  2311 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:20.053164  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.053850  2315 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:20.054447  2315 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:20.045248  8621 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.057695  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2621) took 1487 ms. Trace:
I20260709 13:04:20.063151  1735 rpcz_store.cc:276] 0709 13:04:18.570483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.290135 (+719652us) service_pool.cc:224] Handling call
0709 13:04:20.057536 (+767401us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.057637 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":299106048}
I20260709 13:04:20.053475  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.054217  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1143) took 1602 ms. Trace:
I20260709 13:04:20.065313  1591 rpcz_store.cc:276] 0709 13:04:18.452178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.555894 (+103716us) service_pool.cc:224] Handling call
0709 13:04:18.556261 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:18.590321 (+ 34060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.053951 (+1463630us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b54000019d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:20.054149 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34724096}
I20260709 13:04:20.066818  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:20.024497  8752 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:20.054288  8843 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:20.049386  8665 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.070222  8743 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:20.070649  8695 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:20.059255  8786 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:20.029970  8728 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:20.057322  8796 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.031095  8844 consensus_queue.cc:1243] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:04:20.089263  1735 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:04:20.105614  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3343) took 2071 ms. Trace:
I20260709 13:04:20.064873  8705 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.106118  1735 rpcz_store.cc:276] 0709 13:04:18.033984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.064165 (+2030181us) service_pool.cc:224] Handling call
0709 13:04:20.088992 (+ 24827us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:20.105161 (+ 16169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.105526 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.071552  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.103299  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.073782  1734 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 13:04:20.109131  8769 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:20.110451  8705 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.110698  8851 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:20.111722  8851 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.111809  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3342) took 2084 ms. Trace:
W20260709 13:04:20.112198  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 116663us
I20260709 13:04:20.113246  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.113992  1741 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.112941  1734 rpcz_store.cc:276] 0709 13:04:18.027252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.050681 (+2023429us) service_pool.cc:224] Handling call
0709 13:04:20.073366 (+ 22685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:20.111523 (+ 38157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.111744 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.114655  8733 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:20.115549  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.120792  8387 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:20.121316  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.122597  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.128712  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3400) took 1515 ms. Trace:
I20260709 13:04:20.137303  1741 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:20.134721  1590 rpcz_store.cc:276] 0709 13:04:18.613007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.613179 (+   172us) service_pool.cc:224] Handling call
0709 13:04:18.613539 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:18.613984 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:18.614016 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:18.625678 (+ 11662us) log.cc:844] Serialized 26 byte log entry
0709 13:04:20.056205 (+1430527us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b18004417c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:20.090566 (+ 34361us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:20.090631 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.091746 (+  1115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:20.122124 (+ 30378us) raft_consensus.cc:1685] finished
0709 13:04:20.122144 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.128634 (+  6490us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2425,"raft.queue_time_us":849,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":14285,"spinlock_wait_cycles":8173440,"thread_start_us":34094,"threads_started":1,"wal-append.queue_time_us":1487386}
I20260709 13:04:20.129914  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.109228  8398 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:20.137949  8647 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.003s
I20260709 13:04:20.072247  8847 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 13:04:20.137696  8665 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:20.150180  1738 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:04:20.150067  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:42028 (request call id 1227) took 2327 ms. Trace:
I20260709 13:04:20.147789  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1228) took 1701 ms. Trace:
I20260709 13:04:20.153321  1734 rpcz_store.cc:276] 0709 13:04:18.446298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.128911 (+1682613us) service_pool.cc:224] Handling call
0709 13:04:20.147188 (+ 18277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.147396 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:20.147423 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:20.147472 (+    49us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:20.147512 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.147566 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.147711 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.157788  1741 rpcz_store.cc:276] 0709 13:04:17.822176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.800189 (+1978013us) service_pool.cc:224] Handling call
0709 13:04:20.138002 (+337813us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.135093  8786 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:20.150842  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.146215  8398 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:20.169734  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 146181us
I20260709 13:04:20.172339  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.182149  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:20.183759  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78774us
I20260709 13:04:20.183866  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:20.184831  1557 connection.cc:256] RPC call timeout handler was delayed by -1.5927429899999765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260709 13:04:20.185211  1660 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
W20260709 13:04:20.185187  1557 connection.cc:256] RPC call timeout handler was delayed by -1.4186345609999194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
I20260709 13:04:20.159384  8647 raft_consensus.cc:2764] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:04:20.187731  1741 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:20.188408  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1230) took 1642 ms. Trace:
I20260709 13:04:20.186950  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:20.188800  1741 rpcz_store.cc:276] 0709 13:04:18.546437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.175248 (+1628811us) service_pool.cc:224] Handling call
0709 13:04:20.187549 (+ 12301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.188143 (+   594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.188344 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.189401  1590 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:20.171058  8791 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.191838  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.186955  8786 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:20.193627  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.194329  1660 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:20.178661  8730 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:20.197350  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2548) took 1121 ms. Trace:
I20260709 13:04:20.177979  8398 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.197808  1662 rpcz_store.cc:276] 0709 13:04:19.075424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.078585 (+  3161us) service_pool.cc:224] Handling call
0709 13:04:20.197108 (+1118523us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b54001a9854. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.197285 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1961411328}
I20260709 13:04:20.198609  1667 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:20.178390  8665 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:20.200078  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1346429899999748s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
W20260709 13:04:20.202747  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1294884269999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
W20260709 13:04:20.204615  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0594434879999994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260709 13:04:20.205021  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0392921629999705s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1799s for a response
I20260709 13:04:20.205403  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:04:20.205915  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:20.206535  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:20.207182  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.208177  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.209892  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.205788  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:20.215119  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:20.211668  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2617) took 1642 ms (client timeout 1867 ms). Trace:
W20260709 13:04:20.216781  1733 rpcz_store.cc:269] 0709 13:04:18.569370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.772753 (+203383us) service_pool.cc:224] Handling call
0709 13:04:20.211464 (+1438711us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b5400352ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.211603 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244676992}
W20260709 13:04:20.221014  1588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3385) took 2793 ms (client timeout 1864 ms). Trace:
W20260709 13:04:20.221764  1588 rpcz_store.cc:269] 0709 13:04:17.410499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:17.410699 (+   200us) service_pool.cc:224] Handling call
0709 13:04:20.203647 (+2792948us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b540025e254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.203802 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723116416}
I20260709 13:04:20.200133  8730 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.223507  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:20.224854  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:20.227210  1588 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:20.227408  2315 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:20.203596  8861 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.228413  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.199151  8657 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.193320  8791 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:20.224891  8845 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 13:04:20.232230  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:20.240774  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:20.241629  1588 raft_consensus.cc:2410] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:04:20.231038  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.238224  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:20.242892  2313 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 13:04:20.243192  2313 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:20.247143  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.228386  8849 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:04:20.252364  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.177402  8851 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.227718  8744 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.227921  8730 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.256372  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3338) took 2253 ms. Trace:
I20260709 13:04:20.260308  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.262172  8847 raft_consensus.cc:2764] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 13:04:20.266983  8540 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.274000  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.274506  8796 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:20.300966  8656 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.301403  8656 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.301843  8656 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:20.234412  8861 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.305295  8861 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.239655  8398 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:20.306137  8398 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.306551  8398 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.256170  8855 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:20.310105  8855 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.310526  8855 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:20.311285  8656 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.256457  8863 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.277740  8665 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.314016  8863 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.314616  8863 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:20.278432  8849 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.278116  8730 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.318367  8796 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.319041  8796 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.319240  8849 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.320595  8730 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.279248  8468 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:20.321485  8468 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.322798  8796 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.322463  8468 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.323341  8796 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.280066  8657 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.281549  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:20.325237  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43827.2us
I20260709 13:04:20.326138  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.283475  8668 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 13:04:20.259155  1739 rpcz_store.cc:276] 0709 13:04:18.002925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.962218 (+1959293us) service_pool.cc:224] Handling call
0709 13:04:19.988315 (+ 26097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:20.008306 (+ 19991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:20.008341 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:20.022777 (+ 14436us) log.cc:844] Serialized 26 byte log entry
0709 13:04:20.159646 (+136869us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:20.159709 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.245926 (+ 86217us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b54001077d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:20.245984 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:20.246029 (+    45us) raft_consensus.cc:1685] finished
0709 13:04:20.246042 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:20.256290 (+ 10248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12495,"raft.queue_time_us":5304,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":3580,"spinlock_wait_cycles":32881024,"thread_start_us":129072,"threads_started":1,"wal-append.queue_time_us":132155}
I20260709 13:04:20.327951  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.284502  8751 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:20.331898  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.332547  1662 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:20.334659  1660 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []
I20260709 13:04:20.336141  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.337078  1664 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.338112  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.285113  8746 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.260915  2313 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:20.286346  8858 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 13:04:20.288728  1591 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:20.370788  1591 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.292353  1740 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [18.11-21.13]
I20260709 13:04:20.293901  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1020) took 1461 ms. Trace:
I20260709 13:04:20.372215  2317 rpcz_store.cc:276] 0709 13:04:18.832388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.834373 (+  1985us) service_pool.cc:224] Handling call
0709 13:04:20.293646 (+1459273us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b5400572e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.293829 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2253373312}
I20260709 13:04:20.340783  1588 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:20.347525  1664 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:20.328799  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
W20260709 13:04:20.366479  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.377696  1739 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:20.378751  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2620) took 1808 ms. Trace:
I20260709 13:04:20.379163  1739 rpcz_store.cc:276] 0709 13:04:18.570164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.328721 (+1758557us) service_pool.cc:224] Handling call
0709 13:04:20.377467 (+ 48746us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.378372 (+   905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.378670 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.380086  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2616) took 1811 ms. Trace:
I20260709 13:04:20.380548  1740 rpcz_store.cc:276] 0709 13:04:18.569091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.291410 (+1722319us) service_pool.cc:224] Handling call
0709 13:04:20.291958 (+   548us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:20.379774 (+ 87816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.380029 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.384159  1591 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.385541  1591 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:20.382982  8476 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:20.389731  1740 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:20.390630  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.330468  8844 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:20.369486  8847 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.395160  1739 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 13:04:20.395686  1592 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:20.395730  2313 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 11 FOLLOWER]: Advancing to term 12
I20260709 13:04:20.399948  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.405910  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3348) took 1825 ms. Trace:
I20260709 13:04:20.406327  1739 rpcz_store.cc:276] 0709 13:04:18.580069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.380049 (+1799980us) service_pool.cc:224] Handling call
0709 13:04:20.394789 (+ 14740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.405572 (+ 10783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.405844 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:20.396324  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58298.1us
I20260709 13:04:20.393890  8476 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:20.324528  8796 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.398432  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3349) took 1811 ms. Trace:
I20260709 13:04:20.383878  8677 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:20.410146  1557 leader_election.cc:400] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:20.440984  1557 leader_election.cc:403] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:20.411060  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.381916  8861 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:20.417836  1740 rpcz_store.cc:276] 0709 13:04:18.586963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.381301 (+1794338us) service_pool.cc:224] Handling call
0709 13:04:20.381673 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.398131 (+ 16458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.398370 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.444237  1740 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:20.444886  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3351) took 1847 ms. Trace:
I20260709 13:04:20.445235  1740 rpcz_store.cc:276] 0709 13:04:18.597837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.443538 (+1845701us) service_pool.cc:224] Handling call
0709 13:04:20.443865 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.444615 (+   750us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.444828 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:20.445981  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 253988us
I20260709 13:04:20.446708  1740 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:20.392920  8751 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:20.440083  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.448426  1666 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.447501  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1232) took 1509 ms. Trace:
I20260709 13:04:20.440317  8656 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:20.445348  8731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 3.106s	user 0.001s	sys 0.012s
I20260709 13:04:20.453431  2313 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
I20260709 13:04:20.461014  1591 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:20.463949  8705 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:20.440845  8677 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:20.401612  8844 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.455581  8621 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:20.452805  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.491752  1740 rpcz_store.cc:276] 0709 13:04:18.937828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.446075 (+1508247us) service_pool.cc:224] Handling call
0709 13:04:20.446408 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.447197 (+   789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.447437 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.496347  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:20.496305  8731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.168s	user 0.003s	sys 0.012s
I20260709 13:04:20.500859  1590 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 12.
W20260709 13:04:20.370635  8695 consensus_queue.cc:1175] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "df6a18af06d543b2931837f5e6bab95f" responder_term: 12 status { last_received { term: 11 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 11. Current term is 12. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:20.502211  1591 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.497591  1666 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
W20260709 13:04:20.499289  8747 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
W20260709 13:04:20.500954  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 48261us
I20260709 13:04:20.505635  1740 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:20.515021  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3354) took 1524 ms. Trace:
I20260709 13:04:20.515555  1740 rpcz_store.cc:276] 0709 13:04:18.990866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.502599 (+1511733us) service_pool.cc:224] Handling call
0709 13:04:20.503189 (+   590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.514583 (+ 11394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.514904 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.510418  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.500486  8621 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.519202  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.500980  8844 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.459805  8751 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.517122  1740 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:20.533118  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3355) took 1514 ms. Trace:
I20260709 13:04:20.533468  1740 rpcz_store.cc:276] 0709 13:04:19.018890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.516442 (+1497552us) service_pool.cc:224] Handling call
0709 13:04:20.516791 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.532791 (+ 16000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.533042 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:20.464676  8476 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.502759  8656 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:20.540894  1591 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:20.548925  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.558661  8540 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:20.523242  8621 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.565346  1591 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 13:04:20.567508  8627 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.568254  8627 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:20.568764  8627 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:20.568892  8751 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:20.531450  8517 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:20.572155  8517 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.572593  8517 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:20.575909  8476 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:20.548579  8657 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 13:04:20.620383  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 211202us
I20260709 13:04:20.622907  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.584650  8656 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:20.632723  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:20.635826  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 86979.4us
I20260709 13:04:20.735105  8863 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.679893  8656 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:20.739643  8852 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.594s	user 0.005s	sys 0.003s
I20260709 13:04:20.779428  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.646277  8627 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.760118  8517 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.774612  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.734202  8540 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:20.771979  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.708324  1659 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:04:20.774133  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:20.640936  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 108929us
W20260709 13:04:20.729183  8876 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:20.781471  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.680950  8621 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.683398  8801 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:20.549274  8677 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.673943  8860 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:20.639325  8705 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:20.684397  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.710222  8855 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.780233  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.682682  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.623288  8796 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.755615  1586 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.730683  8398 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.753455  8824 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.670187  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.637210  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.661571  1589 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
W20260709 13:04:20.748734  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.729444  8861 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:20.700784  8650 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.696470  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3271) took 1762 ms. Trace:
I20260709 13:04:20.826403  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.827337  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.824263  8732 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.823247  8879 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:20.761389  8809 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.831739  2311 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:04:20.834867  8791 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:20.837102  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.900305  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.904492  1665 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:20.905548  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.766115  8844 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:20.906286  1665 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:20.907485  8863 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.907895  8844 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.908239  1590 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:20.908329  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:20.767517  8847 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:20.910365  1660 raft_consensus.cc:2376] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17. Re-sending same reply.
I20260709 13:04:20.919545  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.911418  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.912176  1589 pending_rounds.cc:85] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 12
I20260709 13:04:20.920670  1589 pending_rounds.cc:107] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 13:04:20.900135  2317 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:20.921022  1589 raft_consensus.cc:2889] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:20.840852  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:20.921887  2315 raft_consensus.cc:2393] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:20.922456  1586 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:20.912772  1667 rpcz_store.cc:276] 0709 13:04:18.933963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.766866 (+832903us) service_pool.cc:224] Handling call
0709 13:04:20.198368 (+431502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.199029 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:20.696126 (+497097us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b54001a9850. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:20.696374 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":991053184}
I20260709 13:04:20.914345  1664 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:20.914284  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 231626us
I20260709 13:04:20.925611  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:20.926941  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:20.915956  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.918000  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:20.918084  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:20.928511  1588 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:20.928624  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:20.928934  1591 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:20.923725  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:20.930178  1589 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:20.932974  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 295751us
I20260709 13:04:20.934882  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.936640  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:20.938472  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:20.939321  2311 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:20.940292  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:20.941447  2313 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:20.942945  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.923978  1668 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:20.947233  1588 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:04:20.949018  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.913538  8705 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.917790  8801 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:20.909690  8860 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:20.957321  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:20.963075  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:20.937458  8796 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:20.940418  8792 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:20.968145  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.949685  8832 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:20.941275  8824 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:20.947315  8863 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:20.952616  8847 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:20.970902  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:20.954386  8801 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.954084  8705 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:20.976904  1666 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:20.968739  8650 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:20.923945  8732 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.923635  8879 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.978806  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:20.946547  8667 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 13:04:20.969825  8875 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:20.987175  1666 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:20.951627  8861 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:20.997103  1589 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:20.973158  8849 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:20.998660  1586 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 13:04:21.002099  1586 pending_rounds.cc:85] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:04:20.979472  8468 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:21.003000  8468 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:21.003064  8667 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:21.003602  8667 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:21.004132  8468 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:20.991968  8656 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 13:04:20.973481  8824 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:20.970183  8792 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.002460  1586 pending_rounds.cc:107] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 13:04:21.018782  1586 raft_consensus.cc:2889] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:21.019186  1586 pending_rounds.cc:107] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 13:04:21.019470  1586 raft_consensus.cc:2889] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:20.981035  8650 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.960255  8774 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 13:04:20.977609  8863 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:20.979100  8801 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.035360  8650 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:21.037019  8863 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:21.041843  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.043243  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:21.043846  1665 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.019310  8824 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:21.016394  8796 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:21.020332  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1149) took 1779 ms. Trace:
I20260709 13:04:21.010289  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:21.050486  1666 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:21.017370  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68548.8us
I20260709 13:04:21.022509  8517 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.019567  8878 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 13:04:21.026654  8621 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:20.981866  8739 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 13:04:20.999454  8860 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.068665  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3340) took 3049 ms. Trace:
I20260709 13:04:21.069553  8860 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.030179  8398 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.031280  8855 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.076613  8786 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:21.047768  8796 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.081192  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3441) took 1218 ms. Trace:
I20260709 13:04:21.081987  1590 rpcz_store.cc:276] 0709 13:04:19.863023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.519919 (+656896us) service_pool.cc:224] Handling call
0709 13:04:20.548385 (+ 28466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.908971 (+360586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.080880 (+171909us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b5400002654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:21.081110 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826910080}
W20260709 13:04:21.084863  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 876056us
I20260709 13:04:20.988387  8844 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.032255  8849 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:21.032958  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3261) took 2336 ms (client timeout 1642 ms). Trace:
W20260709 13:04:21.087920  1659 rpcz_store.cc:269] 0709 13:04:18.696163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.738463 (+ 42300us) service_pool.cc:224] Handling call
0709 13:04:21.032689 (+2294226us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001a81d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:21.032855 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4410351360}
I20260709 13:04:21.029239  8732 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.033576  8018 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:21.020967  8792 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.050282  1586 rpcz_store.cc:276] 0709 13:04:19.240984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.807221 (+566237us) service_pool.cc:224] Handling call
0709 13:04:19.807563 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.755116 (+947553us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b4c0027b5c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:04:21.019993 (+264877us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.020244 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63753472}
I20260709 13:04:21.051385  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:21.062052  8885 log.cc:927] Time spent T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:04:21.062858  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2604) took 3433 ms. Trace:
I20260709 13:04:21.110255  1737 rpcz_store.cc:276] 0709 13:04:17.628937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.756282 (+2127345us) service_pool.cc:224] Handling call
0709 13:04:19.756671 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:19.795872 (+ 39201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:19.795926 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:19.796007 (+    81us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:19.796073 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.062093 (+1266020us) spinlock_profiling.cc:243] Waited 38.8 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:21.062150 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.062774 (+   624us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89339648}
W20260709 13:04:21.063943  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2602) took 3435 ms (client timeout 3339 ms). Trace:
W20260709 13:04:21.112090  1738 rpcz_store.cc:269] 0709 13:04:17.628380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:18.393338 (+764958us) service_pool.cc:224] Handling call
0709 13:04:21.063651 (+2670313us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b540014f0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:21.063847 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4020682496}
I20260709 13:04:21.065475  1736 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 13:04:21.088646  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.113852  1736 pending_rounds.cc:85] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 10
I20260709 13:04:21.114326  1736 pending_rounds.cc:107] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 13:04:21.114645  1736 raft_consensus.cc:2889] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:21.115621  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64325.5us
I20260709 13:04:21.029286  8879 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.090999  1666 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:21.122191  1665 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:21.125631  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:21.126613  1586 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.069161  1742 rpcz_store.cc:276] 0709 13:04:18.018894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.024913 (+2006019us) service_pool.cc:224] Handling call
0709 13:04:20.025256 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.058070 (+ 32814us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:20.091376 (+ 33306us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.067720 (+976344us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b180041b5a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:04:21.067930 (+   210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:21.067992 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:21.068057 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.068142 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.068591 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7818,"prepare.queue_time_us":33254,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":893943,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":92,"spinlock_wait_cycles":5126912,"thread_start_us":103735,"threads_started":2,"wal-append.queue_time_us":87252}
I20260709 13:04:21.100878  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:21.102017  1663 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:21.120622  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.123900  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.089622  8387 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:21.144717  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.144963  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:21.138796  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 183133us
I20260709 13:04:21.139076  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3345) took 2581 ms. Trace:
I20260709 13:04:21.146257  1733 rpcz_store.cc:276] 0709 13:04:18.557519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.217998 (+1660479us) service_pool.cc:224] Handling call
0709 13:04:20.218452 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.222177 (+  3725us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:20.245209 (+ 23032us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.138646 (+893437us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:21.138715 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:21.138779 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.138861 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.139009 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":735023,"prepare.queue_time_us":22801,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"thread_start_us":157856,"threads_started":1,"wal-append.queue_time_us":259486}
I20260709 13:04:21.096912  8743 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:21.147401  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.152520  1590 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:21.155916  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:21.109819  8832 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.172060  1586 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:21.172688  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3350) took 2578 ms. Trace:
I20260709 13:04:21.145874  8730 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.178664  1739 rpcz_store.cc:276] 0709 13:04:18.594699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.407311 (+1812612us) service_pool.cc:224] Handling call
0709 13:04:20.407724 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:20.413866 (+  6142us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:20.523272 (+109406us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.172338 (+649066us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:21.172392 (+    54us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:21.172446 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.172514 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.172631 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":515253,"prepare.queue_time_us":45136,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"thread_start_us":193807,"threads_started":2,"wal-append.queue_time_us":152164}
I20260709 13:04:21.124701  8844 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:21.135355  8890 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:21.188642  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 198281us
I20260709 13:04:21.196676  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:21.197772  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.200049  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.190155  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:21.202363  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.203377  1733 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:21.206838  1739 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:04:21.154554  8743 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:21.212515  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.216652  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.147958  8732 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 13:04:21.176770  8809 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.217386  1737 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.158046  8848 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:21.180434  8728 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.184674  8730 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.227146  2312 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:21.228466  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.229103  8730 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.231678  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.232594  1742 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:21.234905  1737 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:21.201030  8890 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:21.204624  8733 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.172387  8855 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 13:04:21.182904  8747 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 13:04:21.216444  8743 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:21.273190  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 206287us
I20260709 13:04:21.171429  8705 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.189203  8832 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.177239  8863 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.277909  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.235601  8844 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:21.296502  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.297293  1733 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.298259  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.225912  2311 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:21.301400  2318 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:21.304270  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.304986  1738 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.312234  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:21.289969  8891 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.313407  2283 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:21.313722  8891 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.314438  8879 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:21.314868  8879 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:21.315384  8879 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:21.316759  8891 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.316881  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3344) took 2759 ms. Trace:
I20260709 13:04:21.155181  8849 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.317382  1741 rpcz_store.cc:276] 0709 13:04:18.557130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.189806 (+1632676us) service_pool.cc:224] Handling call
0709 13:04:20.201088 (+ 11282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:21.078510 (+877422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:21.078546 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:21.079399 (+   853us) log.cc:844] Serialized 26 byte log entry
0709 13:04:21.160138 (+ 80739us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:21.160204 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.160324 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:21.316648 (+156324us) raft_consensus.cc:1685] finished
0709 13:04:21.316670 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.316816 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":876706,"raft.queue_time_us":882404,"raft.run_cpu_time_us":51,"raft.run_wall_time_us":50,"thread_start_us":79781,"threads_started":1,"wal-append.queue_time_us":235022}
I20260709 13:04:21.320524  8658 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:21.321295  8658 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:21.181993  8786 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.321768  8650 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.322403  8658 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.255419  8848 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:21.325928  8848 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.245967  8809 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.329692  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:21.289604  8733 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.334872  1739 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:04:21.337028  1668 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:21.338514  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:21.329015  8677 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:21.256611  8890 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.358387  8809 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.354084  8731 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 13:04:21.360802  8801 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.361943  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.384564  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.363080  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.385367  1742 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:21.363554  8733 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.364794  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1229) took 2892 ms. Trace:
I20260709 13:04:21.394620  1734 rpcz_store.cc:276] 0709 13:04:18.472868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.154963 (+1682095us) service_pool.cc:224] Handling call
0709 13:04:20.175544 (+ 20581us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:20.176123 (+   579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:20.176167 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:20.179061 (+  2894us) log.cc:844] Serialized 26 byte log entry
0709 13:04:20.325434 (+146373us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:20.405169 (+ 79735us) log.cc:844] Serialized 14 byte log entry
0709 13:04:21.364419 (+959250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:21.364502 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:21.364550 (+    48us) raft_consensus.cc:1685] finished
0709 13:04:21.364564 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:21.364714 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1617514,"prepare.queue_time_us":76909,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":24953,"raft.queue_time_us":773669,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":13216,"thread_start_us":348771,"threads_started":2,"wal-append.queue_time_us":310247}
I20260709 13:04:21.365034  1584 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:21.343190  1634 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.368825  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.397961  1590 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:21.399545  1590 raft_consensus.cc:1240] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:21.397166  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54033.2us
I20260709 13:04:21.370352  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:21.246573  8728 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.413077  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.413939  1591 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:21.413841  8398 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:21.414121  8891 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:21.415361  8809 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.415930  8809 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.417328  8540 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:21.376550  1741 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:21.413079  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.378338  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:21.379593  1588 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:21.420289  1593 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:21.393594  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:21.403889  1589 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:21.385762  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:21.365571  8900 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:21.293258  8870 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260709 13:04:21.371137  8855 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 13:04:21.412158  8798 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:21.420032  1734 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:21.383245  8892 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:21.355666  8650 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.421222  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:21.417852  8540 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:21.403440  8832 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:21.423465  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 327967us
I20260709 13:04:21.444653  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.447144  1589 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:21.423053  8733 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:21.461982  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:21.464047  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:21.464123  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.431211  8878 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.015s
I20260709 13:04:21.429381  8752 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.473618  1668 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:21.471176  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:21.485947  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.489264  1738 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:21.490693  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:21.485181  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 201491us
I20260709 13:04:21.495663  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:21.496588  2313 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:21.500316  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:21.447888  8892 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.484417  1742 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:21.508507  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.509250  1742 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.511933  8892 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:21.523947  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:21.525079  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.446381  8798 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:21.539063  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:21.543892  1738 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:21.433400  8900 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.472687  8517 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:21.450703  8621 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:21.475700  8832 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.558962  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:21.482378  8809 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.559095  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.572238  1741 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:21.502727  8730 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.543531  8847 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:21.539381  8695 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.468683  8733 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.583884  1735 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.579789  1742 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:21.590277  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.574186  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.598009  1738 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:21.576122  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:21.599325  1741 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.536358  8752 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.592135  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.603134  1742 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.605340  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.606423  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:21.608227  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.576481  8728 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.614636  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:21.547026  8879 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.612932  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.583707  8809 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:21.575050  8517 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:21.584456  8847 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:21.619863  1738 raft_consensus.cc:2410] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 21 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
I20260709 13:04:21.591622  8901 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.625660  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.629154  1741 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.632839  1735 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:21.633677  1742 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:21.622319  1668 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:21.584899  8730 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.575155  8832 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.610641  8863 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.634117  1733 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:21.645718  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.647007  1733 raft_consensus.cc:2368] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:21.650846  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.622870  8809 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.654829  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.655805  1742 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:21.594357  8695 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.657346  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.658452  1738 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.658694  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:21.626295  8769 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:21.628594  8728 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:21.677063  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125437us
I20260709 13:04:21.663405  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.670836  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.681134  1733 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 12: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.685205  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.686004  1738 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:21.630319  8892 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 13:04:21.684170  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.689606  1741 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:21.696691  1742 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.698369  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.731428  1742 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.662094  8769 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.735095  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.735889  1733 raft_consensus.cc:2376] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19. Re-sending same reply.
I20260709 13:04:21.745453  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.657800  8809 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:21.622844  8798 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.757630  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.758473  1733 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:21.679315  8786 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:21.677129  8811 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 13:04:21.763722  1742 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:21.744840  8769 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:21.632617  8901 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:21.779153  1557 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.582s (SENT)
I20260709 13:04:21.776829  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.781325  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.783179  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.694100  8765 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 13:04:21.786479  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.738266  8517 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.820247  1733 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:21.790709  8909 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 13:04:21.793373  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.826902  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:21.797997  8900 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.798415  8832 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.826967  1668 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:21.790831  8695 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.799031  8752 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:21.804315  8878 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:21.736794  8912 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.108s
I20260709 13:04:21.806759  8730 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:21.808017  8915 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:04:21.809255  8879 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:21.810832  8907 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 13:04:21.813683  1742 raft_consensus.cc:2393] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:21.825853  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:21.830341  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51243.7us
I20260709 13:04:21.831476  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:21.851715  1557 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.672s (SENT)
I20260709 13:04:21.834389  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.836093  1738 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:21.838181  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:21.852746  1557 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.738s (SENT)
I20260709 13:04:21.841600  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.853157  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.853588  1666 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:21.842300  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.842703  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.857714  1733 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:21.843201  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.858712  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.859859  1659 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:21.841955  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 148088us
I20260709 13:04:21.693642  8847 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:21.870134  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.833791  8848 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:21.890237  8705 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.860262  1667 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:04:21.868784  1557 leader_election.cc:400] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:21.893357  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.893729  1557 leader_election.cc:403] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:21.895291  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.833719  8870 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.871645  1659 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:21.862471  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:21.874132  8792 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.740s	user 0.014s	sys 0.000s
I20260709 13:04:21.870138  8468 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:21.842124  8878 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:21.881146  8908 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 13:04:21.883443  8824 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.821s	user 0.010s	sys 0.003s
W20260709 13:04:21.883731  8743 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.586s	user 0.007s	sys 0.005s
I20260709 13:04:21.886636  8913 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 13:04:21.889946  8018 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.001s
I20260709 13:04:21.899292  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:21.906040  8705 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:21.903100  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.907265  1733 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:21.904747  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:21.927599  1666 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:04:21.942320  2315 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:21.942611  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.943238  1733 raft_consensus.cc:2376] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:04:21.943637  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.900156  8848 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:21.941331  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.899786  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:21.926829  8677 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.001s
I20260709 13:04:21.921748  8912 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:21.947232  8912 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:21.947410  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47692.4us
I20260709 13:04:21.947640  8912 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:21.949101  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.950444  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.951334  8918 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:21.952123  8918 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.952497  8918 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:21.955430  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.962394  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:21.963055  1733 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.963516  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.955832  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:21.943725  8387 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:21.923179  8792 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.968470  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.953738  8863 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:21.945741  8752 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Leader election won for term 12
I20260709 13:04:21.954820  8913 consensus_queue.cc:1243] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:04:21.955120  8929 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:21.977633  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:21.967134  8540 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:21.982352  8540 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.987326  8929 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:21.988044  8929 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:21.989705  8849 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 13:04:21.937760  8743 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:21.995021  8903 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.395s	user 0.002s	sys 0.000s
I20260709 13:04:21.996330  8743 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:21.998081  8801 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.998256  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:21.998919  1737 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:21.999122  2315 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:22.000695  1706 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:22.001004  1706 leader_election.cc:403] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 13:04:21.934283  8824 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.003922  8824 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:21.975119  8705 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:21.967293  8848 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:22.005155  8848 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:21.965978  8018 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:22.010610  8018 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.011052  8848 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.014652  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3332) took 4033 ms. Trace:
I20260709 13:04:22.015159  1736 rpcz_store.cc:276] 0709 13:04:17.980932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:19.869246 (+1888314us) service_pool.cc:224] Handling call
0709 13:04:21.064929 (+1195683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.134176 (+ 69247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.014372 (+880196us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b540014f0d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:22.014569 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2748326272}
I20260709 13:04:22.016496  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:21.922854  8870 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.020663  8870 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:22.020787  8890 consensus_meta.cc:287] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.627s	user 0.007s	sys 0.006s
I20260709 13:04:22.021324  8890 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.022789  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.022884  8809 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.023727  1741 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:22.024019  8890 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.025655  2313 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:21.960608  8875 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:22.017191  1736 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:22.024636  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.026532  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.042546  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.044341  1585 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:21.945993  8900 consensus_queue.cc:1059] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 13:04:22.011196  8018 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.045961  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:22.048297  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 70768.4us
I20260709 13:04:22.055297  2315 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:04:22.055572  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.049046  1741 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:22.067210  1737 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:22.067739  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.060276  8900 consensus_queue.cc:1243] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:04:22.046537  8752 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:22.054100  8929 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.075991  8929 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.076251  8855 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:22.070067  8649 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:21.975440  8387 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.072422  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.055303  8875 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:22.071324  8878 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.026340  8926 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:22.046903  8791 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.081928  1733 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:22.084095  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.025099  8809 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.100934  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.101006  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.087792  8855 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:22.085270  8647 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.085947  8796 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.093109  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:22.079777  8891 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.002s	sys 0.005s
I20260709 13:04:22.076792  8749 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.095760  1593 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:22.106655  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.097707  1590 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:22.075181  8752 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.113392  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.086364  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.088289  8849 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.083683  8733 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.059643  8863 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:22.086320  8926 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.106025  8809 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.125705  8926 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.125903  8809 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.127601  8904 raft_consensus.cc:3055] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:22.128360  8809 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.128052  8904 raft_consensus.cc:740] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:22.047806  8705 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:22.062633  8918 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.090718  8540 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:22.104457  1589 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:22.096557  1586 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:22.108212  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.137810  2312 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:22.091156  8913 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:22.108884  8796 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.115423  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:22.107126  8891 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.003s	sys 0.005s
I20260709 13:04:22.107872  8824 consensus_queue.cc:1059] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:04:22.135525  8863 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:22.146763  8863 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.147660  8918 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.148317  8936 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 13:04:22.149295  8827 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 13:04:22.149839  8765 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:22.150604  8827 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:22.150993  1591 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 13:04:22.150964  8827 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.110103  8749 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.152743  8827 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.091457  8801 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 13:04:22.108115  8929 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:22.174383  8929 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.174839  8929 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.174932  1666 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 13:04:22.177088  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.139461  8875 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.131665  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:22.133700  8896 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.715s	user 0.003s	sys 0.000s
I20260709 13:04:22.133109  8855 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.137887  1590 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.099262  1742 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:22.138957  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3445) took 1223 ms. Trace:
I20260709 13:04:22.183211  1587 rpcz_store.cc:276] 0709 13:04:20.915430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.915993 (+   563us) service_pool.cc:224] Handling call
0709 13:04:20.916516 (+   523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:20.920423 (+  3907us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:20.920458 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:20.920994 (+   536us) log.cc:844] Serialized 26 byte log entry
0709 13:04:22.013195 (+1092201us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b18004b9a60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:22.077789 (+ 64594us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:22.077830 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.077900 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:22.134168 (+ 56268us) raft_consensus.cc:1685] finished
0709 13:04:22.134209 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:22.138891 (+  4682us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3423,"raft.queue_time_us":4377,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":8203,"spinlock_wait_cycles":8998144,"thread_start_us":64348,"threads_started":1,"wal-append.queue_time_us":1211338}
I20260709 13:04:22.183998  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.184501  1733 raft_consensus.cc:2376] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21. Re-sending same reply.
I20260709 13:04:22.184723  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.118583  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:22.121646  8870 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:22.138940  8540 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.192425  8540 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:22.192684  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74201.3us
I20260709 13:04:22.193712  8936 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:22.192971  8798 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:22.195807  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:22.196653  8832 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:22.197059  8832 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:22.124991  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 464498us
I20260709 13:04:22.197836  8936 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:22.198376  1708 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.199491  8730 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.199717  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.200623  8730 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.142418  8913 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.154059  8749 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.115801  8849 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.156364  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:22.203037  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46758.1us
I20260709 13:04:22.203464  8913 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.204089  8849 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.197429  8832 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:22.156702  8796 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.158255  8677 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:22.207101  8749 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.208751  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.158607  8792 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.160022  8751 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:22.210915  8751 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:22.211254  8751 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.211627  8743 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:22.212159  8751 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.214293  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46322 (request call id 1162) took 1094 ms. Trace:
I20260709 13:04:22.214694  1584 rpcz_store.cc:276] 0709 13:04:21.119761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.120357 (+   596us) service_pool.cc:224] Handling call
0709 13:04:22.214067 (+1093710us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54003b15d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:22.214221 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":538095488}
I20260709 13:04:22.189826  8827 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.216427  1584 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:22.107688  8791 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.197419  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:22.222831  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:22.223284  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.223769  8677 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:22.224313  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:22.224665  8677 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.225746  8921 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:22.162248  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.227552  1741 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:22.166378  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.228415  1734 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:22.167912  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.108736  8647 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.226095  8677 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.236697  8657 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:22.170197  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.237703  1668 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:22.170800  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.238582  8647 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.171355  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.238430  1584 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:22.239531  1665 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:22.171872  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.180366  1593 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 12.
I20260709 13:04:22.241650  8792 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:22.242292  8792 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.244689  8792 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.245690  8731 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.246225  8731 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.246912  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.248304  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.248756  8731 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.223421  8791 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.250025  1737 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:22.251981  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.252689  1737 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.185271  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.114115  8739 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:22.190158  1587 raft_consensus.cc:2410] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 20 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:22.189232  8018 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.191943  8891 raft_consensus.cc:3055] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:22.223490  8476 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:22.229738  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61896.9us
I20260709 13:04:22.230101  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:22.260574  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.260905  2311 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:22.164114  8468 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:22.267810  8018 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.223306  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.232281  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.209427  1740 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.225102  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.213406  8751 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:22.275588  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.279377  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:22.280720  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:22.281080  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48891.2us
I20260709 13:04:22.282012  8018 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:22.282891  1587 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:22.275969  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.266052  8891 raft_consensus.cc:740] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.287691  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.289069  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.289436  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:22.291584  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.256122  8769 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.290761  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.299899  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:22.301008  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 72101.1us
I20260709 13:04:22.305163  1741 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.301160  1705 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17.
I20260709 13:04:22.143477  8809 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.088s
I20260709 13:04:22.226955  8736 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:22.230791  8870 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.266041  8739 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.335253  2318 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:22.273249  8860 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.339715  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.342432  1735 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.346186  1590 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:22.348982  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.338109  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:22.265801  8657 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:22.265784  8468 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:22.353231  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:22.321894  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:22.314606  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.354725  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:22.354775  1587 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
W20260709 13:04:22.356194  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.365274  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.356617  1734 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:22.339083  2311 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:22.361403  1736 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:22.299367  8732 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:22.173899  8912 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.377760  1741 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:22.367269  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 75753.9us
I20260709 13:04:22.379086  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.380080  8912 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.380510  1737 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:22.377425  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.385365  1740 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:22.385607  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.386701  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:22.376827  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 90001.6us
I20260709 13:04:22.388093  1740 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:22.389530  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.389111  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.391013  1737 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 13:04:22.391225  1740 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 13:04:22.392200  2312 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:22.392527  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3360) took 1034 ms. Trace:
I20260709 13:04:22.393123  1740 rpcz_store.cc:276] 0709 13:04:21.358146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.390544 (+1032398us) service_pool.cc:224] Handling call
0709 13:04:22.390872 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.392211 (+  1339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.392449 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:22.394232  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:22.395227  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.395875  1589 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.396245  1740 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 13:04:22.397432  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3366) took 1037 ms. Trace:
I20260709 13:04:22.397585  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.398310  1738 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
I20260709 13:04:22.398130  1740 rpcz_store.cc:276] 0709 13:04:21.359880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.395326 (+1035446us) service_pool.cc:224] Handling call
0709 13:04:22.395779 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.397072 (+  1293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.397359 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:22.400694  1737 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:22.335475  8736 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.408310  1735 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:22.401347  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3363) took 1042 ms. Trace:
I20260709 13:04:22.402760  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.350821  8739 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:22.404424  1738 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:22.410547  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3368) took 1050 ms. Trace:
I20260709 13:04:22.411127  1738 rpcz_store.cc:276] 0709 13:04:21.360364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.399487 (+1039123us) service_pool.cc:224] Handling call
0709 13:04:22.404160 (+  4673us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.410055 (+  5895us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.410473 (+   418us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:22.411020  1735 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:22.400897  1740 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:22.381003  8809 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:22.302716  8890 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.083s
I20260709 13:04:22.351791  8870 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.413012  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3372) took 1051 ms. Trace:
I20260709 13:04:22.412745  1737 rpcz_store.cc:276] 0709 13:04:21.359144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.392878 (+1033734us) service_pool.cc:224] Handling call
0709 13:04:22.400473 (+  7595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.401089 (+   616us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.401294 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37120}
I20260709 13:04:22.303135  8695 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.095s
I20260709 13:04:22.189815  8904 consensus_queue.cc:260] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.449316  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3373) took 1027 ms. Trace:
I20260709 13:04:22.298563  8848 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:22.447846  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.451359  1735 rpcz_store.cc:276] 0709 13:04:21.387539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.409901 (+1022362us) service_pool.cc:224] Handling call
0709 13:04:22.410547 (+   646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.414396 (+  3849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.414628 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:22.452967  1738 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 13:04:22.382071  8791 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.462342  1738 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 13:04:22.463086  1738 pending_rounds.cc:85] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:04:22.463433  1738 pending_rounds.cc:107] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 13:04:22.463809  1738 raft_consensus.cc:2889] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:22.400360  8728 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:22.480260  1738 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:22.480904  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:22.393301  8878 raft_consensus.cc:2764] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 13:04:22.363899  8860 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.493718  1737 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:22.487221  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:22.434952  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 211646us
I20260709 13:04:22.498333  1738 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:04:22.499963  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.312868  8705 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.495378  8732 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:22.410426  8912 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.073s
I20260709 13:04:22.458016  8792 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:22.481328  8827 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 13:04:22.481864  8904 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:22.507375  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.450477  1740 rpcz_store.cc:276] 0709 13:04:21.368292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.400098 (+1031806us) service_pool.cc:224] Handling call
0709 13:04:22.400400 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.412588 (+ 12188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.412945 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:22.530445  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.531041  1589 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:22.501585  8476 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 13:04:22.483091  8743 consensus_queue.cc:1175] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ff44a4643e01482cb6b6a79b099fa766" responder_term: 14 status { last_received { term: 12 index: 8 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: [13.8-13.8]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 13:04:22.539940  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.539940  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.503610  8749 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.474611  8731 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 13:04:22.545859  8774 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:22.544184  1589 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22.
I20260709 13:04:22.548316  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.407711  8891 consensus_queue.cc:260] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:22.549791  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147080us
I20260709 13:04:22.550793  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:22.551559  2315 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:04:22.519706  8918 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:22.551409  2318 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.566159  8731 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:22.567103  8731 consensus_queue.cc:1243] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:22.534696  8945 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:22.568799  8945 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.570104  8945 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.571470  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2626) took 1018 ms. Trace:
I20260709 13:04:22.571928  1742 rpcz_store.cc:276] 0709 13:04:21.553307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.026264 (+472957us) service_pool.cc:224] Handling call
0709 13:04:22.571228 (+544964us) spinlock_profiling.cc:243] Waited 51.9 ms on lock 0x7b54000e1d54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:22.571409 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119287552}
I20260709 13:04:22.573827  8728 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.519764  8827 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.553453  1740 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 13:04:22.573486  8769 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.576596  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.576608  1733 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:22.578248  1735 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 13:04:22.578603  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:22.586499  1735 pending_rounds.cc:85] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 11
I20260709 13:04:22.552111  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.580886  1742 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 16. Current term is 17. Ops: []
W20260709 13:04:22.588287  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.586839  1735 pending_rounds.cc:107] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 13:04:22.588919  1735 raft_consensus.cc:2889] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:22.586804  1590 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.593367  1589 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 25: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:22.552388  8774 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.586501  8827 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:22.595160  8476 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:22.598474  8942 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 13:04:22.599404  8943 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 13:04:22.601986  8667 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.609767  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.615931  8848 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.574580  8728 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.604941  8832 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.618137  8939 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.258s	user 0.000s	sys 0.002s
I20260709 13:04:22.510555  8736 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.621255  8926 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.620023  8929 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.626430  8860 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:22.630024  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.625336  8540 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.510406  8468 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.511142  8657 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.659837  1740 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:04:22.666123  2318 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:22.654476  8476 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.698927  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.720801  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3446) took 1803 ms. Trace:
I20260709 13:04:22.721441  1585 rpcz_store.cc:276] 0709 13:04:20.917270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:20.931739 (+ 14469us) service_pool.cc:224] Handling call
0709 13:04:20.950435 (+ 18696us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.047270 (+1096835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.720443 (+673173us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540025dd54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:22.720695 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2514477312}
I20260709 13:04:22.723413  1663 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:22.723799  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.724350  1708 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 21. Current term is 22.
W20260709 13:04:22.725044  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.726567  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.727116  1664 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 13:04:22.724757  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 351168us
I20260709 13:04:22.551280  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:22.728488  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 177279us
I20260709 13:04:22.728643  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.728950  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.730280  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:22.703075  1590 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:04:22.664634  8863 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.656026  8954 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:22.681298  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.681684  8792 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.736133  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.736469  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.701051  8736 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:22.750989  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.752233  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.739528  1662 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:22.629580  8732 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.761044  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 394745us
I20260709 13:04:22.740324  8746 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:22.741309  8912 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:22.765111  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.765826  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:22.739763  8647 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:22.733951  8468 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:22.770596  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.750666  8731 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.750763  8798 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.773967  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.751246  8904 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.779472  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:22.773945  2283 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:22.783095  1661 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:22.788846  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.764137  8769 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:22.761456  8949 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 13:04:22.762578  8929 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.770569  8912 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:22.808769  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.815089  1590 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
W20260709 13:04:22.821156  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47404.1us
I20260709 13:04:22.826545  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.828792  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:22.753314  8863 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.775015  8953 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.863384  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.784442  8658 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:22.811502  8647 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.740569  8774 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:22.838989  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.816036  8912 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:22.814718  8958 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:22.839221  8848 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:22.785156  8731 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.855046  8918 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:22.869871  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.806141  8946 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 13:04:22.881676  1708 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:22.871616  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.788321  8798 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:22.886508  8848 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:22.894320  8540 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:22.874972  8647 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.889201  8947 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.226s	user 0.003s	sys 0.000s
I20260709 13:04:22.812570  8904 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.857522  8861 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 13:04:22.869369  8965 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:22.878669  8730 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.881479  8658 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:22.929875  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:22.914160  8769 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:22.942756  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.950479  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:22.951964  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.913892  8832 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:22.959758  1585 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:22.920185  8752 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:22.964174  8739 consensus_meta.cc:287] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.003s
I20260709 13:04:22.963938  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.964794  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1237) took 1754 ms. Trace:
I20260709 13:04:22.965837  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:22.881567  8958 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.929303  8926 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:22.969550  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.974370  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:22.978120  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:22.978402  1739 rpcz_store.cc:276] 0709 13:04:21.210808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.210978 (+   170us) service_pool.cc:224] Handling call
0709 13:04:21.211424 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:21.335348 (+123924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.964519 (+1629171us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001086d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:22.964731 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":282825088}
I20260709 13:04:22.986333  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.939114  8965 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:22.919171  8848 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:22.941715  8746 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:22.991813  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:23.037597  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:22.996714  8658 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:22.998926  8962 log.cc:927] Time spent T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 13:04:23.000674  8774 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:23.000928  8904 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:23.005455  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 414821us
I20260709 13:04:22.983558  8739 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:23.014762  8948 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 13:04:22.987269  8744 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:23.018338  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1246) took 1381 ms. Trace:
I20260709 13:04:23.017205  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:23.027086  8967 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 13:04:23.058038  8647 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:23.030905  8957 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.263s	user 0.000s	sys 0.002s
I20260709 13:04:23.059412  1662 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:23.060236  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1251) took 1004 ms. Trace:
I20260709 13:04:23.054324  2313 rpcz_store.cc:276] 0709 13:04:21.637232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:21.637380 (+   148us) service_pool.cc:224] Handling call
0709 13:04:21.637766 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.026051 (+388285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.017914 (+991863us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b5400610154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:23.018238 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891306368}
I20260709 13:04:23.033560  1584 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 13:04:23.034871  8964 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:04:22.995893  8731 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.038281  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:23.039638  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:23.068158  1737 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:23.068434  1592 raft_consensus.cc:2410] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:22.974401  8476 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.044616  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:23.072023  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:23.046967  8387 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.048400  1585 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 13:04:23.049160  8751 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.750s	user 0.013s	sys 0.000s
I20260709 13:04:23.050040  8946 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:23.053210  1742 raft_consensus.cc:1217] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Deduplicated request from leader. Original: 13.8->[15.9-18.10]   Dedup: 15.9->[18.10-18.10]
I20260709 13:04:23.093791  1742 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: [15.9-18.10]
I20260709 13:04:23.094880  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:23.095484  1742 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:23.053565  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:22.968159  8695 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 13:04:23.097941  8746 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.057175  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:23.100726  8946 consensus_queue.cc:1243] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:23.102017  8861 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:23.102437  8861 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:23.103262  8848 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.104043  8965 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.062887  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:23.060878  1738 rpcz_store.cc:276] 0709 13:04:22.056317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.499744 (+443427us) service_pool.cc:224] Handling call
0709 13:04:22.502149 (+  2405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.521519 (+ 19370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.521568 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:22.534802 (+ 13234us) log.cc:844] Serialized 26 byte log entry
0709 13:04:22.743377 (+208575us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:22.743444 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.743520 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.059961 (+316441us) raft_consensus.cc:1685] finished
0709 13:04:23.059994 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.060162 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18628,"raft.queue_time_us":26734,"spinlock_wait_cycles":6137728,"thread_start_us":177967,"threads_started":1,"wal-append.queue_time_us":412447}
W20260709 13:04:23.063369  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.081214  8920 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 13:04:23.084115  8792 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.087770  1593 raft_consensus.cc:1240] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:23.054119  8744 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:23.051291  8958 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.089972  1737 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:23.058076  8863 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.108373  1592 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 13:04:23.123463  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:22.986162  8752 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.124853  1742 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:23.126909  1737 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:23.128458  1738 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:23.126467  1742 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:23.080273  8731 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:23.142362  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.166725  1592 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:23.302966  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.311542  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.167951  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:23.315838  1592 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 13:04:23.110044  8476 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:23.322803  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:23.337329  1592 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
W20260709 13:04:23.298946  8870 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.847s	user 0.014s	sys 0.002s
I20260709 13:04:23.169526  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:23.154852  1664 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:23.219708  1593 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:23.452299  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3434) took 1062 ms. Trace:
I20260709 13:04:23.452831  1593 rpcz_store.cc:276] 0709 13:04:22.389837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.111292 (+721455us) service_pool.cc:224] Handling call
0709 13:04:23.219418 (+108126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.451985 (+232567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.452223 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:23.152769  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.341436  1632 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.783s (SENT) [suppressed 24 similar messages]
W20260709 13:04:23.382364  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.299356  8921 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.073s	user 0.014s	sys 0.005s
I20260709 13:04:23.144951  8832 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:23.256826  8744 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.479825  8832 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.480355  8744 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:23.482821  8832 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.316578  8635 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:23.229187  8827 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.566s	user 0.013s	sys 0.002s
W20260709 13:04:23.174984  8964 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 13:04:23.144490  8736 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.346s
I20260709 13:04:23.497815  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:23.495708  8953 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.623s	user 0.009s	sys 0.005s
I20260709 13:04:23.313134  8798 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.326336  8739 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.450583  8983 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:23.323321  8752 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.173259  8954 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:23.127130  8792 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.491174  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:23.559615  1592 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:23.148286  8751 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.349s
I20260709 13:04:23.149755  8929 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.314s
I20260709 13:04:23.150508  8952 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.352s
I20260709 13:04:23.491878  8921 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:23.496287  8855 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.103271  8861 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.142264  8975 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.382s
I20260709 13:04:23.467101  8870 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.526096  8860 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:23.509142  8635 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.575024  1585 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:04:23.531816  8953 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:23.583379  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3303) took 1304 ms. Trace:
I20260709 13:04:23.560983  8972 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:23.560503  8983 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:23.522861  8827 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.567299  8751 consensus_queue.cc:1243] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 19.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:04:23.612079  1660 rpcz_store.cc:276] 0709 13:04:22.285753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.285946 (+   193us) service_pool.cc:224] Handling call
0709 13:04:22.286418 (+   472us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:22.301743 (+ 15325us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:22.352280 (+ 50537us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.857782 (+505502us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.857989 (+   207us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:22.863768 (+  5779us) log.cc:844] Serialized 52 byte log entry
0709 13:04:22.863958 (+   190us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:22.864245 (+   287us) log.cc:844] Serialized 14 byte log entry
0709 13:04:22.865821 (+  1576us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.865902 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.582283 (+716381us) raft_consensus.cc:1685] finished
0709 13:04:23.582313 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.583168 (+   855us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54125,"prepare.queue_time_us":210075,"prepare.run_cpu_time_us":159,"prepare.run_wall_time_us":147,"raft.queue_time_us":78492,"raft.run_cpu_time_us":643,"raft.run_wall_time_us":47581,"thread_start_us":407835,"threads_started":2,"wal-append.queue_time_us":469903}
I20260709 13:04:23.624221  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3394) took 1386 ms. Trace:
I20260709 13:04:23.629942  1742 rpcz_store.cc:276] 0709 13:04:22.238082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.136970 (+898888us) service_pool.cc:224] Handling call
0709 13:04:23.489020 (+352050us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.489720 (+   700us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:23.509574 (+ 19854us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.623718 (+114144us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.623783 (+    65us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:23.623856 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.623931 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.624131 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":33285,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":68,"raft.queue_time_us":3815,"thread_start_us":113755,"threads_started":1}
W20260709 13:04:23.637094  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 295600us
I20260709 13:04:23.607184  8736 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.623347  8791 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.652426  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:23.668761  8989 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 13:04:23.670482  8981 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 13:04:23.642778  8983 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.673307  8018 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:23.673811  8018 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:23.674345  8982 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.152s	user 0.000s	sys 0.002s
I20260709 13:04:23.674769  8018 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.676481  8018 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:23.677479  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.648806  8736 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.589856  8855 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.682696  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:23.728806  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1253) took 1097 ms. Trace:
I20260709 13:04:23.729238  1740 rpcz_store.cc:276] 0709 13:04:22.631024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.724849 (+1093825us) service_pool.cc:224] Handling call
0709 13:04:23.728222 (+  3373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.728447 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.728481 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.728527 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:23.728568 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.728629 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.728743 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:23.731011  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:23.692191  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.683919  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:23.737846  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:23.739900  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:23.740859  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.628859  8972 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.643991  8979 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:23.676615  8959 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.767s	user 0.000s	sys 0.010s
I20260709 13:04:23.753325  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.755667  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:23.780583  8018 raft_consensus.cc:2764] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 13:04:23.762737  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:23.782440  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.783187  8910 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:23.786226  8993 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:04:23.722259  8792 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.788790  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3393) took 1550 ms. Trace:
I20260709 13:04:23.754595  8920 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:23.724951  8635 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.789355  1737 rpcz_store.cc:276] 0709 13:04:22.237847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.127668 (+889821us) service_pool.cc:224] Handling call
0709 13:04:23.216479 (+ 88811us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:23.217129 (+   650us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:23.257406 (+ 40277us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.578703 (+321297us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:23.588177 (+  9474us) log.cc:844] Serialized 26 byte log entry
0709 13:04:23.588449 (+   272us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:23.608245 (+ 19796us) log.cc:844] Serialized 14 byte log entry
0709 13:04:23.611922 (+  3677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.612015 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.788441 (+176426us) raft_consensus.cc:1685] finished
0709 13:04:23.788478 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.788698 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3152,"prepare.queue_time_us":48480,"prepare.run_cpu_time_us":191,"prepare.run_wall_time_us":8625,"raft.queue_time_us":9250,"raft.run_cpu_time_us":471,"raft.run_wall_time_us":26250,"thread_start_us":235006,"threads_started":1,"wal-append.queue_time_us":355842}
I20260709 13:04:23.792183  8887 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:23.792572  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:23.794620  1737 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:23.656297  8791 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.649727  8954 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.793236  8855 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:23.850957  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.837379  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:23.758114  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1252) took 1127 ms. Trace:
W20260709 13:04:23.720750  8986 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 13:04:23.865648  1742 rpcz_store.cc:276] 0709 13:04:22.630681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.654205 (+1023524us) service_pool.cc:224] Handling call
0709 13:04:23.756767 (+102562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.757257 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.757302 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:23.757735 (+   433us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:23.757810 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.757888 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:23.758042 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:23.796260  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:23.780836  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98233.6us
W20260709 13:04:23.871754  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:23.797585  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.873900  1663 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:23.723632  8870 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.877928  1708 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:23.684113  8987 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 13:04:23.884577  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.887348  1663 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:23.898573  8991 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.086s	user 0.000s	sys 0.003s
I20260709 13:04:23.902966  8739 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:23.903565  8739 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:23.904325  8018 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.754843  8649 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.905305  8018 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.905923  8649 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.908594  8018 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.875936  8635 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:23.909482  8635 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.909999  8847 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:23.911672  8635 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.911829  8901 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.915010  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3388) took 1679 ms. Trace:
I20260709 13:04:23.912812  8791 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:23.915957  8912 consensus_meta.cc:287] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.036s	user 0.016s	sys 0.002s
I20260709 13:04:23.684578  8953 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:23.915534  1738 rpcz_store.cc:276] 0709 13:04:22.236139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.107878 (+871739us) service_pool.cc:224] Handling call
0709 13:04:23.124047 (+ 16169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.129101 (+  5054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:23.914731 (+785630us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5400404550. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:23.914926 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29698816}
W20260709 13:04:23.922309  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49854.8us
I20260709 13:04:23.823854  8827 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.923911  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.930188  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.760298  8728 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.931070  1738 raft_consensus.cc:2410] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 18 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 13:04:23.801291  8736 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:23.891319  8910 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.933856  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:23.778785  8798 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.141s
I20260709 13:04:23.864559  8792 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.877013  8855 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.747445  8952 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.149s
I20260709 13:04:23.932657  8953 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.942414  8855 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.904489  8739 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.923686  8901 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:23.916905  8912 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.929718  8744 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:23.950184  8744 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.952751  8649 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.930985  1742 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:23.755357  8921 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.174s
I20260709 13:04:23.933463  8791 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:23.953465  8774 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.902s	user 0.010s	sys 0.006s
I20260709 13:04:23.879525  8979 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.953971  8739 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:23.940105  8792 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:23.964701  8731 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:23.940510  8910 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:23.962116  8953 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.990386  8731 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:24.002846  8731 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.003408  8979 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:23.987241  8744 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:23.924150  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:23.936715  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:24.139673  1737 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:23.961750  8791 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.091732  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.112242  1668 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:24.116710  8855 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.101042  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1167) took 1846 ms. Trace:
I20260709 13:04:24.125800  8912 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:24.097070  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.151878  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2647) took 2396 ms. Trace:
W20260709 13:04:24.154107  1736 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2636) took 2574 ms (client timeout 3298 ms). Trace:
W20260709 13:04:24.154718  1736 rpcz_store.cc:269] 0709 13:04:21.579777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.041938 (+462161us) service_pool.cc:224] Handling call
0709 13:04:24.153529 (+2111591us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b540039e9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:24.153864 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":718282112}
I20260709 13:04:24.159271  1739 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:24.007779  8844 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.150430  1738 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:24.153929  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.124898  8744 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:24.137017  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 213383us
I20260709 13:04:24.152487  1735 rpcz_store.cc:276] 0709 13:04:21.755487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.453339 (+697852us) service_pool.cc:224] Handling call
0709 13:04:22.577751 (+124412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.589409 (+ 11658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.151236 (+1561827us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b54000e1d50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:24.151584 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191808896}
I20260709 13:04:24.141938  8743 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:24.178797  8855 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:24.179855  8855 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.096920  8731 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:24.152904  8792 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.188383  8844 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.178871  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.199488  8953 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 13:04:24.145623  1587 rpcz_store.cc:276] 0709 13:04:22.253790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.375864 (+122074us) service_pool.cc:224] Handling call
0709 13:04:22.376328 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:22.513999 (+137671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.514052 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:22.518728 (+  4676us) log.cc:844] Serialized 26 byte log entry
0709 13:04:22.805553 (+286825us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:22.805616 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:22.808569 (+  2953us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:23.675806 (+867237us) spinlock_profiling.cc:243] Waited 52 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:24.098238 (+422432us) raft_consensus.cc:1685] finished
0709 13:04:24.098299 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.100336 (+  2037us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10341,"raft.queue_time_us":106988,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9021,"spinlock_wait_cycles":119619968,"thread_start_us":386054,"threads_started":2,"wal-append.queue_time_us":319572}
I20260709 13:04:24.210232  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:24.210697  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:24.217067  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.217141  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1255) took 1585 ms. Trace:
I20260709 13:04:24.219269  1735 rpcz_store.cc:276] 0709 13:04:22.631933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.211765 (+1579832us) service_pool.cc:224] Handling call
0709 13:04:24.212470 (+   705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.213235 (+   765us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:24.214202 (+   967us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:24.216660 (+  2458us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:24.216732 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.216815 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.217029 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.221567  8878 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.220817  8744 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.223253  8878 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.227378  8878 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:24.232909  1660 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:24.227914  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:24.205121  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:24.234076  1742 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:24.239776  2283 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:24.247211  8910 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:24.243633  8791 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:24.239796  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1175) took 1184 ms. Trace:
I20260709 13:04:24.247146  1556 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:24.252676  1660 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:24.268618  8892 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:04:24.269726  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 123907us
I20260709 13:04:24.270382  8892 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:24.270967  8892 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:24.271917  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:24.277669  1742 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:24.272811  8892 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.293853  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2667) took 1172 ms. Trace:
I20260709 13:04:24.299911  8979 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.300896  8892 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:24.299224  1742 rpcz_store.cc:276] 0709 13:04:23.114253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.275558 (+1161305us) service_pool.cc:224] Handling call
0709 13:04:24.276349 (+   791us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.285268 (+  8919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.286652 (+  1384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.313342  8983 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:24.314131  8983 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:24.314667  8983 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:24.301005  8979 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.332994  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.343066  1734 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:24.343642  8979 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:24.344295  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:24.344801  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 117220us
I20260709 13:04:24.347096  1708 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:24.348579  8749 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:24.302279  8910 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.358291  8910 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:24.361871  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.345106  1734 raft_consensus.cc:2376] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25. Re-sending same reply.
I20260709 13:04:24.337380  1735 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 13:04:24.368925  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:24.372560  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.366916  1737 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:24.255668  1593 rpcz_store.cc:276] 0709 13:04:23.054928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.454121 (+399193us) service_pool.cc:224] Handling call
0709 13:04:23.502929 (+ 48808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.503175 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:23.828204 (+325029us) log.cc:844] Serialized 14 byte log entry
0709 13:04:24.238933 (+410729us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:24.239068 (+   135us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:24.239145 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.239271 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.239546 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":482886,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":152,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":301,"thread_start_us":647895,"threads_started":2}
W20260709 13:04:24.377183  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:24.382130  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.374944  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1258) took 1737 ms. Trace:
W20260709 13:04:24.388424  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 174656us
I20260709 13:04:24.388849  1735 rpcz_store.cc:276] 0709 13:04:22.632885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.220630 (+1587745us) service_pool.cc:224] Handling call
0709 13:04:24.289961 (+ 69331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.367600 (+ 77639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.370364 (+  2764us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.364274  8744 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.326785  8791 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:24.379820  8844 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.405193  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.408519  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.508436  8749 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.465157  2316 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:24.466856  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:24.471951  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:24.519011  1734 raft_consensus.cc:2410] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:04:24.520831  2317 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 13:04:24.394791  8878 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:24.509495  8749 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.541684  8747 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.542268  8875 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.542827  8747 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.543777  8875 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.549774  1742 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:24.551752  1665 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:24.553977  8749 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:24.555209  8747 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:24.405457  8657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.406s	user 0.004s	sys 0.018s
W20260709 13:04:24.598629  8657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.636s	user 0.008s	sys 0.026s
I20260709 13:04:24.406380  8796 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.600399  8796 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:24.558233  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 318928us
I20260709 13:04:24.511313  8744 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.463562  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:24.463171  1660 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.565729  8798 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:24.566515  8927 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:24.567546  1735 raft_consensus.cc:2376] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21. Re-sending same reply.
I20260709 13:04:24.568679  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3464) took 2473 ms. Trace:
W20260709 13:04:24.571735  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2593) took 1843 ms (client timeout 1798 ms). Trace:
I20260709 13:04:24.572032  8774 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:24.577695  8476 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:24.578949  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2586) took 1858 ms (client timeout 1956 ms). Trace:
I20260709 13:04:23.558815   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:24.478245  8792 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.461349  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:24.617849  1665 rpcz_store.cc:269] 0709 13:04:22.721026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.721897 (+   871us) service_pool.cc:224] Handling call
0709 13:04:24.578541 (+1856644us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540013fbd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:24.578868 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3052705536}
I20260709 13:04:24.407948  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:24.590659  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 243812us
I20260709 13:04:24.601277  8863 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.442055  8979 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:24.618423  1588 rpcz_store.cc:276] 0709 13:04:22.095077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.095205 (+   128us) service_pool.cc:224] Handling call
0709 13:04:22.095510 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.221795 (+126285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:22.221827 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:22.221874 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:22.221917 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.564547 (+2342630us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002d0ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:24.564994 (+   447us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:24.568459 (+  3465us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":289790848}
I20260709 13:04:24.632251  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.632114  8798 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.634961  8912 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.636211  8912 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.606195  8747 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.603889  8910 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:24.634888  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:24.605572  8744 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.643749  8677 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 13:04:24.646601  8863 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.647176  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:24.617129  1664 rpcz_store.cc:269] 0709 13:04:22.728567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.728711 (+   144us) service_pool.cc:224] Handling call
0709 13:04:24.571208 (+1842497us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b540030b454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:24.571441 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":844220288}
I20260709 13:04:24.637424  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.664106  1734 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:24.630759  8749 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
W20260709 13:04:24.669691  9008 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.100s	user 0.000s	sys 0.004s
I20260709 13:04:24.671741  8747 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.673084  8912 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:24.675441  8744 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:24.683609  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45787.8us
I20260709 13:04:24.695161  8861 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:24.698699  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 51683.3us
I20260709 13:04:24.706548  9010 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:24.654376  8910 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:24.707124  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:24.708071  9010 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.708653  1584 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 16.
I20260709 13:04:24.708465  8910 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.711560  9010 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.711769  1734 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:24.711757  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62380us
I20260709 13:04:24.712921  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2669) took 1188 ms. Trace:
I20260709 13:04:24.663582  8927 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.713461  1734 rpcz_store.cc:276] 0709 13:04:23.524802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.710898 (+1186096us) service_pool.cc:224] Handling call
0709 13:04:24.711515 (+   617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.712592 (+  1077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.712844 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.629351  8476 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.701647  8792 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.727463  1735 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:24.738027  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:24.751467  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:24.717432  8878 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 13:04:24.757360  8747 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:24.778244  8863 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:24.758968  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.736394  8801 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:24.779045  1708 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:24.779500  8801 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:24.779981  1664 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 13: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:24.770831  2311 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:24.782845  8744 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:24.789954  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:24.794852  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.796136  1665 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:24.796780  8801 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:24.797540  1664 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:24.798465  8801 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:24.801971  1587 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 13:04:24.801783  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:24.731504  8976 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.680s	user 0.004s	sys 0.002s
I20260709 13:04:24.806864  1664 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:24.733767  8892 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:24.810727  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:24.814208  1736 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:24.815233  1736 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 17: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:24.816128  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.818935  1667 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:24.819015  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:24.820080  2318 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 12.
I20260709 13:04:24.811012  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:24.824388  8705 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:24.825397  8705 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.828118  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:24.727155  9010 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:24.830407  9010 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:24.838347  1735 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:24.839344  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3400) took 1096 ms. Trace:
I20260709 13:04:24.839728  1735 rpcz_store.cc:276] 0709 13:04:23.742958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.735014 (+992056us) service_pool.cc:224] Handling call
0709 13:04:24.837831 (+102817us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.838929 (+  1098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.839225 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.847046  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3439) took 2455 ms. Trace:
I20260709 13:04:24.847697  1585 rpcz_store.cc:276] 0709 13:04:22.391519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.134309 (+742790us) service_pool.cc:224] Handling call
0709 13:04:23.307261 (+172952us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:23.597342 (+290081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.846063 (+1248721us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54000fa354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:24.846819 (+   756us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":613956480}
I20260709 13:04:24.851064  1736 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 13:04:24.852415  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3401) took 1109 ms. Trace:
I20260709 13:04:24.852949  1736 rpcz_store.cc:276] 0709 13:04:23.743209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.816372 (+1073163us) service_pool.cc:224] Handling call
0709 13:04:24.817088 (+   716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.851768 (+ 34680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.852109 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:24.878772  1736 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:24.902334  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1259) took 1157 ms. Trace:
I20260709 13:04:24.922407  1736 rpcz_store.cc:276] 0709 13:04:23.744964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.876562 (+1131598us) service_pool.cc:224] Handling call
0709 13:04:24.877244 (+   682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.902078 (+ 24834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.902274 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:24.939546  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 199181us
W20260709 13:04:24.958112  8731 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.749s	user 0.023s	sys 0.014s
W20260709 13:04:24.981108  9001 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.124s	user 0.002s	sys 0.000s
I20260709 13:04:24.962641  8796 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.010200  8667 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:25.025854  9014 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.055s	user 0.000s	sys 0.002s
I20260709 13:04:25.005409  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:25.032078  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71174us
W20260709 13:04:25.033697  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 50746.1us
I20260709 13:04:25.034415  8972 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.038002  8927 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:25.043206  8739 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.054s	user 0.018s	sys 0.020s
I20260709 13:04:25.046147  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:25.050087  8635 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:25.051064  8635 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:25.050138  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.054342  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:25.059310  8705 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:25.060027  8983 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.745s	user 0.011s	sys 0.011s
I20260709 13:04:25.038003  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.064913  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3408) took 1319 ms. Trace:
I20260709 13:04:25.067282  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2587) took 2345 ms. Trace:
I20260709 13:04:25.054975  1667 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 13:04:25.033898  8744 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:25.074338  1734 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 13:04:25.077816  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72788us
I20260709 13:04:25.082154  1668 rpcz_store.cc:276] 0709 13:04:22.721367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.721980 (+   613us) service_pool.cc:224] Handling call
0709 13:04:22.722573 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.162103 (+1439530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.066746 (+904643us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b540013f6d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:25.067131 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3178395008}
I20260709 13:04:25.081432  1738 rpcz_store.cc:276] 0709 13:04:23.744739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.933364 (+188625us) service_pool.cc:224] Handling call
0709 13:04:25.063730 (+1130366us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.064096 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490166528}
I20260709 13:04:25.094579  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:25.096926  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:25.097908  9010 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:25.101959  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.098251  1586 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:25.100926  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 212924us
I20260709 13:04:25.052598  8635 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.088243  8517 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 13:04:25.099815  1589 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:25.107019  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:25.076016  8749 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:25.107324  1737 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.595s	user 0.013s	sys 0.007s
I20260709 13:04:25.110750  1737 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
W20260709 13:04:25.113775  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 76397.3us
I20260709 13:04:25.131191  2311 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:25.134536  2313 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:25.136830  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3398) took 1369 ms. Trace:
I20260709 13:04:25.131240  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2603) took 1376 ms. Trace:
I20260709 13:04:25.137281  1737 rpcz_store.cc:276] 0709 13:04:23.742337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.173735 (+431398us) service_pool.cc:224] Handling call
0709 13:04:25.111744 (+938009us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:25.141908  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:25.106357  8705 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.143318  1736 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 13:04:25.144877  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.143430  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.091239  8983 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.167189  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.169147  8731 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:25.169904  8635 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:25.170213  1590 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:25.174489  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.174926  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:25.176192  2316 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:25.178148  1737 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:25.179380  8476 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.140822  1666 rpcz_store.cc:276] 0709 13:04:23.733156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.733329 (+   173us) service_pool.cc:224] Handling call
0709 13:04:23.734232 (+   903us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:23.735145 (+   913us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:23.735176 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:23.736124 (+   948us) log.cc:844] Serialized 52 byte log entry
0709 13:04:24.256272 (+520148us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:24.393981 (+137709us) log.cc:844] Serialized 14 byte log entry
0709 13:04:24.566772 (+172791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:24.566946 (+   174us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:24.856488 (+289542us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b180042dde0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:25.101009 (+244521us) raft_consensus.cc:1685] finished
0709 13:04:25.105436 (+  4427us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:25.109838 (+  4402us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":86613,"prepare.queue_time_us":30858,"prepare.run_cpu_time_us":156,"prepare.run_wall_time_us":26428,"raft.queue_time_us":100034,"raft.run_cpu_time_us":695,"raft.run_wall_time_us":26792,"spinlock_wait_cycles":60001152,"thread_start_us":379254,"threads_started":1,"wal-append.queue_time_us":401434}
I20260709 13:04:25.160151  1664 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:25.185359  8705 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.191419  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.204756  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 686) took 2916 ms. Trace:
I20260709 13:04:25.207261  8953 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:25.236559  2312 rpcz_store.cc:276] 0709 13:04:22.288543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.303612 (+ 15069us) service_pool.cc:224] Handling call
0709 13:04:22.391967 (+ 88355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:22.392636 (+   669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.204246 (+2811610us) spinlock_profiling.cc:243] Waited 87.8 ms on lock 0x7b54005c0150. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.204528 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":201978624}
I20260709 13:04:25.229403  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.249147  2311 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
I20260709 13:04:25.249750  8844 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.251649  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:25.252512  1587 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:25.253495  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:25.153244  8749 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.223452  8731 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:25.258500  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 276698us
I20260709 13:04:25.230103  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:25.269801  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.273315  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.244099  8517 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:25.265777  8731 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:25.303054  1738 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 13:04:25.303373  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.250737  8844 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.278175  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:25.278443  1665 raft_consensus.cc:2376] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25. Re-sending same reply.
I20260709 13:04:25.280083  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.281028  8863 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:04:25.281575  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 119415us
I20260709 13:04:25.281872  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.285866  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.292290  8791 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:25.295629  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2670) took 1583 ms. Trace:
W20260709 13:04:25.296901  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.305480  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 169680us
W20260709 13:04:25.308218  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 239907us
I20260709 13:04:25.278460  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:25.307390  1734 rpcz_store.cc:276] 0709 13:04:23.712271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.792435 (+1080164us) service_pool.cc:224] Handling call
0709 13:04:25.070658 (+278223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.089842 (+ 19184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.295321 (+205479us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b5400403650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.295532 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":638235392}
W20260709 13:04:25.304441  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.305480  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1262) took 1559 ms. Trace:
I20260709 13:04:25.315898  1738 rpcz_store.cc:276] 0709 13:04:23.745678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.235352 (+1489674us) service_pool.cc:224] Handling call
0709 13:04:25.301706 (+ 66354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.303579 (+  1873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.305406 (+  1827us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:25.322013  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:25.330955  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 220541us
I20260709 13:04:25.302819  8517 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:25.266057  8749 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.319993  1668 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:25.344302  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.327826  1741 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:25.308884  8791 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:25.349911  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:25.356051  1734 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:25.361332  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.365078  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:25.343431  8517 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:25.353744  8861 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.658s	user 0.018s	sys 0.002s
I20260709 13:04:25.365658  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:25.332515  8677 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.415678  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.416611  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.399158  9014 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 13:04:25.436897  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:25.514923  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:25.437038  1734 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:25.522521  8705 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.515436  1585 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:25.523888  8798 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:25.516294  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 85242.2us
I20260709 13:04:25.526347  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.441766  8849 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:25.528537  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:25.455120  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3375) took 3368 ms (client timeout 1783 ms). Trace:
I20260709 13:04:25.437338  8969 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.529322  1733 rpcz_store.cc:269] 0709 13:04:22.086477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:22.185044 (+ 98567us) service_pool.cc:224] Handling call
0709 13:04:25.454576 (+3269532us) spinlock_profiling.cc:243] Waited 317 ms on lock 0x7b54000e2c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.454854 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":729146624}
I20260709 13:04:25.530668  8969 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.532258  1588 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 19 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 13:04:25.534011  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.449679  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:25.467348  8844 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.535451  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.478569  2318 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:25.536202  1588 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:25.479475  8791 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.536784  9016 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:25.537565  8733 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:25.538399  8910 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.486521  8863 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:25.487051  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2673) took 1218 ms (client timeout 1571 ms). Trace:
I20260709 13:04:25.489748  8677 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.491052  1738 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:25.516984  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:25.492645  9026 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 13:04:25.493090  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1261) took 1747 ms. Trace:
W20260709 13:04:25.495141  8801 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.696s	user 0.018s	sys 0.006s
W20260709 13:04:25.496536  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 148287us
I20260709 13:04:25.496824  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.497341  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.573961  1664 raft_consensus.cc:2376] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21. Re-sending same reply.
I20260709 13:04:25.489184  1592 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:25.505852  8891 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:25.507623  9030 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.508054  8792 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.509158  8824 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:25.509914  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.510080  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.511147  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.511269  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.509354  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:25.513111  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:25.516224  8765 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:25.526702  8667 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.003s
I20260709 13:04:25.527787  8849 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.535349  1740 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:25.511277  8983 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:25.485657  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 69374.9us
I20260709 13:04:25.518801  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:25.521814  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3382) took 3432 ms. Trace:
I20260709 13:04:25.537086  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:25.607234  1588 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:25.467067  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:25.539503  1742 rpcz_store.cc:269] 0709 13:04:24.269020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.310499 (+ 41479us) service_pool.cc:224] Handling call
0709 13:04:25.486771 (+1176272us) spinlock_profiling.cc:243] Waited 38.7 ms on lock 0x7b54000e3654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.486964 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":89098624}
I20260709 13:04:25.477742  8749 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:25.436520  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.553220  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 184952us
I20260709 13:04:25.607025  1739 rpcz_store.cc:276] 0709 13:04:22.088832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.042263 (+953431us) service_pool.cc:224] Handling call
0709 13:04:24.158295 (+1116032us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:24.160135 (+  1840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.521358 (+1361223us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540039e9d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.521591 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2564650880}
I20260709 13:04:25.573124  1736 rpcz_store.cc:276] 0709 13:04:23.745404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.939133 (+1193729us) service_pool.cc:224] Handling call
0709 13:04:25.111581 (+172448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:25.154043 (+ 42462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.492711 (+338668us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54000df2d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:25.493001 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320103680}
I20260709 13:04:25.533231  8969 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:25.554392  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.537487  9016 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.552625  1587 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:25.585971  1737 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:25.591154  1741 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:25.593037  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 96251.5us
I20260709 13:04:25.606544  1734 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
I20260709 13:04:25.529287  1593 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:25.624351  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3404) took 1872 ms. Trace:
I20260709 13:04:25.624882  1741 rpcz_store.cc:276] 0709 13:04:23.743861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.223859 (+1479998us) service_pool.cc:224] Handling call
0709 13:04:25.616252 (+392393us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:25.601990  8855 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.631744  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:25.663517  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:25.668990  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.631107  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:25.670173  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45897.5us
I20260709 13:04:25.662204  1589 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
W20260709 13:04:25.672435  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 132) took 21475 ms (client timeout 15000 ms). Trace:
W20260709 13:04:25.672960  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 73274.8us
I20260709 13:04:25.677305  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.680459  8849 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.681677  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:25.683029  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57085.8us
I20260709 13:04:25.687222  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.688122  1662 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:25.688776  1741 raft_consensus.cc:1217] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Deduplicated request from leader. Original: 14.8->[16.9-20.11]   Dedup: 16.9->[18.10-20.11]
I20260709 13:04:25.591024  8798 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.038s
I20260709 13:04:25.690451  8951 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:25.691619  9029 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.216s	user 0.005s	sys 0.000s
I20260709 13:04:25.693089  1667 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:25.681881  2310 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:25.696961  1584 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:25.697862  8746 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.672899  1486 rpcz_store.cc:269] 0709 13:04:04.197154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:04.211561 (+ 14407us) service_pool.cc:224] Handling call
0709 13:04:25.672097 (+21460536us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:14.799378 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:14.800353 (+   975us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:14.800425 (+    72us) write_op.cc:435] Acquired schema lock
0709 13:04:14.800457 (+    32us) tablet.cc:662] Decoding operations
0709 13:04:14.956483 (+156026us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:14.956614 (+   131us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:14.956649 (+    35us) tablet.cc:685] Acquiring locks for 15 operations
0709 13:04:14.958168 (+  1519us) tablet.cc:701] Row locks acquired
0709 13:04:14.958226 (+    58us) write_op.cc:260] PREPARE: finished
0709 13:04:14.958485 (+   259us) write_op.cc:270] Start()
0709 13:04:14.958666 (+   181us) write_op.cc:276] Timestamp: P: 1783602254958375 usec, L: 0
0709 13:04:14.958691 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:04:14.959661 (+   970us) log.cc:844] Serialized 6209 byte log entry
0709 13:04:15.349371 (+389710us) op_driver.cc:464] REPLICATION: finished
0709 13:04:15.460497 (+111126us) write_op.cc:301] APPLY: starting
0709 13:04:15.467181 (+  6684us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:16.121228 (+654047us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:16.121395 (+   167us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:16.288494 (+167099us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b18004b3700. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedd1c0ec 00007fcdedd1eaf7 00007fcdedd1ffb0 00007fcdf1a341bb 00007fcdf1a7b5d4 00007fcdf19278da 00007fcdf192a3cb 00007fcdf1929fd3 00007fcdf1a28e34
0709 13:04:17.645984 (+1357490us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:17.697417 (+ 51433us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=30,mrs_lookups=0
0709 13:04:17.697490 (+    73us) write_op.cc:312] APPLY: finished
0709 13:04:17.698007 (+   517us) log.cc:844] Serialized 139 byte log entry
0709 13:04:17.930097 (+232090us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:17.942051 (+ 11954us) write_op.cc:454] Released schema lock
0709 13:04:17.969265 (+ 27214us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:18.019784 (+ 50519us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:18.030190 (+ 10406us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5787996,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":67597,"apply.run_cpu_time_us":67269,"apply.run_wall_time_us":2547333,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66857,"num_ops":15,"prepare.queue_time_us":675175,"prepare.run_cpu_time_us":33317,"prepare.run_wall_time_us":310074,"raft.queue_time_us":156602,"raft.run_cpu_time_us":23415,"raft.run_wall_time_us":104200,"replication_time_us":375346,"spinlock_wait_cycles":29739264,"thread_start_us":879204,"threads_started":5,"wal-append.queue_time_us":337410,"wal-append.run_cpu_time_us":12767,"wal-append.run_wall_time_us":2389227}]]}
I20260709 13:04:25.698400  8746 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.699306  1592 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 13:04:25.699747  1592 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:04:25.700157  1592 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 16.12
W20260709 13:04:25.700618  8966 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 2.760s	user 0.004s	sys 0.011s
I20260709 13:04:25.701666  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:25.691448  8951 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:25.702693  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 71090.2us
W20260709 13:04:25.703392  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.703902  8951 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.704057  8677 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:25.701787  8849 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.690532  8954 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.622020  8792 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:25.700474  1659 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:25.700876  1741 pending_rounds.cc:85] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
I20260709 13:04:25.706566  1666 raft_consensus.cc:2393] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:25.706641  1741 pending_rounds.cc:107] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 13:04:25.700819  1592 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:25.707240  1741 raft_consensus.cc:2889] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:25.699568  1584 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 14. (term mismatch)
W20260709 13:04:25.704313  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:25.685212  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:25.710228  8792 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.713811  8792 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:25.714799  8749 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.716033  8749 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.717777  8954 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.718282  8901 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.624423  8824 consensus_queue.cc:1243] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:25.669533  9030 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.747071  1659 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:25.708784  1584 pending_rounds.cc:85] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 13
I20260709 13:04:25.747840  1584 pending_rounds.cc:107] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 17.14
I20260709 13:04:25.746169  1588 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:25.661723  8801 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 13:04:25.623358  9016 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:25.748171  1584 raft_consensus.cc:2889] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:25.704854  8849 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.752254  1585 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 13.
W20260709 13:04:25.746023  8635 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.508s	user 0.014s	sys 0.000s
W20260709 13:04:25.752758  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 386398us
W20260709 13:04:25.756912  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71758.5us
I20260709 13:04:25.758073  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.742974  8765 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:25.761238  1588 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:25.624709  8863 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
W20260709 13:04:25.760519  8979 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.091s	user 0.016s	sys 0.007s
I20260709 13:04:25.760257  8677 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 13:04:25.760272  8731 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.746762  8855 consensus_queue.cc:1059] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 13:04:25.760491  8667 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:25.834357  1592 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:25.836035  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.767104  8951 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:25.750954  8901 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:25.780267  9038 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:25.851390  1592 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:25.777716  8844 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 13:04:25.775368  8824 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
W20260709 13:04:25.852646  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 230182us
W20260709 13:04:25.861629  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 69005.2us
W20260709 13:04:25.869943  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 106410us
I20260709 13:04:25.794864  8979 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:04:25.879241  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 268360us
I20260709 13:04:25.870625  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.883728  1665 raft_consensus.cc:2368] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20.
I20260709 13:04:25.871133  1592 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:25.890280  1659 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:25.893922  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.882750  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 94325.7us
I20260709 13:04:25.857880  8951 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:25.901304  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:25.871644  8844 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:25.877862  8635 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:25.907768  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:25.891639  9039 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:25.915261  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:25.919255  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 208390us
W20260709 13:04:25.909322  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 215333us
I20260709 13:04:25.942481  8913 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:25.944130  8749 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:25.947937  9030 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:25.948227  8517 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.000s
I20260709 13:04:25.903764  8951 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:25.952924  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3406) took 2208 ms. Trace:
I20260709 13:04:25.907756  8844 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:25.934623  8635 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:25.956804  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.878075  9038 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:25.994896  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48700.8us
W20260709 13:04:25.975895  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88981.9us
I20260709 13:04:25.964537  1735 rpcz_store.cc:276] 0709 13:04:23.744297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.840823 (+1096526us) service_pool.cc:224] Handling call
0709 13:04:24.841426 (+   603us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:24.938672 (+ 97246us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:25.238499 (+299827us) log.cc:844] Serialized 14 byte log entry
0709 13:04:25.585285 (+346786us) log.cc:844] Serialized 14 byte log entry
0709 13:04:25.585879 (+   594us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:25.605923 (+ 20044us) log.cc:844] Serialized 52 byte log entry
0709 13:04:25.606111 (+   188us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:25.606146 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.606226 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:25.952597 (+346371us) raft_consensus.cc:1685] finished
0709 13:04:25.952624 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:25.952836 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25061,"prepare.queue_time_us":370346,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":128,"raft.queue_time_us":51555,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":28434,"spinlock_wait_cycles":8228224,"thread_start_us":510363,"threads_started":2,"wal-append.queue_time_us":512559}
I20260709 13:04:25.996042  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1263) took 2043 ms. Trace:
I20260709 13:04:25.909479  8863 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 13:04:26.004776  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:26.017645  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65814.5us
I20260709 13:04:26.004886  1733 rpcz_store.cc:276] 0709 13:04:23.930747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.531149 (+1600402us) service_pool.cc:224] Handling call
0709 13:04:25.532082 (+   933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.532534 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:25.533025 (+   491us) log.cc:844] Serialized 14 byte log entry
0709 13:04:25.967801 (+434776us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:25.967904 (+   103us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:25.967990 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:25.968097 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:25.974359 (+  6262us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":269,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"thread_start_us":406924,"threads_started":2,"wal-append.queue_time_us":330276}
W20260709 13:04:26.008324  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46067.3us
I20260709 13:04:25.961198  8913 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:25.969848  8749 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:26.024340  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 132979us
I20260709 13:04:26.032756  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:26.035367  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:25.996037  8635 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:26.025148  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:25.973696  8517 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:26.050195  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98822.3us
I20260709 13:04:26.050276  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.079599  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:26.085843  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.050072  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:26.099478  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54484.1us
I20260709 13:04:26.102516  1659 raft_consensus.cc:2410] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 13:04:26.046387  8983 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:26.045312  8969 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:26.101295  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:25.977943  9039 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:26.048784  8849 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:26.108285  1733 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.103379  1735 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:26.114936  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:26.064415  8863 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:26.134060  2312 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:26.022743  9021 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.085s
I20260709 13:04:26.139391  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:26.090871  9043 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:26.153043  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 129251us
I20260709 13:04:26.155835  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:26.157302  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 135488us
I20260709 13:04:26.111716  8954 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:26.119725  8983 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:26.107332  8798 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:26.121268  8969 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:26.162457  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:26.186102  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.187309  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.188190  1735 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:26.090868  8749 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:26.199829  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:26.091367  9044 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.104s
I20260709 13:04:26.198630  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.168076  8849 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election won for term 25
W20260709 13:04:26.231657  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 123488us
W20260709 13:04:26.242865  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 224041us
I20260709 13:04:26.189142  8969 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:26.188838  8798 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:26.251932  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43989.3us
I20260709 13:04:26.187533  8983 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:26.251075  1556 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:26.196666  8954 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 13:04:26.286716  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.204872  9060 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:26.198165  8731 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:26.308774  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:26.167806  9053 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 13:04:26.280800  8798 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:26.331504  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.173740  9043 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.262897  8468 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:26.339900  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 225042us
W20260709 13:04:26.349087  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.277063  8954 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.301117  8849 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:26.315582  9049 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.388752  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.401911  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:26.406682  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 63198.2us
I20260709 13:04:26.342339  8468 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.367262  8954 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:26.414780  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 214753us
W20260709 13:04:26.418331  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 117420us
I20260709 13:04:26.231477  8913 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.417331  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.425591  1666 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
W20260709 13:04:26.431288  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 141630us
I20260709 13:04:26.404053  8731 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:26.395355  9046 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.380417  8863 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 13:04:26.419319  8468 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.379967  9021 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 13:04:26.449390  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.326699  9060 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.471026  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:26.476563  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 139556us
I20260709 13:04:26.429096  8901 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:26.493419  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 44321.7us
I20260709 13:04:26.447412  9038 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.467520  8913 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.519147  1660 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:26.639071  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 196631us
I20260709 13:04:26.456951  9030 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.458777  9054 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.539446  9076 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:26.563138  9074 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.111s
I20260709 13:04:26.564386  9062 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:26.567307  9027 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.142s	user 0.002s	sys 0.000s
W20260709 13:04:26.504856  8891 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.903s	user 0.015s	sys 0.003s
I20260709 13:04:26.452857  9046 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.501883  8855 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 13:04:26.567389  9057 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
I20260709 13:04:26.582324  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.584134  8791 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.711066  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.712381  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.712968  1666 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
W20260709 13:04:26.713097  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52062.7us
I20260709 13:04:26.713737  8733 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.714351  8733 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.715380  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.586022  8849 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:26.594357  9042 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.337s	user 0.001s	sys 0.001s
W20260709 13:04:26.523262  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.722048  8855 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:26.722589  8855 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:26.726109  8824 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:26.727352  8824 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.727896  9059 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:26.727872  8824 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.730837  9092 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:26.731504  9092 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:26.732834  1741 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.013s	sys 0.022s
W20260709 13:04:26.651597  1593 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.005s	sys 0.006s
W20260709 13:04:26.740677  1593 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.005s	sys 0.006s
W20260709 13:04:26.655447  1585 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.000s	sys 0.015s
I20260709 13:04:26.437323  9049 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.518062  9039 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:26.762634  1733 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.626s	user 0.014s	sys 0.002s
I20260709 13:04:26.762650  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.763499  1733 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:26.764787  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3422) took 1017 ms. Trace:
I20260709 13:04:26.756505  1661 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:26.681425  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:26.652103  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 240096us
I20260709 13:04:26.699111  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:26.572397  9061 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.157s	user 0.000s	sys 0.003s
I20260709 13:04:26.736680  9092 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.676412  8731 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:26.733930  1741 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.013s	sys 0.022s
W20260709 13:04:26.742895  1585 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.000s	sys 0.016s
W20260709 13:04:26.756032  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48257.6us
I20260709 13:04:26.742204  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3484) took 1719 ms. Trace:
W20260709 13:04:26.758937  1588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.008s	sys 0.018s
I20260709 13:04:26.658196  9053 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:26.774399  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 93297us
I20260709 13:04:26.793104  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:26.765497  1733 rpcz_store.cc:276] 0709 13:04:25.746958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.055617 (+308659us) service_pool.cc:224] Handling call
0709 13:04:26.764688 (+709071us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:26.689440  8878 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:26.778616  1742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.011s	sys 0.020s
W20260709 13:04:26.802228  1742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.011s	sys 0.020s
I20260709 13:04:26.701287  8749 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:26.806280  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.807108  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:26.809381  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.811013  1634 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:26.728961  9059 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:26.788728  1588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.008s	sys 0.019s
I20260709 13:04:26.788348  1593 rpcz_store.cc:276] 0709 13:04:25.022785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.685598 (+662813us) service_pool.cc:224] Handling call
0709 13:04:25.686089 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:25.686578 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.686617 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:25.687247 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 13:04:25.975543 (+288296us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:26.022549 (+ 47006us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.741269 (+718720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.741377 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:26.741445 (+    68us) raft_consensus.cc:1685] finished
0709 13:04:26.741461 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:26.742128 (+   667us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21583,"prepare.queue_time_us":154052,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":103570,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":17742,"spinlock_wait_cycles":17917696,"thread_start_us":982834,"threads_started":2,"wal-append.queue_time_us":469907}
I20260709 13:04:26.658313  9030 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.785472  9038 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.818742  2315 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:26.819059  1660 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:26.819865  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:26.820982  1733 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:26.791082  9043 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.676277  9054 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:26.824744  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:26.696946  9062 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.757366  8744 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 13:04:26.829962  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2605) took 3092 ms. Trace:
I20260709 13:04:26.830729  1660 rpcz_store.cc:276] 0709 13:04:23.737744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:23.738697 (+   953us) service_pool.cc:224] Handling call
0709 13:04:23.782128 (+ 43431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:24.232575 (+450447us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54002e7320. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfdf1c0 00007fcdedfe0231 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c
0709 13:04:25.531825 (+1299250us) spinlock_profiling.cc:243] Waited 448 ms on lock 0x7b58007ae7d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:04:26.379439 (+847614us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54002e7320. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52a1d 00007fcdedf391a4 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:04:26.819514 (+440075us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.829803 (+ 10289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2238566272}
I20260709 13:04:26.832216  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:26.832906  1593 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:04:26.833333  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51791us
I20260709 13:04:26.833951  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:26.834527  1733 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:26.809463  8878 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:26.853770  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:26.848912  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 160685us
I20260709 13:04:26.851967  9093 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:26.809029  9053 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.853647  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:26.831525  9052 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:26.886376  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.871446  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:26.838398  8936 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:26.854261  8824 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:26.831908  8733 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.838431  9067 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:26.889153  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:26.890760  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.838748  9076 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 13:04:26.847597  9032 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:26.886606  9078 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:04:26.854439  9060 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.888154  8936 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.849143  9090 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:26.861675  9059 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:26.893127  8824 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
W20260709 13:04:26.896481  8952 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
I20260709 13:04:26.886376  9092 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:26.837055  9038 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.908840  1733 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 16.
W20260709 13:04:26.911391  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:26.901135  9033 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.024s	sys 0.000s
W20260709 13:04:26.913015  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 59535.9us
I20260709 13:04:26.934242  8875 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:26.939313  9084 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 13:04:26.939908  9053 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.938812  9054 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.848074  8731 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:26.925120  9088 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:04:27.010416  9046 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.934744  8824 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:26.853633  9073 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 13:04:26.938400  8878 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.893750  9052 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:27.027436  8798 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.645s	user 0.008s	sys 0.004s
I20260709 13:04:27.028254  8798 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:26.947208  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:26.948792  8705 raft_consensus.cc:3055] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:26.939723  9038 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:26.911554  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3504) took 1875 ms. Trace:
I20260709 13:04:26.975541  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3409) took 1993 ms. Trace:
W20260709 13:04:26.977322  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81009.5us
I20260709 13:04:26.906724  9076 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:26.906340  9032 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:26.994346  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:26.997639  9050 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Stepping down as leader of term 12
I20260709 13:04:26.998922  9066 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:27.034960  1737 rpcz_store.cc:276] 0709 13:04:24.982683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.661159 (+678476us) service_pool.cc:224] Handling call
0709 13:04:25.661781 (+   622us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:25.880951 (+219170us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.880986 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:25.904468 (+ 23482us) log.cc:844] Serialized 52 byte log entry
0709 13:04:26.055798 (+151330us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:26.102329 (+ 46531us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.124368 (+ 22039us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.124559 (+   191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.124645 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:26.975246 (+850601us) raft_consensus.cc:1685] finished
0709 13:04:26.975274 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:26.975465 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":98315,"prepare.queue_time_us":93627,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":7214,"raft.queue_time_us":458710,"raft.run_cpu_time_us":1588,"raft.run_wall_time_us":70161,"thread_start_us":276591,"threads_started":2,"wal-append.queue_time_us":355779}
W20260709 13:04:27.013397  8969 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.698s	user 0.006s	sys 0.010s
I20260709 13:04:26.999284  8936 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.000265  9049 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:27.000739  8844 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.992s	user 0.015s	sys 0.000s
I20260709 13:04:27.002368  8667 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:27.031649  9052 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.009243  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:26.952191  9016 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.177s	user 0.007s	sys 0.015s
I20260709 13:04:27.093645  9016 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.094981  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.096592  9016 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.096657  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.098481  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:26.979872  9033 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.025s	sys 0.000s
I20260709 13:04:27.033795  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.101354  1587 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 13:04:27.040341  1586 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
W20260709 13:04:27.027478  8792 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.313s	user 0.009s	sys 0.012s
I20260709 13:04:27.103598  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.042424  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.103596  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.105988  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.106667  1587 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 17.
W20260709 13:04:27.001323  8635 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.946s	user 0.008s	sys 0.009s
I20260709 13:04:27.107561  8635 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:27.107661  8983 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.782s	user 0.010s	sys 0.006s
I20260709 13:04:27.108922  8983 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.110112  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.111663  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.112520  1589 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.113428  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.114486  1660 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:27.105043  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63772.6us
I20260709 13:04:27.045920  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.049366  9066 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.044402  8969 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.058394  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:27.060375  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 74942.5us
I20260709 13:04:27.050510  8844 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.043747  9030 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:27.059245  9062 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:27.068153  9032 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:27.049902  9076 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.070591  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:27.070626  8855 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.074546  8936 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:27.038621  8705 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:27.076577  1740 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:04:26.972972  8952 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 13:04:27.078318  8667 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:27.079555  8468 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.006s
I20260709 13:04:27.051697  8798 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.039482  9050 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:27.083304  2315 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: [12.11-12.11]
W20260709 13:04:27.084198  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 196816us
I20260709 13:04:27.084427  9070 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:27.034850  1584 rpcz_store.cc:276] 0709 13:04:25.036495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.784980 (+748485us) service_pool.cc:224] Handling call
0709 13:04:25.787265 (+  2285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.850469 (+ 63204us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:26.034499 (+184030us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.817260 (+782761us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.911047 (+ 93787us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:26.911136 (+    89us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:26.911217 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.911339 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:26.911486 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":359979,"prepare.queue_time_us":152423,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":2612,"raft.queue_time_us":341192,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":143,"thread_start_us":639715,"threads_started":3,"wal-append.queue_time_us":767540}
W20260709 13:04:27.132941  8954 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.001s
I20260709 13:04:27.089219  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3505) took 2052 ms. Trace:
I20260709 13:04:27.085084  9090 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:27.015249  8824 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:27.147003  8824 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:27.103772  8792 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.116329  1733 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.148054  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2683) took 1064 ms. Trace:
I20260709 13:04:27.143779  1588 rpcz_store.cc:276] 0709 13:04:25.038210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.795368 (+757158us) service_pool.cc:224] Handling call
0709 13:04:25.834801 (+ 39433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.851704 (+ 16903us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:26.136869 (+285165us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.011909 (+875040us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.088673 (+ 76764us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.088841 (+   168us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:27.088920 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.089002 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.089155 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":148659,"prepare.queue_time_us":191619,"prepare.run_cpu_time_us":141,"prepare.run_wall_time_us":5084,"raft.queue_time_us":346649,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":216,"spinlock_wait_cycles":21170048,"thread_start_us":976882,"threads_started":3,"wal-append.queue_time_us":743602}
I20260709 13:04:27.148425  1733 rpcz_store.cc:276] 0709 13:04:26.083312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.013813 (+930501us) service_pool.cc:224] Handling call
0709 13:04:27.147968 (+134155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.107630  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.149519  1587 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.045414  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3318) took 1189 ms. Trace:
I20260709 13:04:27.119561  1737 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:04:27.120384  1634 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.122220  1593 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:27.128365  1664 raft_consensus.cc:2393] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.152460  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2685) took 1045 ms. Trace:
I20260709 13:04:27.131126  8844 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.152894  1737 rpcz_store.cc:276] 0709 13:04:26.106859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.057861 (+951002us) service_pool.cc:224] Handling call
0709 13:04:27.152385 (+ 94524us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.131567  8891 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:27.119355  8951 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.103s	user 0.013s	sys 0.003s
I20260709 13:04:27.088887  1734 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 13:04:27.146277  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.159961  1589 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:27.127877  9098 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:27.146668  9050 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.153671  9102 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.155507  8792 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.156127  9100 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:27.132350  8635 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.150738  1662 rpcz_store.cc:276] 0709 13:04:25.856132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.946723 (+1090591us) service_pool.cc:224] Handling call
0709 13:04:27.045352 (+ 98629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.162061  9098 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:27.153709  9072 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:27.138226  8468 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.162447  8705 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.164387  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.197011  8705 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:27.167146  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.168995  9066 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:27.170254  9090 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.203953  8983 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.205725  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.147583  9099 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:27.207376  8844 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:27.160738  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:27.208256  8844 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:27.207994  9099 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.209859  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.211159  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:27.211427  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2613) took 2415 ms (client timeout 2924 ms). Trace:
W20260709 13:04:27.212100  1661 rpcz_store.cc:269] 0709 13:04:24.796362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:24.801172 (+  4810us) service_pool.cc:224] Handling call
0709 13:04:27.211134 (+2409962us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:27.211300 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4491330304}
I20260709 13:04:27.212710  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:27.157323  8954 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.137387  8936 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:27.178290  8891 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.218966  8936 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.220031  8891 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.223439  9066 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.224463  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.177692  8798 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:27.225245  8798 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:27.186681  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.225737  8798 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:27.166301  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.226897  1593 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.228140  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.197230  8792 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:27.178088  8951 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:27.231287  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.232393  8951 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.175273  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3415) took 2070 ms. Trace:
W20260709 13:04:27.232654  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71996.3us
I20260709 13:04:27.233009  1734 rpcz_store.cc:276] 0709 13:04:25.105271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.087983 (+1982712us) service_pool.cc:224] Handling call
0709 13:04:27.088523 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:27.174797 (+ 86274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.175133 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.207458  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:27.234553  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.235765  1734 raft_consensus.cc:2376] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18. Re-sending same reply.
I20260709 13:04:27.236413  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3322) took 1332 ms. Trace:
I20260709 13:04:27.236510  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.237555  8910 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:27.237938  1659 raft_consensus.cc:2393] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.237703  9070 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:27.226007  1733 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:27.172605  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.240316  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:27.215319  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.180183  9062 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.064s
W20260709 13:04:27.243067  9051 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.102s	user 0.002s	sys 0.000s
I20260709 13:04:27.243826  1739 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
I20260709 13:04:27.244421  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.245072  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.245850  1739 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:27.229930  8792 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:27.247080  9050 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 12 FOLLOWER]: Advancing to term 13
I20260709 13:04:27.247982  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3416) took 2099 ms. Trace:
I20260709 13:04:27.247499  8792 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.224807  9066 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.220089  8954 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.245885  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1264) took 3314 ms. Trace:
I20260709 13:04:27.250152  1584 raft_consensus.cc:2368] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20.
I20260709 13:04:27.188920  9102 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.251451  8955 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:27.251930  8955 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:27.250284  1738 rpcz_store.cc:276] 0709 13:04:23.931035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.573635 (+1642600us) service_pool.cc:224] Handling call
0709 13:04:25.613003 (+ 39368us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:25.630155 (+ 17152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.630212 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:25.717923 (+ 87711us) log.cc:844] Serialized 52 byte log entry
0709 13:04:25.875916 (+157993us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:25.904619 (+ 28703us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.043010 (+138391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.043251 (+   241us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:27.242223 (+1198972us) spinlock_profiling.cc:243] Waited 6.76 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:27.245537 (+  3314us) raft_consensus.cc:1685] finished
0709 13:04:27.245659 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.245815 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107278,"prepare.queue_time_us":28494,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":219338,"raft.run_cpu_time_us":412,"raft.run_wall_time_us":54814,"spinlock_wait_cycles":15556864,"thread_start_us":252391,"threads_started":2,"wal-append.queue_time_us":272173}
I20260709 13:04:27.252527  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.224917  8517 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.188710  9098 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:27.232719  8765 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:27.255300  1662 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23.
I20260709 13:04:27.256330  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.257529  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1256) took 1748 ms. Trace:
I20260709 13:04:27.257925  9096 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:27.258545  9096 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:27.258719  8635 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:27.259020  9096 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:27.259203  8635 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:27.259701  8635 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:27.263808  9062 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.264215  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.265233  8791 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.264605  9062 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.265815  8791 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.216472  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:27.268103  8791 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:27.268983  8791 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:27.270612  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.271384  1591 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:27.272637  9103 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:27.236991  1660 rpcz_store.cc:276] 0709 13:04:25.904351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.890439 (+986088us) service_pool.cc:224] Handling call
0709 13:04:27.236151 (+345712us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540013c9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:27.236317 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510980864}
I20260709 13:04:27.278020  9103 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:27.285147  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1267) took 2081 ms. Trace:
I20260709 13:04:27.285631  8969 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.268095  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.261485  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1268) took 1591 ms. Trace:
I20260709 13:04:27.290364  1735 rpcz_store.cc:276] 0709 13:04:25.670259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.260269 (+1590010us) service_pool.cc:224] Handling call
0709 13:04:27.260731 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.261057 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.261091 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.261149 (+    58us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:27.261202 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.261271 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.261393 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.292156  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:27.292667  1735 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 13:04:27.162554  8667 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.216017  8844 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.294890  9066 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.295598  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:27.296891  1667 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.297197  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.258054  2312 rpcz_store.cc:276] 0709 13:04:25.509106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.509277 (+   171us) service_pool.cc:224] Handling call
0709 13:04:25.509646 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:26.134643 (+624997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.257144 (+1122501us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b54004e61d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:27.257395 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1386408320}
I20260709 13:04:27.298170  1584 raft_consensus.cc:2410] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 18 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 13:04:27.299154  9030 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:27.300482  9059 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:27.253480  8955 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.301087  9059 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:27.190524  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.285874  1738 rpcz_store.cc:276] 0709 13:04:25.204088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.253245 (+2049157us) service_pool.cc:224] Handling call
0709 13:04:27.284537 (+ 31292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.284789 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.284816 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.284863 (+    47us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:27.284895 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.284954 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.285075 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.299294  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.301230  1634 leader_election.cc:400] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20.
I20260709 13:04:27.298097  9102 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.366129  1634 leader_election.cc:403] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:27.237408  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:27.366833  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 65805.2us
I20260709 13:04:27.293326  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1269) took 1622 ms. Trace:
W20260709 13:04:27.367004  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 102875us
I20260709 13:04:27.367437  1735 rpcz_store.cc:276] 0709 13:04:25.704713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.291882 (+1587169us) service_pool.cc:224] Handling call
0709 13:04:27.292213 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.293054 (+   841us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.293271 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:27.297298  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3319) took 1427 ms (client timeout 1739 ms). Trace:
I20260709 13:04:27.248379  1739 rpcz_store.cc:276] 0709 13:04:25.153001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.246969 (+2093968us) service_pool.cc:224] Handling call
0709 13:04:27.247353 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.247593 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.247623 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.247687 (+    64us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:27.247723 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.247786 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.247900 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:27.224102  9095 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.288s	user 0.004s	sys 0.000s
W20260709 13:04:27.370607  1667 rpcz_store.cc:269] 0709 13:04:25.870045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.250307 (+1380262us) service_pool.cc:224] Handling call
0709 13:04:27.297233 (+ 46926us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.373175  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:27.374099  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 64967.2us
I20260709 13:04:27.374718  1735 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 19: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:27.342483  1591 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:27.369750  1737 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.370469  1733 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
W20260709 13:04:27.369236  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.368005  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:27.367347  1734 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.377054  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.379465  1733 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:27.382408  1634 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.383283  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.377653  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.377794  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.373760  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.349670  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3499) took 2315 ms. Trace:
I20260709 13:04:27.384987  1734 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:27.385552  1738 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:27.342010  9093 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:27.386478  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.384886  1735 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:27.387806  1739 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:27.313150  9062 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.313627  9103 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.389947  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.390668  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:27.393566  1737 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:27.391551  1667 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:27.394390  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:27.378449  9102 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:27.385977  1585 rpcz_store.cc:276] 0709 13:04:25.034582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.753997 (+719415us) service_pool.cc:224] Handling call
0709 13:04:25.754349 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:25.787311 (+ 32962us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:25.995367 (+208056us) log.cc:844] Serialized 14 byte log entry
0709 13:04:26.832756 (+837389us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.080474 (+247718us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b180047fd40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdedfd5060 00007fcdedfe7716 00007fcdedff8287 00007fcdedff8208
0709 13:04:27.080990 (+   516us) log.cc:844] Serialized 14 byte log entry
0709 13:04:27.303965 (+222975us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.304026 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:27.304100 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.304171 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.349560 (+ 45389us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":298684,"prepare.queue_time_us":179762,"prepare.run_cpu_time_us":160,"prepare.run_wall_time_us":165,"raft.queue_time_us":328883,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":10113,"spinlock_wait_cycles":6303104,"thread_start_us":1089966,"threads_started":3,"wal-append.queue_time_us":676384}
W20260709 13:04:27.396423  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2625) took 2080 ms (client timeout 1658 ms). Trace:
W20260709 13:04:27.398954  1665 rpcz_store.cc:269] 0709 13:04:25.315991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.324353 (+  8362us) service_pool.cc:224] Handling call
0709 13:04:27.396193 (+2071840us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b540013f1d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:27.396344 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1237753216}
W20260709 13:04:27.400936  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 200761us
I20260709 13:04:27.404703  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2629) took 1368 ms. Trace:
I20260709 13:04:27.405817  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:27.387514  8667 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:27.387151  8955 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:27.413870  1665 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 13:04:27.414485  1665 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 13
I20260709 13:04:27.414763  1665 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 13:04:27.415055  1665 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:27.405819  1667 rpcz_store.cc:276] 0709 13:04:26.031265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.373936 (+1342671us) service_pool.cc:224] Handling call
0709 13:04:27.391255 (+ 17319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.398626 (+  7371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.398970 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.410246  1589 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:27.417577  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:27.417615  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:27.422050  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 140014us
W20260709 13:04:27.422597  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54654.2us
I20260709 13:04:27.423525  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.427320  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1229) took 1380 ms. Trace:
I20260709 13:04:27.409168  8849 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:27.429184  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43794us
I20260709 13:04:27.429759  1665 rpcz_store.cc:276] 0709 13:04:26.047000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.399985 (+1352985us) service_pool.cc:224] Handling call
0709 13:04:27.409241 (+  9256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.426429 (+ 17188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.427187 (+   758us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:27.403904  8733 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:27.431983  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.404649  9032 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 13:04:27.399861  9103 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.442512  9103 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.443151  9070 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 13:04:27.444550  8824 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.421216  8667 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:27.446431  8667 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.447085  8824 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.434211  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:27.437671  9093 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.432916  8733 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:27.459587  8733 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:27.463613  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1231) took 1320 ms. Trace:
I20260709 13:04:27.465059  8746 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.433470  9062 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:27.397033  9059 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.464269  1665 rpcz_store.cc:276] 0709 13:04:26.143000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.435115 (+1292115us) service_pool.cc:224] Handling call
0709 13:04:27.462783 (+ 27668us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.463106 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.463148 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:27.463213 (+    65us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:27.463264 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.463345 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:27.463508 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:27.479583  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.479283  1665 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:27.480974  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1265) took 3549 ms. Trace:
I20260709 13:04:27.481001  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.485674  1740 rpcz_store.cc:276] 0709 13:04:23.931897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.624038 (+1692141us) service_pool.cc:224] Handling call
0709 13:04:25.697621 (+ 73583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.134430 (+1436809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.480467 (+346037us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:27.480821 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3168836864}
I20260709 13:04:27.488354  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.488916  9062 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:27.467458  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2628) took 1437 ms. Trace:
I20260709 13:04:27.489859  1659 rpcz_store.cc:276] 0709 13:04:26.030254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.157096 (+126842us) service_pool.cc:224] Handling call
0709 13:04:27.467175 (+1310079us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400117454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:27.467372 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2413118208}
I20260709 13:04:27.491668  1659 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 13:04:27.468132  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:27.464304  8635 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.466576  9096 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.494503  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.489097  1740 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.454047  1739 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.518983  9066 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.520169  8746 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:27.519925  9032 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:27.481006  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1232) took 1283 ms. Trace:
I20260709 13:04:27.522222  1735 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:27.519690  8677 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:27.521019  8969 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:27.684311  8969 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:27.546478  8798 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:27.606490  9096 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.612044  9102 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20.
I20260709 13:04:27.615303  9098 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.066294  9098 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:27.555995  9066 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.621212  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:27.638690  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:28.067610  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 428964us
W20260709 13:04:27.519397  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 85274.1us
I20260709 13:04:27.668260  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:28.070012  1740 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:28.071208  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.669878  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:28.073635  9096 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:28.073992  1739 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:28.072708  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:27.682677  8677 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.685804  8969 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:27.686378  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:27.737279  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:28.078152  1663 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:27.745532  1592 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:27.766175  8468 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.794982  1733 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.807443  1738 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:27.835942  2280 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23.
I20260709 13:04:27.857208  1667 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:27.869706  8936 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:27.872211  8635 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:27.878404  9076 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:27.879691  9093 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:27.963812  9087 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 1.030s	user 0.004s	sys 0.000s
I20260709 13:04:27.976015  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:27.967788  9099 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:27.991549  8792 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.030s	sys 0.034s
W20260709 13:04:28.040483  1584 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.011s	sys 0.018s
I20260709 13:04:28.076408  1739 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:27.676719  1665 rpcz_store.cc:276] 0709 13:04:26.197361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.478671 (+1281310us) service_pool.cc:224] Handling call
0709 13:04:27.479085 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.480652 (+  1567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.480940 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.382076  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3440) took 1086 ms. Trace:
I20260709 13:04:28.382804  1733 rpcz_store.cc:276] 0709 13:04:27.295679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.376728 (+ 81049us) service_pool.cc:224] Handling call
0709 13:04:28.381941 (+1005213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.082706  1735 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 23 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 13:04:28.386023  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3437) took 1091 ms. Trace:
I20260709 13:04:28.386618  1738 rpcz_store.cc:276] 0709 13:04:27.294687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.373088 (+ 78401us) service_pool.cc:224] Handling call
0709 13:04:28.385945 (+1012857us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.388391  2280 leader_election.cc:403] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:28.389407  1662 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:28.390512  1584 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.011s	sys 0.022s
I20260709 13:04:28.433967  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3423) took 2686 ms. Trace:
I20260709 13:04:28.434598  1739 rpcz_store.cc:276] 0709 13:04:25.747326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.075536 (+2328210us) service_pool.cc:224] Handling call
0709 13:04:28.075981 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.432816 (+356835us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.433835 (+  1019us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.460803  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 624948us
I20260709 13:04:28.464135  1735 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:28.465406  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2687) took 2357 ms. Trace:
I20260709 13:04:28.465953  1735 rpcz_store.cc:276] 0709 13:04:26.118282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.385381 (+2267099us) service_pool.cc:224] Handling call
0709 13:04:28.463862 (+ 78481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.464938 (+  1076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.465242 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.078696  1663 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3320) took 2198 ms (client timeout 2238 ms). Trace:
I20260709 13:04:28.302233  1592 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:28.515995  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3502) took 3480 ms. Trace:
I20260709 13:04:28.260095  1740 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:28.517643  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3444) took 1091 ms. Trace:
I20260709 13:04:28.518150  1740 rpcz_store.cc:276] 0709 13:04:27.426634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.667792 (+241158us) service_pool.cc:224] Handling call
0709 13:04:28.517552 (+849760us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.519816  1740 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:28.520676  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1273) took 2055 ms. Trace:
I20260709 13:04:28.521128  1740 rpcz_store.cc:276] 0709 13:04:26.465447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.519147 (+2053700us) service_pool.cc:224] Handling call
0709 13:04:28.519569 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.520369 (+   800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.520610 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.188764  8824 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.534791  8953 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:04:28.534224  8824 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.543031  8824 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.545145  8824 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:28.556408  1522 debug-util.cc:398] Leaking SignalData structure 0x7b080027a0e0 after lost signal to thread 434
I20260709 13:04:28.598650  9093 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.128798  9102 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:28.663208  9102 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.516472  1592 rpcz_store.cc:276] 0709 13:04:25.035666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.784634 (+748968us) service_pool.cc:224] Handling call
0709 13:04:25.834158 (+ 49524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.744713 (+1910555us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b540047a2a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52a1d 00007fcdedf391a4 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:04:28.515581 (+770868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.515917 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":539558784}
I20260709 13:04:28.664680  9093 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.604426  8936 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.074188  9096 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.130609  9052 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:28.459661  9076 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.692848  9076 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:28.153143  8733 consensus_meta.cc:287] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.693s	user 0.014s	sys 0.000s
I20260709 13:04:28.693928  8733 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.696504  8733 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:28.077009  8746 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.699189  8746 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.184757  8801 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:28.700389  8801 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.703033  8801 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:28.514743  1663 rpcz_store.cc:269] 0709 13:04:25.881011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.736940 (+1855929us) service_pool.cc:224] Handling call
0709 13:04:28.078616 (+341676us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.650509  1668 raft_consensus.cc:2368] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 17. Current term is 18.
I20260709 13:04:28.651795  8798 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.079264  8677 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:28.069275  9066 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.658187  8635 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:28.658708  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 585905us
I20260709 13:04:28.788686  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:28.551388  1739 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 13:04:28.665535  9102 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.679412  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1270) took 2259 ms. Trace:
I20260709 13:04:28.810810  1738 rpcz_store.cc:276] 0709 13:04:26.419940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.387569 (+1967629us) service_pool.cc:224] Handling call
0709 13:04:28.387988 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.678785 (+290797us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.678833 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.678909 (+    76us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:28.679020 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.679142 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.679327 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.680805  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:28.682480  9096 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:28.665809  8824 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:28.689823  9052 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:28.616895  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3424) took 2862 ms. Trace:
W20260709 13:04:28.801424  8667 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.309s	user 0.026s	sys 0.042s
I20260709 13:04:28.757194  8936 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:28.757608  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3410) took 3757 ms. Trace:
W20260709 13:04:28.786162  1591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.009s	sys 0.010s
I20260709 13:04:28.794498  8677 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:28.795449  8635 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.808483  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1271) took 2388 ms. Trace:
I20260709 13:04:28.853900  1739 rpcz_store.cc:276] 0709 13:04:26.420627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.438739 (+2018112us) service_pool.cc:224] Handling call
0709 13:04:28.549547 (+110808us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.808017 (+258470us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.808390 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.802028  8798 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:28.813084  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:28.830305  1733 rpcz_store.cc:276] 0709 13:04:25.755898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.461919 (+2706021us) service_pool.cc:224] Handling call
0709 13:04:28.462512 (+   593us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.616231 (+153719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.616292 (+    61us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.616382 (+    90us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:28.616454 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.616573 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.616797 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.391664  8792 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.035s	sys 0.035s
I20260709 13:04:28.832904  1663 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20260709 13:04:28.851882  1591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.009s	sys 0.010s
I20260709 13:04:28.861006  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1190) took 2844 ms. Trace:
W20260709 13:04:28.862082  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60603.5us
I20260709 13:04:28.862594  8677 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.863780  8824 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:28.864928  9052 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 16.15, Last appended by leader: 15, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:28.865720  1739 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:28.866278  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2633) took 2363 ms. Trace:
I20260709 13:04:28.866660  1663 rpcz_store.cc:276] 0709 13:04:26.503487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.792476 (+2288989us) service_pool.cc:224] Handling call
0709 13:04:28.829105 (+ 36629us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:28.865951 (+ 36846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.866214 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.867734  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:28.844251  1741 rpcz_store.cc:276] 0709 13:04:25.000396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.661892 (+661496us) service_pool.cc:224] Handling call
0709 13:04:25.662500 (+   608us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:25.871706 (+209206us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:04:25.871741 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:25.872377 (+   636us) log.cc:844] Serialized 52 byte log entry
0709 13:04:26.785593 (+913216us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:26.785650 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:26.785755 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:27.962588 (+1176833us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:28.757184 (+794596us) raft_consensus.cc:1685] finished
0709 13:04:28.757219 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.757522 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32559,"raft.queue_time_us":255916,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":12972,"spinlock_wait_cycles":88030464,"thread_start_us":955748,"threads_started":2,"wal-append.queue_time_us":954845}
I20260709 13:04:28.863662  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:28.878628  1662 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
W20260709 13:04:28.879796  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2640) took 1721 ms (client timeout 2042 ms). Trace:
W20260709 13:04:28.880545  1662 rpcz_store.cc:269] 0709 13:04:27.158476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.256085 (+ 97609us) service_pool.cc:224] Handling call
0709 13:04:27.975792 (+719707us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:28.879696 (+903904us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86540544}
I20260709 13:04:28.883466  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:28.884367  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1189) took 2868 ms. Trace:
W20260709 13:04:28.838043  8667 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.026s	sys 0.045s
I20260709 13:04:28.884900  1584 rpcz_store.cc:276] 0709 13:04:26.016119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.422078 (+1405959us) service_pool.cc:224] Handling call
0709 13:04:27.432238 (+ 10160us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.432485 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:27.432779 (+   294us) log.cc:844] Serialized 14 byte log entry
0709 13:04:28.883008 (+1450229us) log.cc:844] Serialized 14 byte log entry
0709 13:04:28.883655 (+   647us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.883788 (+   133us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:28.883919 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.884148 (+   229us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.884296 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3919,"prepare.queue_time_us":459,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":40624,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":77916,"thread_start_us":1449140,"threads_started":2}
I20260709 13:04:28.902570  1662 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:28.903554  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2635) took 2399 ms. Trace:
I20260709 13:04:28.903927  1662 rpcz_store.cc:276] 0709 13:04:26.505023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.902090 (+2397067us) service_pool.cc:224] Handling call
0709 13:04:28.902394 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.903189 (+   795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.903478 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.861595  1591 rpcz_store.cc:276] 0709 13:04:26.016465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.982816 (+1966351us) service_pool.cc:224] Handling call
0709 13:04:27.983268 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.983528 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:28.015300 (+ 31772us) log.cc:844] Serialized 14 byte log entry
0709 13:04:28.860407 (+845107us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.860507 (+   100us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:28.860600 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.860708 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.860917 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4510,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":875353,"threads_started":2}
I20260709 13:04:28.902570  8824 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:28.912108  9102 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:28.918460  9102 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:28.918879  9102 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:28.919459  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1235) took 2279 ms. Trace:
I20260709 13:04:28.919996  1663 rpcz_store.cc:276] 0709 13:04:26.639359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.918145 (+2278786us) service_pool.cc:224] Handling call
0709 13:04:28.918506 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.918707 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.918731 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.918769 (+    38us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:28.918808 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.918855 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.918987 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.864928  8677 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:28.913700  1733 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:28.930644  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3434) took 1769 ms. Trace:
I20260709 13:04:28.931038  1733 rpcz_store.cc:276] 0709 13:04:27.160861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.859536 (+1698675us) service_pool.cc:224] Handling call
0709 13:04:28.902141 (+ 42605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.929427 (+ 27286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.930396 (+   969us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:28.941411  1739 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:28.868788  8635 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:28.870137  8969 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.025s	sys 0.041s
I20260709 13:04:28.943533  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3432) took 1808 ms. Trace:
I20260709 13:04:28.949375  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1193) took 2482 ms. Trace:
I20260709 13:04:28.908841  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:28.982810  1584 rpcz_store.cc:276] 0709 13:04:26.467506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.946215 (+2478709us) service_pool.cc:224] Handling call
0709 13:04:28.947642 (+  1427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.948125 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.948188 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:28.948313 (+   125us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:28.948404 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.948656 (+   252us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:28.949079 (+   423us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.984498  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75794.2us
I20260709 13:04:28.985080  1584 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 12. Current term is 13. Ops: []
I20260709 13:04:28.985765  1739 rpcz_store.cc:276] 0709 13:04:27.135416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.865016 (+1729600us) service_pool.cc:224] Handling call
0709 13:04:28.865554 (+   538us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.942878 (+ 77324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.943444 (+   566us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:28.957396  1634 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.729s (SENT) [suppressed 15 similar messages]
I20260709 13:04:29.011114  8677 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:29.011818  8677 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:28.906342  8798 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:28.915515  8824 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:29.012777  8677 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.036448  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3454) took 2312 ms. Trace:
W20260709 13:04:28.987805  8969 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.026s	sys 0.042s
I20260709 13:04:29.047693  8798 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.053560  8824 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:29.056605  8953 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.002s
I20260709 13:04:29.064828  1584 rpcz_store.cc:276] 0709 13:04:26.677087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.984380 (+2307293us) service_pool.cc:224] Handling call
0709 13:04:28.984800 (+   420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.988227 (+  3427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.988680 (+   453us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.076499  1584 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:29.103320  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3461) took 2051 ms. Trace:
I20260709 13:04:29.104676  1584 rpcz_store.cc:276] 0709 13:04:27.033067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.075497 (+2042430us) service_pool.cc:224] Handling call
0709 13:04:29.076196 (+   699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.084407 (+  8211us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.084733 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.125124  1584 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:29.125938  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3510) took 2039 ms. Trace:
I20260709 13:04:29.126255  1584 rpcz_store.cc:276] 0709 13:04:27.086372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.121819 (+2035447us) service_pool.cc:224] Handling call
0709 13:04:29.124938 (+  3119us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.125659 (+   721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.125880 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:29.123589  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 308408us
I20260709 13:04:29.101560  8953 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23.
I20260709 13:04:29.133313  1663 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:29.129071  9127 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:29.148125  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 190749us
I20260709 13:04:29.161856  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1191) took 2811 ms. Trace:
I20260709 13:04:29.162400  1592 rpcz_store.cc:276] 0709 13:04:26.350843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.665021 (+2314178us) service_pool.cc:224] Handling call
0709 13:04:28.665628 (+   607us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:28.665940 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:28.665977 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:28.666701 (+   724us) log.cc:844] Serialized 26 byte log entry
0709 13:04:28.961894 (+295193us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:28.961963 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:28.962088 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.161585 (+199497us) raft_consensus.cc:1685] finished
0709 13:04:29.161611 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.161783 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":294738,"threads_started":1,"wal-append.queue_time_us":469889}
I20260709 13:04:29.164615  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3517) took 2076 ms. Trace:
I20260709 13:04:29.164793  8635 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.165004  1592 rpcz_store.cc:276] 0709 13:04:27.088186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.163654 (+2075468us) service_pool.cc:224] Handling call
0709 13:04:29.163994 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.164231 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.164262 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.164308 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:29.164362 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.164432 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.164559 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.204696  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3519) took 2116 ms. Trace:
I20260709 13:04:29.145711  9127 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.205610  8635 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.207337  9127 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.146821  8953 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 13:04:29.209708  8792 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.205082  1592 rpcz_store.cc:276] 0709 13:04:27.088834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.203765 (+2114931us) service_pool.cc:224] Handling call
0709 13:04:29.204154 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.204351 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.204379 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.204424 (+    45us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:29.204470 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.204522 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.204636 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.210455  8792 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.181090  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1230) took 3095 ms. Trace:
I20260709 13:04:29.181331  8824 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.216033  8824 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.213768  1667 rpcz_store.cc:276] 0709 13:04:26.087723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.416198 (+1328475us) service_pool.cc:224] Handling call
0709 13:04:27.427050 (+ 10852us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.434884 (+1007834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.180730 (+745846us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54002f1a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:29.181001 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":988369280}
I20260709 13:04:29.217845  8792 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.220902  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:29.222597  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1278) took 1201 ms. Trace:
I20260709 13:04:29.231559  9038 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.232374  9038 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.190430  9128 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:29.193681  9096 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.195027  8855 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.195624  9102 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.196951  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.150153  8969 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:29.238903  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110280us
I20260709 13:04:29.252161  1632 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:29.252943  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.253278  8824 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:29.254187  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:29.254370  8824 raft_consensus.cc:697] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:29.254510  9123 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:29.255458  8731 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:29.256003  8731 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:29.256352  9127 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:29.255873  8824 consensus_queue.cc:237] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.257506  9127 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:29.257175  8731 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.259569  9127 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 13:04:29.275509  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54668.5us
I20260709 13:04:29.240479  8792 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:29.254024  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.265733  8635 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:04:29.286239  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79780.4us
I20260709 13:04:29.223047  1733 rpcz_store.cc:276] 0709 13:04:28.021010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.932123 (+911113us) service_pool.cc:224] Handling call
0709 13:04:28.932556 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.932817 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:29.122850 (+190033us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.222127 (+ 99277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.222185 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:29.222272 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.222352 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.222496 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33330,"prepare.queue_time_us":209758,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":72,"raft.queue_time_us":11550,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":288556,"threads_started":2,"wal-append.queue_time_us":99151}
W20260709 13:04:29.286252  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45363.9us
I20260709 13:04:29.301445  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1280) took 1240 ms. Trace:
I20260709 13:04:29.302063  1733 rpcz_store.cc:276] 0709 13:04:28.060794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.299832 (+1239038us) service_pool.cc:224] Handling call
0709 13:04:29.300349 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.300951 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.300994 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.301060 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:29.301102 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.301185 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.301362 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:29.305136  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.8676e+06us
I20260709 13:04:29.305179  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:29.310052  9136 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:29.311094  9136 consensus_queue.cc:1243] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:29.313122  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3515) took 2225 ms. Trace:
I20260709 13:04:29.313611  1584 rpcz_store.cc:276] 0709 13:04:27.087704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.126992 (+2039288us) service_pool.cc:224] Handling call
0709 13:04:29.127351 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.137542 (+ 10191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.137593 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.138778 (+  1185us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.312696 (+173918us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:29.312747 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.312835 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.312890 (+    55us) raft_consensus.cc:1685] finished
0709 13:04:29.312907 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.313041 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6104,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":2822,"thread_start_us":163939,"threads_started":1,"wal-append.queue_time_us":164858}
I20260709 13:04:29.315636  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.317004  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2636) took 2756 ms. Trace:
I20260709 13:04:29.275522  9128 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:29.322507  1584 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 25: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:29.322633  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.323441  1733 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:29.323675  1591 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 13:04:29.324115  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1281) took 1262 ms. Trace:
I20260709 13:04:29.324476  1733 rpcz_store.cc:276] 0709 13:04:28.061433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.303337 (+1241904us) service_pool.cc:224] Handling call
0709 13:04:29.322186 (+ 18849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.323852 (+  1666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.324056 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.240290  8969 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.329547  9152 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:29.331004  9151 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:29.331820  1708 connection.cc:256] RPC call timeout handler was delayed by -1.072547037999982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260709 13:04:29.331676  9151 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:29.332692  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0158470379999471s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1693s for a response
W20260709 13:04:29.333097  1708 connection.cc:256] RPC call timeout handler was delayed by -1.0068470379999894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1703s for a response
I20260709 13:04:29.335457  9151 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.277011  9123 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
W20260709 13:04:29.337868  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.84882e+06us
I20260709 13:04:29.337632  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.342566  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3463) took 2308 ms. Trace:
I20260709 13:04:29.344206  1705 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.343173  1587 rpcz_store.cc:276] 0709 13:04:27.033783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.099745 (+2065962us) service_pool.cc:224] Handling call
0709 13:04:29.100359 (+   614us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.121777 (+ 21418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.121836 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.142504 (+ 20668us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.272607 (+130103us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:29.273582 (+   975us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.274257 (+   675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.274394 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.342310 (+ 67916us) raft_consensus.cc:1685] finished
0709 13:04:29.342331 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.342492 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1547,"prepare.queue_time_us":251,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":44,"raft.queue_time_us":17439,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":1820,"thread_start_us":123872,"threads_started":1,"wal-append.queue_time_us":126933}
I20260709 13:04:29.321941  9128 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:29.342375  1733 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:29.325451  8792 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.352169  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:29.354349  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98528.3us
I20260709 13:04:29.355093  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.356088  1591 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:29.351431  9049 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 13:04:29.343546  9151 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:29.360313  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.367439  9148 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.368202  9148 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:29.368742  9137 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 13:04:29.369580  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3428) took 2245 ms. Trace:
I20260709 13:04:29.369524  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.370707  9148 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.372433  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.361872  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:29.374050  9151 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:29.375344  8969 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.376420  9049 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:29.372675  9038 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.378582  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.379122  9038 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:29.379248  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.377372  9049 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:29.370153  1740 rpcz_store.cc:276] 0709 13:04:27.123836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.522215 (+1398379us) service_pool.cc:224] Handling call
0709 13:04:28.762612 (+240397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.238895 (+476283us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:29.238929 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.266905 (+ 27976us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.327822 (+ 60917us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:29.327944 (+   122us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.328102 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.369285 (+ 41183us) raft_consensus.cc:1685] finished
0709 13:04:29.369312 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.369475 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":423298,"raft.queue_time_us":501896,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":9143,"thread_start_us":50833,"threads_started":1,"wal-append.queue_time_us":67244}
I20260709 13:04:29.379276  2310 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:29.386215  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:29.389235  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.403981  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.408278  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.409206  2313 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:29.415625  1733 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 26.
I20260709 13:04:29.417439  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:29.418792  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.418342  9128 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.420302  9128 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:29.422106  8824 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:04:29.422766  8824 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:29.423112  8517 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:29.423738  8517 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.423811  8824 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.382722  8953 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:29.425650  9038 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:29.425911  8953 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:29.384481  9089 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.444s	user 0.010s	sys 0.006s
I20260709 13:04:29.437077  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1237) took 2739 ms. Trace:
I20260709 13:04:29.429996  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.381071  9049 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:29.408699  9151 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:29.432586  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.446062  2313 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:29.446259  1740 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:29.434989  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.435112  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.429157  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.444844  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:29.428278  9138 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 13:04:29.356788  8792 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.486352  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3436) took 2293 ms. Trace:
I20260709 13:04:29.486129  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.486967  1741 rpcz_store.cc:276] 0709 13:04:27.193224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.869463 (+1676239us) service_pool.cc:224] Handling call
0709 13:04:28.916848 (+ 47385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.917104 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:28.955163 (+ 38059us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.325200 (+370037us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.450724 (+125524us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.450812 (+    88us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:29.450897 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.450992 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.486279 (+ 35287us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":320000,"prepare.queue_time_us":212490,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":90,"raft.queue_time_us":388564,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":4072,"thread_start_us":213876,"threads_started":4,"wal-append.queue_time_us":191823}
I20260709 13:04:29.448693  9049 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:29.445978  9151 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:29.417274  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.494071  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.562203  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3523) took 2316 ms. Trace:
I20260709 13:04:29.496470  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.567579  1592 rpcz_store.cc:276] 0709 13:04:27.245920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.212407 (+1966487us) service_pool.cc:224] Handling call
0709 13:04:29.213354 (+   947us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.216753 (+  3399us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:29.220056 (+  3303us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.316648 (+ 96592us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.482951 (+166303us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.483499 (+   548us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.483727 (+   228us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:29.483780 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.483884 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.561950 (+ 78066us) raft_consensus.cc:1685] finished
0709 13:04:29.561972 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.562126 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8706,"prepare.queue_time_us":2681,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":2693,"raft.queue_time_us":135620,"raft.run_cpu_time_us":445,"raft.run_wall_time_us":51260,"thread_start_us":252508,"threads_started":2,"wal-append.queue_time_us":156202}
I20260709 13:04:29.569674  1733 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.571273  2314 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:29.559357  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.445706  9147 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:29.449734  9038 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:29.525110  1705 connection.cc:256] RPC call timeout handler was delayed by -3.0523001980000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2238s for a response
I20260709 13:04:29.574190  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:29.562145  9086 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.624s	user 0.014s	sys 0.010s
I20260709 13:04:29.576622  1740 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:29.609397  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:29.609732  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1272) took 3188 ms. Trace:
W20260709 13:04:29.609920  1705 connection.cc:256] RPC call timeout handler was delayed by -1.9363001980000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.34780000000000005s for a response
W20260709 13:04:29.558876  9156 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 13:04:29.493734  8765 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.586365  8517 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:29.615816  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.586910  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.594973  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.600092  1733 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:29.617219  1592 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:29.617884  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.601784  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.602094  8847 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:29.560391  8792 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.490396  9090 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 13:04:29.558291  9102 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:29.610371  1705 connection.cc:256] RPC call timeout handler was delayed by -1.798600198000031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3631s for a response
I20260709 13:04:29.612139  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.631825  2318 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:29.632371  1741 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:29.610289  1735 rpcz_store.cc:276] 0709 13:04:26.420983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.515058 (+2094075us) service_pool.cc:224] Handling call
0709 13:04:28.515731 (+   673us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.327708 (+811977us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:29.349585 (+ 21877us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.482339 (+132754us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.518366 (+ 36027us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.518610 (+   244us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:29.583249 (+ 64639us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.609187 (+ 25938us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.609363 (+   176us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.609425 (+    62us) raft_consensus.cc:1685] finished
0709 13:04:29.609447 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.609644 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":691482,"prepare.queue_time_us":87650,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":644888,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":47388,"thread_start_us":160674,"threads_started":2,"wal-append.queue_time_us":232336}
I20260709 13:04:29.491650  9096 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:29.621636  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1240) took 2588 ms (client timeout 2209 ms). Trace:
W20260709 13:04:29.635807  1661 rpcz_store.cc:269] 0709 13:04:27.032997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.267511 (+234514us) service_pool.cc:224] Handling call
0709 13:04:29.601491 (+2333980us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:29.621518 (+ 20027us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49606528}
I20260709 13:04:29.618887  1632 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:29.614132  8635 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:29.623044  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.637531  1740 raft_consensus.cc:2376] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21. Re-sending same reply.
I20260709 13:04:29.611402  9152 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:29.638444  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 282284us
I20260709 13:04:29.639524  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.639695  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.572647  9151 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:29.633335  1587 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:29.620533  9030 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.616055  9148 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.620718  8791 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:29.621155  8847 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:29.648411  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.649590  1587 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:29.638346  8792 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.657070  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.639215  8969 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:29.608685  9038 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.661861  1741 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 13:04:29.620965  8765 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.664168  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.657076  8847 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:29.644771  8635 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:29.665977  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.663492  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.685567  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.686597  2318 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:29.686630  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.687378  2310 raft_consensus.cc:2376] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23. Re-sending same reply.
I20260709 13:04:29.688982  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.647771  9148 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.696107  9148 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:29.674556  8635 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:29.678056  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.681749  9158 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.667168  8517 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.015s
I20260709 13:04:29.655207  8791 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.664422  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.685648  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.647612  9030 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.711261  9158 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.689951  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.691013  9164 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:29.662853  1557 connection.cc:256] RPC call timeout handler was delayed by -3.5835313810000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 13:04:29.721779  1557 connection.cc:256] RPC call timeout handler was delayed by -3.5835313810000002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
W20260709 13:04:29.722182  1557 connection.cc:256] RPC call timeout handler was delayed by -3.4503313810000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1718s for a response
I20260709 13:04:29.669142  8969 raft_consensus.cc:697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:29.722488  1557 connection.cc:256] RPC call timeout handler was delayed by -3.2242351649999819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21000000000000002s for a response
W20260709 13:04:29.722822  1557 connection.cc:256] RPC call timeout handler was delayed by -2.9778313809999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
I20260709 13:04:29.713047  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:29.723129  1557 connection.cc:256] RPC call timeout handler was delayed by -2.9778313809999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260709 13:04:29.723580  1557 connection.cc:256] RPC call timeout handler was delayed by -2.6968351649999818s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2686s for a response
W20260709 13:04:29.724009  1557 connection.cc:256] RPC call timeout handler was delayed by -2.6700313810000011s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2585s for a response
I20260709 13:04:29.724316  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.725240  1585 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:04:29.691708  1592 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:29.725847  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:29.724316  1557 connection.cc:256] RPC call timeout handler was delayed by -2.3649313810000194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2924s for a response
W20260709 13:04:29.726819  1557 connection.cc:256] RPC call timeout handler was delayed by -2.0292313809999882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32970000000000005s for a response
W20260709 13:04:29.727260  1557 connection.cc:256] RPC call timeout handler was delayed by -1.2066313810000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.42110000000000003s for a response
I20260709 13:04:29.721436  9158 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:29.721787  9164 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.729661  9102 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:29.730368  9102 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:29.727756  1557 connection.cc:256] RPC call timeout handler was delayed by -1.2066313810000224s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.42110000000000003s for a response
I20260709 13:04:29.732321  9102 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.701720  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.735309  1741 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:29.737066  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.734289  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:27.492301  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1234) took 1154 ms. Trace:
I20260709 13:04:29.739183  1659 rpcz_store.cc:276] 0709 13:04:26.337567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.490870 (+1153303us) service_pool.cc:224] Handling call
0709 13:04:27.491270 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:27.492002 (+   732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.492216 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.739755  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.740465  1660 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:29.740968  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1238) took 2980 ms. Trace:
I20260709 13:04:29.741386  1659 rpcz_store.cc:276] 0709 13:04:26.760775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.740150 (+2979375us) service_pool.cc:224] Handling call
0709 13:04:29.740457 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.740653 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.740694 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.740734 (+    40us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:29.740766 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.740817 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.740910 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.705993  1587 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
W20260709 13:04:29.742799  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.738521  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.732414  9164 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:29.708393  1734 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 2.309s	user 0.017s	sys 0.010s
I20260709 13:04:29.744421  9096 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:29.764537  9096 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:29.764618  1659 raft_consensus.cc:1240] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.11-20.11]
W20260709 13:04:29.740998  1660 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1241) took 2671 ms (client timeout 2516 ms). Trace:
I20260709 13:04:29.766758  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.766932  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.664009  8792 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.768517  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3431) took 2634 ms. Trace:
I20260709 13:04:29.738073  1741 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:29.769999  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:29.770141  1734 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:29.729326  8791 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:29.801872  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3441) took 2505 ms (client timeout 2395 ms). Trace:
I20260709 13:04:29.743219  9038 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:29.802397  1734 rpcz_store.cc:269] 0709 13:04:27.295939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.377527 (+ 81588us) service_pool.cc:224] Handling call
0709 13:04:29.801762 (+2424235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.802140  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:29.802690  1660 rpcz_store.cc:269] 0709 13:04:27.069232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.274051 (+204819us) service_pool.cc:224] Handling call
0709 13:04:29.739565 (+2465514us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:29.740912 (+  1347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44459776}
I20260709 13:04:29.802862  1739 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:04:29.804221  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.805267  1738 rpcz_store.cc:276] 0709 13:04:27.133789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.812033 (+1678244us) service_pool.cc:224] Handling call
0709 13:04:28.812540 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:29.313818 (+501278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.313876 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:29.315099 (+  1223us) log.cc:844] Serialized 52 byte log entry
0709 13:04:29.517184 (+202085us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:29.582948 (+ 65764us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.604496 (+ 21548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.604627 (+   131us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.758165 (+153538us) raft_consensus.cc:1685] finished
0709 13:04:29.758191 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.768446 (+ 10255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":434056,"prepare.queue_time_us":65629,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3358,"raft.queue_time_us":528598,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":6343,"thread_start_us":212794,"threads_started":2,"wal-append.queue_time_us":328408}
W20260709 13:04:29.809811  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167632us
I20260709 13:04:29.813787  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.814067  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2637) took 2773 ms. Trace:
I20260709 13:04:29.814602  1734 raft_consensus.cc:2368] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20.
I20260709 13:04:29.814451  1659 rpcz_store.cc:276] 0709 13:04:27.036055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.763909 (+2727854us) service_pool.cc:224] Handling call
0709 13:04:29.764361 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.808688 (+ 44327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.808980 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:29.815979  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.816483  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.817216  1738 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:29.813241  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:29.818274  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.769795  9158 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:29.819870  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.820185  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.805042  8847 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.823227  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.824842  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.809669  9096 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:29.827314  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.827589  1734 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:29.827212  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.833244  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.723729  8824 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.098s
I20260709 13:04:29.830235  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.842724  1593 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:29.832942  1592 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:29.841012  1585 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:29.809643  8861 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 13:04:29.847668  1659 raft_consensus.cc:2393] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 25: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:29.850416  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 46304.8us
I20260709 13:04:29.851135  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.853972  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:29.854651  2315 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:29.825644  9149 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:29.858304  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.858037  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.808220  9167 consensus_queue.cc:1059] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 13:04:29.855286  9158 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:29.865754  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.867115  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.867976  1660 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:29.868726  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.871559  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.872059  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1275) took 2489 ms. Trace:
I20260709 13:04:29.873855  9164 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:29.859151  9030 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:29.874938  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:29.884055  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.886191  1593 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:29.847234  8969 consensus_queue.cc:237] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.319963  1662 rpcz_store.cc:276] 0709 13:04:26.561369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.904810 (+2343441us) service_pool.cc:224] Handling call
0709 13:04:28.959239 (+ 54429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:28.987637 (+ 28398us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:29.149519 (+161882us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.316652 (+167133us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.316700 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:29.316755 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.316819 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.316946 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":236351,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":3168,"thread_start_us":327683,"threads_started":2,"wal-append.queue_time_us":161005}
I20260709 13:04:29.858656  9149 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:29.893893  9149 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:29.894764  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.895155  8792 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:29.895980  1662 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:29.897405  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.901947  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:29.903162  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.903854  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:29.904230  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.872606  1736 rpcz_store.cc:276] 0709 13:04:27.384305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.920391 (+1536086us) service_pool.cc:224] Handling call
0709 13:04:28.921065 (+   674us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.550474 (+629409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:29.550536 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:29.573110 (+ 22574us) log.cc:844] Serialized 26 byte log entry
0709 13:04:29.760414 (+187304us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:29.760475 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.760593 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:29.871752 (+111159us) raft_consensus.cc:1685] finished
0709 13:04:29.871787 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.871976 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":321682,"raft.queue_time_us":407656,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":156,"thread_start_us":349226,"threads_started":2,"wal-append.queue_time_us":226069}
I20260709 13:04:29.908277  1662 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:29.908715  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.911211  1592 raft_consensus.cc:2376] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23. Re-sending same reply.
I20260709 13:04:29.917622  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.918064  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:29.920347  1734 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:29.923012  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:29.924017  1660 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:29.928000  1705 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42028 dropped due to backpressure: service queue is full with 50 items
I20260709 13:04:29.928478  9049 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.929539  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:29.930909  1585 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:29.938385  1738 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:29.929989  9059 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:29.939334  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:29.943266  1736 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:29.941470  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:29.948550  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.949263  1662 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
I20260709 13:04:29.950022  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.950829  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.951587  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.952052  1662 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:29.953197  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.957818  1734 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:29.955817  1662 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:29.960825  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:29.961825  1660 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:29.947973  9177 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 13:04:29.928428  1705 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:42028 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1287)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3453)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3463)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3467)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3468)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1284)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2699)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2704)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1291)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3473)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3475)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2714)
I20260709 13:04:29.966526  9177 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:04:29.966929  9177 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:29.967801  9177 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:29.969017  9177 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:29.944641  9097 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:04:29.974229  9097 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:29.979637  9146 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:29.980412  9146 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:29.966183  1705 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:42028 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3453)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3463)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3467)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3468)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1284)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2699)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2704)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1291)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3473)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3475)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2714)
I20260709 13:04:29.985255  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.986460  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.987896  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:29.990417  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.005782  1557 leader_election.cc:400] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20.
I20260709 13:04:30.006443  1557 leader_election.cc:403] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:30.009258  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.020165  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:29.438002  1667 rpcz_store.cc:276] 0709 13:04:26.697648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.218015 (+2520367us) service_pool.cc:224] Handling call
0709 13:04:29.218521 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.218748 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:29.286691 (+ 67943us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.430110 (+143419us) log.cc:844] Serialized 14 byte log entry
0709 13:04:29.436706 (+  6596us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:29.436767 (+    61us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:29.436817 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:29.436884 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:29.437015 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6294,"prepare.queue_time_us":5723,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":36,"raft.queue_time_us":10870,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":164833,"threads_started":2}
I20260709 13:04:30.020990  1587 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:30.005836  9179 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:30.025944  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.026257  9179 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:30.026901  9179 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.027452  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.028254  1660 raft_consensus.cc:2393] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 25: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:30.029067  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:30.030087  1660 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:30.034840  8861 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.015625  9149 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.036139  2313 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:30.039723  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.042476  1705 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.043501  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.049413  9030 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:30.051964  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.052340  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.053220  1667 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 8.
W20260709 13:04:30.053144  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93862.3us
I20260709 13:04:30.053750  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.055356  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.056074  1587 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:30.045862  9059 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:30.075053  9157 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:30.077564  9049 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:30.081480  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.081861  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.082197  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.047784  9151 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.044845  9178 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.109905  1741 raft_consensus.cc:2393] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:30.111094  8798 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:30.113670  9096 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:30.114511  9112 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.188s	user 0.000s	sys 0.005s
I20260709 13:04:30.110833  1736 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:30.115890  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1188) took 4100 ms. Trace:
I20260709 13:04:30.098380  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:30.054785  8733 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.057265  8791 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.103624  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.109786  1659 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:30.116268  1588 rpcz_store.cc:276] 0709 13:04:26.015765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.413745 (+1397980us) service_pool.cc:224] Handling call
0709 13:04:27.414068 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:27.414432 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:27.414454 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:27.431144 (+ 16690us) log.cc:844] Serialized 26 byte log entry
0709 13:04:27.841358 (+410214us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:27.841415 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:27.861873 (+ 20458us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.113715 (+2251842us) spinlock_profiling.cc:243] Waited 89 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:30.115537 (+  1822us) raft_consensus.cc:1685] finished
0709 13:04:30.115558 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.115821 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10599,"raft.queue_time_us":108,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":11695,"spinlock_wait_cycles":204702976,"thread_start_us":386183,"threads_started":1,"wal-append.queue_time_us":1476026}
W20260709 13:04:30.117853  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57323.8us
I20260709 13:04:30.118871  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.118927  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.114096  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.119951  1588 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:30.120435  1587 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:30.121045  1659 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:30.121812  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.122154  1667 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:04:30.123776  1667 raft_consensus.cc:606] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:30.120447  8733 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.123154  1661 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 26: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:30.123020  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.101552  1660 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:30.061684  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.124783  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.126422  1587 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:30.128511  1660 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:30.129292  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1244) took 1337 ms. Trace:
W20260709 13:04:30.129345  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67809.6us
I20260709 13:04:30.133857  1588 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:30.135540  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.150038  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.145475  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.173739  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.152381  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:30.129771  1660 rpcz_store.cc:276] 0709 13:04:28.791824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.128021 (+1336197us) service_pool.cc:224] Handling call
0709 13:04:30.128324 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.129051 (+   727us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.129224 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.175814  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.176555  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.177373  1736 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:04:30.177893  8910 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:30.178721  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:30.180491  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55775.2us
I20260709 13:04:30.181308  1634 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:30.179637  1588 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 17: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:30.182826  1592 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 13:04:30.185011  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:30.195372  1736 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:30.131309  9096 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:30.178124  9151 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:30.194279  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.130614  8791 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:30.197831  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.200441  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.130362  9149 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:30.231948  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.240732  1591 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:30.198341  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.219201  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.238504  1662 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:30.139725  8733 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:30.247148  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52958.2us
I20260709 13:04:30.195290  8910 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:30.254361  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.259881  1592 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 13:04:30.184437  9167 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:30.196381  8824 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:30.267349  1587 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:30.268682  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.264256  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.273589  1736 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:30.273672  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.275120  1591 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:04:30.252585  8910 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.267593  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:30.281080  1587 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:30.194624  9102 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:30.187230  9097 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.229223  9148 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.282065  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.286684  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.270045  8791 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.270469  8824 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.293440  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.289135  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.290670  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.299008  1592 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:30.302973  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:30.287602  8861 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:04:30.304210  8861 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:30.304361  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.305277  8861 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.305476  1586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3514) took 3218 ms (client timeout 1831 ms). Trace:
W20260709 13:04:30.308174  1586 rpcz_store.cc:269] 0709 13:04:27.087402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.103098 (+ 15696us) service_pool.cc:224] Handling call
0709 13:04:30.305210 (+3202112us) spinlock_profiling.cc:243] Waited 3.2 s on lock 0x7b54004001d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:30.305355 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7357580928}
I20260709 13:04:30.287320  9151 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.306661  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.287896  9102 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.313233  1591 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:30.307636  8733 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:30.292590  9059 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:30.302973  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.301759  8824 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.291998  9148 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.320438  9148 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:30.320555  8847 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:30.326308  1590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3458) took 3439 ms (client timeout 2535 ms). Trace:
W20260709 13:04:30.326870  1590 rpcz_store.cc:269] 0709 13:04:26.888484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.209541 (+321057us) service_pool.cc:224] Handling call
0709 13:04:30.326062 (+3116521us) spinlock_profiling.cc:243] Waited 2.97 s on lock 0x7b540031f454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:30.326219 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6829464960}
I20260709 13:04:30.329903  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.330227  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3413) took 5309 ms. Trace:
I20260709 13:04:30.331418  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.332239  2310 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:30.336355  1557 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:30.338965  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.340113  9152 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:30.340534  9152 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.340886  9152 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:30.342022  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.342832  1737 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:30.343293  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.330687  1742 rpcz_store.cc:276] 0709 13:04:25.021273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.661706 (+640433us) service_pool.cc:224] Handling call
0709 13:04:25.686440 (+ 24734us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:25.917319 (+230879us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:25.917395 (+    76us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:25.976813 (+ 59418us) log.cc:844] Serialized 52 byte log entry
0709 13:04:26.802866 (+826053us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:26.802930 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.329733 (+3526803us) spinlock_profiling.cc:243] Waited 89.4 ms on lock 0x7b5400284954. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:30.329791 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.329861 (+    70us) raft_consensus.cc:1685] finished
0709 13:04:30.329875 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.330135 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55192,"raft.queue_time_us":269571,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":40403,"spinlock_wait_cycles":205657984,"thread_start_us":892375,"threads_started":2,"wal-append.queue_time_us":886615,"wal-append.run_cpu_time_us":2994,"wal-append.run_wall_time_us":1097590}
I20260709 13:04:30.353276  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.316474  9102 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:30.365732  8969 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.368323  9021 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:30.330524  9157 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:30.370963  1634 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:30.317013  8733 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.374907  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:30.351181  9059 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:30.380242  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.399310  1742 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:30.380466  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.381215  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.382337  9021 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:30.384971  9030 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:30.351240  9188 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:30.356150  9178 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20.
I20260709 13:04:30.414785  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:30.403678  8733 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:30.378333  8861 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:30.421267  1591 raft_consensus.cc:2393] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:30.404021  9106 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:30.408283  9191 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:30.409862  9188 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:30.416580  9052 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.548s	user 0.012s	sys 0.002s
I20260709 13:04:30.430334  9032 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:30.429873  9021 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:30.432327  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.437264  1706 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.443161  9191 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:30.450342  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:30.452742  1736 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:30.453975  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.454999  1736 raft_consensus.cc:2368] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:30.433957  9188 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:30.420380  9179 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.437090  9052 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.569s	user 0.012s	sys 0.002s
I20260709 13:04:30.438408  9032 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:30.455612  9030 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:30.496927  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.497764  8861 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:30.498004  1736 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:30.455103  9157 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:30.497287  1557 leader_election.cc:400] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:30.501248  1557 leader_election.cc:403] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 13:04:30.482717  8791 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:30.504627  8791 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.506685  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1247) took 1203 ms. Trace:
I20260709 13:04:30.507136  1660 rpcz_store.cc:276] 0709 13:04:29.302864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.183339 (+880475us) service_pool.cc:224] Handling call
0709 13:04:30.196714 (+ 13375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:30.281841 (+ 85127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:30.281899 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:30.284354 (+  2455us) log.cc:844] Serialized 26 byte log entry
0709 13:04:30.421384 (+137030us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:30.421422 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.421524 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:30.506394 (+ 84870us) raft_consensus.cc:1685] finished
0709 13:04:30.506419 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:30.506616 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1891,"raft.queue_time_us":49708,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":10405,"thread_start_us":156826,"threads_started":2,"wal-append.queue_time_us":198259}
I20260709 13:04:30.483058  9173 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 13:04:29.945220  1707 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55272 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2694)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2700)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3451)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2717)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2711)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3466)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3472)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2690)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2707)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2715)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3459)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2691)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2710)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2701)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1289)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3452)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2706)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2692)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3455)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3476)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2695)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2709)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3449)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3450)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3453)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3461)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3463)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3467)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3468)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1284)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1285)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1286)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2698)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2699)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1290)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2704)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1291)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2705)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2712)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3469)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3471)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3473)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3474)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3475)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2714)
I20260709 13:04:30.492508  9097 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:30.513937  9097 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.517573  9179 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:30.509191  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:30.445732  9149 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 13:04:30.495313  9059 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.531852  2310 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:04:30.534436  9146 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:30.535032  9030 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.438179  9106 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.540735  9151 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.498413  8861 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.538638  9148 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:30.552661  1736 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:30.536005  9178 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22.
I20260709 13:04:30.569666  1742 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:30.570868  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3466) took 1195 ms. Trace:
I20260709 13:04:30.571404  1742 rpcz_store.cc:276] 0709 13:04:29.375818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.350170 (+974352us) service_pool.cc:224] Handling call
0709 13:04:30.570777 (+220607us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.572762  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.573621  1742 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:30.574661  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3459) took 1234 ms. Trace:
W20260709 13:04:30.575060  1632 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.587s (SENT)
I20260709 13:04:30.575166  1742 rpcz_store.cc:276] 0709 13:04:29.340499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.572482 (+1231983us) service_pool.cc:224] Handling call
0709 13:04:30.574593 (+  2111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.577618  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.580161  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:30.580680  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83552.1us
I20260709 13:04:30.580977  1736 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:30.582728  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.582419  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.585744  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:30.578253  9194 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 13:04:30.549373  9030 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.589191  9195 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 13:04:30.591600  8733 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.588913  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3313) took 4919 ms (client timeout 1961 ms). Trace:
I20260709 13:04:30.562760  9151 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.549150  8824 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.604919  1668 rpcz_store.cc:269] 0709 13:04:25.669249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.620698 (+1951449us) service_pool.cc:224] Handling call
0709 13:04:30.588610 (+2967912us) spinlock_profiling.cc:243] Waited 583 ms on lock 0x7b54000aa5d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:30.588793 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1339871488}
I20260709 13:04:30.570938  9096 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 13:04:30.609794  1668 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:04:30.598537  9178 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:30.567634  9148 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.618103  1736 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:30.579360  9152 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.619668  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:30.624361  1660 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:30.549701  9191 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.620569  1742 raft_consensus.cc:2368] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:30.625242  1668 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: []
I20260709 13:04:30.624157  1557 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.577171  9032 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:30.638298  9032 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.625593  9178 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:30.641603  9178 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.643229  9183 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.560s	user 0.003s	sys 0.000s
I20260709 13:04:30.646422  8792 raft_consensus.cc:3055] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:30.647091  8792 raft_consensus.cc:740] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:30.648063  8792 consensus_queue.cc:260] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:30.651139  1737 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1274) took 3268 ms (client timeout 2040 ms). Trace:
W20260709 13:04:30.651798  1737 rpcz_store.cc:269] 0709 13:04:27.382599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.417257 (+ 34658us) service_pool.cc:224] Handling call
0709 13:04:30.650757 (+3233500us) spinlock_profiling.cc:243] Waited 2.9 s on lock 0x7b54003f9dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:30.650999 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6676215680}
I20260709 13:04:30.652640  9151 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:30.652992  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.653239  9152 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:30.654021  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3316) took 4902 ms (client timeout 1735 ms). Trace:
W20260709 13:04:30.654403  1664 rpcz_store.cc:269] 0709 13:04:25.752494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:27.069890 (+1317396us) service_pool.cc:224] Handling call
0709 13:04:30.653780 (+3583890us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b540030af54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:30.653950 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4723456}
I20260709 13:04:30.655272  1660 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 13:04:30.655674  9106 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:30.656575  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.657269  8861 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:30.656594  1668 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:30.659104  8824 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:30.658864  1659 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:30.662655  8733 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:30.628921  9077 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:30.663928  9077 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:30.664194  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.664858  9077 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.665561  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:30.668853  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.670193  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:30.651612  9030 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:30.680126  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.684808  8861 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:30.685245  8861 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:30.685631  8861 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:30.681629  1737 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:30.697337  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.698330  1737 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:30.699056  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3452) took 1360 ms. Trace:
I20260709 13:04:30.699460  1737 rpcz_store.cc:276] 0709 13:04:29.338432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.697030 (+1358598us) service_pool.cc:224] Handling call
0709 13:04:30.698978 (+  1948us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.701409  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.702191  1737 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:30.703233  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.704133  1737 raft_consensus.cc:2393] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:30.701200  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.709952  1736 raft_consensus.cc:2376] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17. Re-sending same reply.
I20260709 13:04:30.697798  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:30.714082  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:30.757716  9163 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 13:04:30.758257  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.758451  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2692) took 1089 ms. Trace:
I20260709 13:04:30.759042  1737 rpcz_store.cc:276] 0709 13:04:29.668791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.702949 (+1034158us) service_pool.cc:224] Handling call
0709 13:04:30.758381 (+ 55432us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.759352  1736 raft_consensus.cc:2410] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 13:04:30.760020  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3455) took 1420 ms. Trace:
I20260709 13:04:30.760188  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.760484  1736 rpcz_store.cc:276] 0709 13:04:29.339328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.711056 (+1371728us) service_pool.cc:224] Handling call
0709 13:04:30.759945 (+ 48889us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:30.760845  1737 raft_consensus.cc:2376] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18. Re-sending same reply.
I20260709 13:04:30.762163  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:30.763182  1737 raft_consensus.cc:2410] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 14 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
W20260709 13:04:30.805241  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:30.833863  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.835493  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.836169  1737 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:30.852151  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.853264  1736 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:30.695947  8792 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:30.839762  9030 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:30.842008  9179 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:30.833138  8910 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:30.857177  8733 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:30.895238  1737 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:30.947574  9102 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.549s	user 0.012s	sys 0.000s
I20260709 13:04:30.948316  9102 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:30.952148  1736 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:30.953338  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:30.954689  1736 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:30.876255  9179 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:31.023032  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.024322  1736 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:30.966950  9179 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:31.027062  9188 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.532s	user 0.007s	sys 0.005s
I20260709 13:04:30.878750  9030 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.009964  9178 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:31.051684  9178 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.053627  9188 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.054461  9178 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.056115  9188 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.056435  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:31.057291  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:31.057991  9102 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:30.950531  8733 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:31.032280  9199 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 13:04:31.046628  8910 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:31.066165  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3535) took 1254 ms. Trace:
I20260709 13:04:31.067485  1586 rpcz_store.cc:276] 0709 13:04:29.811929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.447555 (+635626us) service_pool.cc:224] Handling call
0709 13:04:30.484092 (+ 36537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:30.492885 (+  8793us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:04:30.573692 (+ 80807us) log.cc:844] Serialized 14 byte log entry
0709 13:04:30.668145 (+ 94453us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:30.671952 (+  3807us) log.cc:844] Serialized 26 byte log entry
0709 13:04:30.672162 (+   210us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:30.672212 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:30.672315 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:31.065913 (+393598us) raft_consensus.cc:1685] finished
0709 13:04:31.065937 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.066095 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":115331,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":1671,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":3748,"spinlock_wait_cycles":18454656,"thread_start_us":132313,"threads_started":2,"wal-append.queue_time_us":71203}
I20260709 13:04:31.072168  1586 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 13:04:31.072799  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3537) took 1260 ms. Trace:
I20260709 13:04:31.073120  1586 rpcz_store.cc:276] 0709 13:04:29.813002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.071419 (+1258417us) service_pool.cc:224] Handling call
0709 13:04:31.071789 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.072522 (+   733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.072741 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:31.076042  9116 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.170s	user 0.011s	sys 0.001s
I20260709 13:04:31.075742  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.076872  1736 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:31.092410  1586 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:31.092629  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.093200  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3550) took 1195 ms. Trace:
I20260709 13:04:31.093305  1736 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:31.093633  1586 rpcz_store.cc:276] 0709 13:04:29.898285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.080295 (+1182010us) service_pool.cc:224] Handling call
0709 13:04:31.092023 (+ 11728us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.092886 (+   863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.093138 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.093997  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2695) took 1331 ms. Trace:
I20260709 13:04:31.094666  1736 rpcz_store.cc:276] 0709 13:04:29.762901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.080136 (+1317235us) service_pool.cc:224] Handling call
0709 13:04:31.093930 (+ 13794us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.101382  9030 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.106691  8733 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.113385  1586 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:31.114826  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3553) took 1215 ms. Trace:
I20260709 13:04:31.113780  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.115269  1586 rpcz_store.cc:276] 0709 13:04:29.899248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.094596 (+1195348us) service_pool.cc:224] Handling call
0709 13:04:31.099523 (+  4927us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.114444 (+ 14921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.114756 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.119122  1736 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:31.089820  8910 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.124758  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3333) took 1344 ms. Trace:
I20260709 13:04:31.125221  1662 rpcz_store.cc:276] 0709 13:04:29.780682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.987551 (+206869us) service_pool.cc:224] Handling call
0709 13:04:31.124567 (+1137016us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54003c8f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.124689 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574229376}
I20260709 13:04:31.091208  8861 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:31.128245  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800340520 after lost signal to thread 1525
I20260709 13:04:31.127853  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3557) took 1190 ms. Trace:
I20260709 13:04:31.131281  1590 rpcz_store.cc:276] 0709 13:04:29.937332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.126910 (+1189578us) service_pool.cc:224] Handling call
0709 13:04:31.127270 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.127495 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.127526 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.127586 (+    60us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:31.127627 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.127680 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.127794 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.135646  1586 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 13:04:31.150328  9184 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.065s	user 0.000s	sys 0.010s
I20260709 13:04:31.146611  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3554) took 1232 ms. Trace:
W20260709 13:04:31.153092  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.152434  1586 rpcz_store.cc:276] 0709 13:04:29.911218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.120636 (+1209418us) service_pool.cc:224] Handling call
0709 13:04:31.134977 (+ 14341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.141245 (+  6268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.141609 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:31.157071  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.156562  8861 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.163561  1586 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [18.11-20.12]
W20260709 13:04:31.164183  9121 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.238s	user 0.004s	sys 0.000s
W20260709 13:04:31.170127  9113 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.243s	user 0.003s	sys 0.000s
I20260709 13:04:31.177345  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1279) took 3156 ms. Trace:
I20260709 13:04:31.177917  1739 rpcz_store.cc:276] 0709 13:04:28.021428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.986650 (+965222us) service_pool.cc:224] Handling call
0709 13:04:29.802587 (+815937us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:29.807656 (+  5069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.177095 (+1369439us) spinlock_profiling.cc:243] Waited 815 ms on lock 0x7b5400404a50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.177273 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1875064064}
I20260709 13:04:31.179805  1739 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:31.180410  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3449) took 1842 ms. Trace:
I20260709 13:04:31.180727  1739 rpcz_store.cc:276] 0709 13:04:29.337571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.179185 (+1841614us) service_pool.cc:224] Handling call
0709 13:04:31.179591 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.180175 (+   584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.180354 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.186583  1736 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:31.187327  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2709) took 1311 ms. Trace:
I20260709 13:04:31.187681  1736 rpcz_store.cc:276] 0709 13:04:29.876036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.095776 (+1219740us) service_pool.cc:224] Handling call
0709 13:04:31.187261 (+ 91485us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:31.189240  2315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1069) took 1706 ms (client timeout 1587 ms). Trace:
W20260709 13:04:31.189719  2315 rpcz_store.cc:269] 0709 13:04:29.482724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.482972 (+   248us) service_pool.cc:224] Handling call
0709 13:04:31.189005 (+1706033us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54007233d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.189136 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263610112}
W20260709 13:04:31.188184  9117 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 2.326s	user 0.004s	sys 0.011s
I20260709 13:04:31.197075  2317 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:31.198984  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1076) took 1513 ms. Trace:
I20260709 13:04:31.200819  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:31.199375  2317 rpcz_store.cc:276] 0709 13:04:29.685664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.687673 (+  2009us) service_pool.cc:224] Handling call
0709 13:04:31.196713 (+1509040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.197492 (+   779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.198749 (+  1257us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54007233d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.198913 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3468969856}
I20260709 13:04:31.203369  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.204325  8910 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:31.207093  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.208312  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.209681  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.210634  1557 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:31.208279  8910 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:31.214340  8910 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.208976  9167 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:31.218605  9167 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:31.218994  9167 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.212592  8855 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:31.226966  8855 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:31.227396  8855 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:31.212437  9176 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:31.229943  9176 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:31.230396  9176 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:31.238826  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.239681  1734 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
W20260709 13:04:31.240243  1734 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2716) took 1313 ms (client timeout 1570 ms). Trace:
W20260709 13:04:31.240659  1734 rpcz_store.cc:269] 0709 13:04:29.927211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.238443 (+1311232us) service_pool.cc:224] Handling call
0709 13:04:31.240172 (+  1729us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.223039  8792 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader election won for term 15
I20260709 13:04:31.242399  8792 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:31.242794  9021 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:31.243475  8792 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:31.246351  1522 debug-util.cc:398] Leaking SignalData structure 0x7b080003dc80 after lost signal to thread 1617
I20260709 13:04:31.250979  8861 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:31.237128  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 498408us
I20260709 13:04:31.260788  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.262375  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.221375  1483 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 13, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 13 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:31.266577  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.203102  9179 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.267644  9207 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:31.251808  8861 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.268102  9159 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:31.268914  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.282672  1588 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 24.
I20260709 13:04:31.286371  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.268893  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:31.270975  1632 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.649s (SENT)
I20260709 13:04:31.298187  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:31.274324  9188 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:31.274951  8765 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.276356  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1236) took 4617 ms. Trace:
I20260709 13:04:31.302878  1663 rpcz_store.cc:276] 0709 13:04:26.658813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:28.921227 (+2262414us) service_pool.cc:224] Handling call
0709 13:04:28.935266 (+ 14039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:29.139630 (+204364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.276099 (+2136469us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54001a86d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:31.276290 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447458176}
I20260709 13:04:31.277302  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.304862  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.304641  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.305490  1733 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.305876  1588 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:04:31.299911  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3562) took 1109 ms. Trace:
I20260709 13:04:31.307268  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.284932  9159 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.311288  1736 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.308080  8635 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:31.303622  9188 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:31.331259  9164 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:31.331790  9164 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:31.332681  9164 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.334095  8861 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.327770  9188 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:31.307770  1483 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 3 to 15, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 15 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:31.410871  9215 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 13:04:31.349083  1733 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:31.466492  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.467162  1733 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:31.467247  1483 catalog_manager.cc:5697] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 12, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 12 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:31.468145  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.468794  1733 raft_consensus.cc:2376] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21. Re-sending same reply.
I20260709 13:04:31.469904  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.470496  1733 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:31.470721  9159 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:31.471549  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.472644  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.473308  1733 raft_consensus.cc:2410] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 13:04:31.358567  1662 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:31.459192  8953 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.459851  9164 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.462391  9179 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.462567  8855 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.463874  1736 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:31.486231  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.487411  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:31.487815  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.488139  1736 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:31.464443  8733 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:31.453923  9182 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.349s	user 0.010s	sys 0.000s
I20260709 13:04:31.463886  9176 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.483901  1483 catalog_manager.cc:5697] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 11, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:31.506588  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.507295  1593 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 13.
I20260709 13:04:31.488950  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.490265  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1080) took 1589 ms. Trace:
I20260709 13:04:31.505669  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.508484  1736 raft_consensus.cc:2376] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22. Re-sending same reply.
I20260709 13:04:31.508926  1733 raft_consensus.cc:2410] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 13:04:31.508443  2313 rpcz_store.cc:276] 0709 13:04:29.901040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.902408 (+  1368us) service_pool.cc:224] Handling call
0709 13:04:29.902726 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.036656 (+133930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.489940 (+1453284us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5400724054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:31.490187 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":306233728}
I20260709 13:04:31.521028  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.529078  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.530282  1736 raft_consensus.cc:2376] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15. Re-sending same reply.
I20260709 13:04:31.531636  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.532701  1736 raft_consensus.cc:2393] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:31.534107  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.535189  1593 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 24.
I20260709 13:04:31.540496  2283 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 24.
I20260709 13:04:31.542618  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.543932  1733 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:31.538395  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.547441  1736 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:31.554157  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.555255  1593 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:31.505882  8953 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 13:04:31.505329  8733 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.568665  1736 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 11. (term mismatch)
I20260709 13:04:31.566440  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.569422  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.568713  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:31.564875  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101563us
I20260709 13:04:31.569455  1736 pending_rounds.cc:85] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 10
I20260709 13:04:31.566535  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.571727  1733 raft_consensus.cc:2410] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:04:31.572487  1588 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:31.570962  1736 pending_rounds.cc:107] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 12.11
I20260709 13:04:31.571664  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:31.563221  9077 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.010s	sys 0.029s
I20260709 13:04:31.573094  1736 raft_consensus.cc:2889] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:31.568418  9119 log.cc:927] Time spent T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 2.641s	user 0.005s	sys 0.001s
I20260709 13:04:31.575542  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.574512  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:31.578500  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.579375  9179 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:31.580018  9179 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:31.580865  9179 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.586725  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3453) took 2238 ms. Trace:
I20260709 13:04:31.587669  1736 rpcz_store.cc:276] 0709 13:04:29.338719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.567752 (+2229033us) service_pool.cc:224] Handling call
0709 13:04:31.568296 (+   544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.576777 (+  8481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.576973 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.589284  1736 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:31.586826  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:31.575599  9077 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.010s	sys 0.029s
I20260709 13:04:31.600157  1590 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.590524  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3463) took 2235 ms. Trace:
I20260709 13:04:31.600793  1736 rpcz_store.cc:276] 0709 13:04:29.355083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.588586 (+2233503us) service_pool.cc:224] Handling call
0709 13:04:31.588894 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.590250 (+  1356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.590469 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.602480  1736 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [18.11-20.12]
I20260709 13:04:31.603183  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3467) took 2227 ms. Trace:
I20260709 13:04:31.584509  8657 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:31.603658  1736 rpcz_store.cc:276] 0709 13:04:29.376151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.601842 (+2225691us) service_pool.cc:224] Handling call
0709 13:04:31.602209 (+   367us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:31.602904 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.603128 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.587337  8910 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:31.605216  1736 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.587510  9021 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:31.605863  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3468) took 2229 ms. Trace:
I20260709 13:04:31.606395  1736 rpcz_store.cc:276] 0709 13:04:29.376460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.604720 (+2228260us) service_pool.cc:224] Handling call
0709 13:04:31.605031 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.605626 (+   595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.605811 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.606812  1734 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 16. Current term is 17. Ops: []
I20260709 13:04:31.607566  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3461) took 2265 ms. Trace:
I20260709 13:04:31.607893  1734 rpcz_store.cc:276] 0709 13:04:29.342849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.588229 (+2245380us) service_pool.cc:224] Handling call
0709 13:04:31.606600 (+ 18371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.607286 (+   686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.607485 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.603909  8657 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:31.576913  8792 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:31.637300  8792 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:31.637593  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3534) took 1825 ms. Trace:
I20260709 13:04:31.640674  8657 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.605757  8953 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:31.619457  8861 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:31.644619  1738 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2696) took 1881 ms (client timeout 1599 ms). Trace:
I20260709 13:04:31.598028  9136 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 13:04:31.658275  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:31.621282  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.621713  8765 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:31.622852  9079 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 13:04:31.624388  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2638) took 4588 ms. Trace:
I20260709 13:04:31.666214  1659 rpcz_store.cc:276] 0709 13:04:27.036411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.121601 (+3085190us) service_pool.cc:224] Handling call
0709 13:04:30.658588 (+536987us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:30.659924 (+  1336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.624183 (+964259us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b540030af50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.624325 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1223746304}
I20260709 13:04:31.624914  8855 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:31.625059  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1249) took 2035 ms (client timeout 2220 ms). Trace:
W20260709 13:04:31.668733  1661 rpcz_store.cc:269] 0709 13:04:29.590176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.724810 (+134634us) service_pool.cc:224] Handling call
0709 13:04:31.624846 (+1900036us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.624999 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715159424}
I20260709 13:04:31.625273  1665 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.625492  1667 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 25. Current term is 26. Ops: []
I20260709 13:04:31.628155  1737 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:31.628001  9167 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:31.633028  1741 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55272 (request call id 3456) took 2293 ms (client timeout 1972 ms). Trace:
I20260709 13:04:31.633651  9148 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.589202  9188 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.637836  8792 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:31.611806  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800064b40 after lost signal to thread 1692
W20260709 13:04:31.694279  1522 debug-util.cc:398] Leaking SignalData structure 0x7b080034c840 after lost signal to thread 1766
W20260709 13:04:31.696305  1522 debug-util.cc:398] Leaking SignalData structure 0x7b0800219200 after lost signal to thread 2342
W20260709 13:04:31.696622  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08003fb700 after lost signal to thread 2343
I20260709 13:04:31.638212  1591 rpcz_store.cc:276] 0709 13:04:29.811656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.423552 (+611896us) service_pool.cc:224] Handling call
0709 13:04:30.423861 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.134667 (+710806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.134716 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.134779 (+    63us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:31.134844 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.637223 (+502379us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b540031ef54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:31.637298 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.637491 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1633697152}
I20260709 13:04:31.698395  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.699200  1591 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:31.705737  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:31.678043  1741 rpcz_store.cc:269] 0709 13:04:29.339588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.812588 (+473000us) service_pool.cc:224] Handling call
0709 13:04:31.632832 (+1820244us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54001095d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.632958 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":649390464}
I20260709 13:04:31.643313  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1285) took 2002 ms. Trace:
I20260709 13:04:31.707643  1734 rpcz_store.cc:276] 0709 13:04:29.640847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.608706 (+1967859us) service_pool.cc:224] Handling call
0709 13:04:31.642789 (+ 34083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.642998 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.643017 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.643052 (+    35us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:31.643089 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.643128 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.643223 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.708477  1592 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.708946  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.709596  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.709625  1734 raft_consensus.cc:2376] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25. Re-sending same reply.
I20260709 13:04:31.646076  1593 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:31.712762  2315 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "219129ac643e4650b611a85f565f5990"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:31.713371  2315 raft_consensus.cc:606] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:31.714511  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.715298  1734 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:31.716346  1589 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:31.716809  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.717671  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.718451  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.719033  1665 raft_consensus.cc:2376] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25. Re-sending same reply.
I20260709 13:04:31.719219  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.717909  1734 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:31.721824  1734 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.12-21.13]
W20260709 13:04:31.654832  1738 rpcz_store.cc:269] 0709 13:04:29.763360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.826960 (+ 63600us) service_pool.cc:224] Handling call
0709 13:04:31.644390 (+1817430us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54003511d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.644542 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240313088}
I20260709 13:04:31.730456  1659 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.730774  1738 raft_consensus.cc:1240] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.731369  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2699) took 1967 ms. Trace:
I20260709 13:04:31.731720  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2698) took 1967 ms. Trace:
I20260709 13:04:31.731683  1738 rpcz_store.cc:276] 0709 13:04:29.764355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.730338 (+1965983us) service_pool.cc:224] Handling call
0709 13:04:31.730603 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.731150 (+   547us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.731315 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.732196  1734 rpcz_store.cc:276] 0709 13:04:29.763970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.721201 (+1957231us) service_pool.cc:224] Handling call
0709 13:04:31.721548 (+   347us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:31.731439 (+  9891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.731665 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.733419  1734 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: []
I20260709 13:04:31.657198  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:31.658026  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2630) took 5143 ms (client timeout 2394 ms). Trace:
W20260709 13:04:31.660106  9124 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 2.732s	user 0.006s	sys 0.007s
W20260709 13:04:31.733969  1666 rpcz_store.cc:269] 0709 13:04:26.501910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:26.762271 (+260361us) service_pool.cc:224] Handling call
0709 13:04:31.644490 (+4882219us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b54000a96d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.644629 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":853866240}
I20260709 13:04:31.661170  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:31.653380  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.654341  9158 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.751693  9158 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.671092  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1286) took 2030 ms. Trace:
I20260709 13:04:31.726648  9123 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:31.677325  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:31.754114  9158 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.753679  1737 rpcz_store.cc:276] 0709 13:04:29.641189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.627566 (+1986377us) service_pool.cc:224] Handling call
0709 13:04:31.627901 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.670820 (+ 42919us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.671033 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.635646  8733 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:31.711943  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08001ec540 after lost signal to thread 9044
I20260709 13:04:31.754846  8855 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:31.755241  1522 debug-util.cc:398] Leaking SignalData structure 0x7b08003372a0 after lost signal to thread 9053
W20260709 13:04:31.756093  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2691) took 2096 ms (client timeout 2115 ms). Trace:
I20260709 13:04:31.756421  1739 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:31.756486  1742 rpcz_store.cc:269] 0709 13:04:29.659713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.576298 (+916585us) service_pool.cc:224] Handling call
0709 13:04:31.755872 (+1179574us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.756011 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77616000}
I20260709 13:04:31.693923  9176 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.666395  8861 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.758387  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3450) took 2420 ms. Trace:
I20260709 13:04:31.758481  8955 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:31.758935  8955 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:31.758744  1739 rpcz_store.cc:276] 0709 13:04:29.337874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.181452 (+1843578us) service_pool.cc:224] Handling call
0709 13:04:31.756230 (+574778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.756974 (+   744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.758205 (+  1231us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b540039dad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.758331 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1319043456}
I20260709 13:04:31.759925  8861 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.719820  1661 raft_consensus.cc:2376] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:04:31.762248  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.764006  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.764794  1661 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.766661  1740 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:04:31.767334  9188 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:31.711535  9128 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 13:04:31.768534  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:31.771845  9219 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 13:04:31.772337  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3558) took 1583 ms. Trace:
W20260709 13:04:31.755280  1585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3532) took 1944 ms (client timeout 1734 ms). Trace:
W20260709 13:04:31.773116  1585 rpcz_store.cc:269] 0709 13:04:29.811107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:29.819952 (+  8845us) service_pool.cc:224] Handling call
0709 13:04:31.755064 (+1935112us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b540042e354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.755184 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38828416}
I20260709 13:04:31.719724  1483 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 15. New cstate: current_term: 15 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:31.774523  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.774745  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3343) took 1988 ms. Trace:
I20260709 13:04:31.775112  1667 rpcz_store.cc:276] 0709 13:04:29.786659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:30.253470 (+466811us) service_pool.cc:224] Handling call
0709 13:04:31.625292 (+1371822us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.676325 (+ 51033us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.774560 (+ 98235us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b540013ecd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:31.774687 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3153351936}
I20260709 13:04:31.775738  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:31.776310  1585 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:31.776260  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.778048  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.779359  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.772771  1590 rpcz_store.cc:276] 0709 13:04:30.189192 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.136448 (+947256us) service_pool.cc:224] Handling call
0709 13:04:31.136880 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.600689 (+463809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.771978 (+171289us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400431054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:31.772267 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1064652544}
I20260709 13:04:31.780418  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:31.781025  1588 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:31.781109  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.781669  8791 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:31.781778  1667 raft_consensus.cc:2376] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15. Re-sending same reply.
I20260709 13:04:31.783111  1667 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.783685  8955 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.785014  8955 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.785629  1667 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 13:04:31.786762  9149 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:31.786991  1667 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:31.727649  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.788321  1667 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 13:04:31.754006  9123 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:31.781041  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.307720  1586 rpcz_store.cc:276] 0709 13:04:30.190423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.164841 (+974418us) service_pool.cc:224] Handling call
0709 13:04:31.165164 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.165385 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:31.199883 (+ 34498us) log.cc:844] Serialized 14 byte log entry
0709 13:04:31.299503 (+ 99620us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.299564 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:31.299632 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.299699 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.299847 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":96138,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":4022,"raft.queue_time_us":92,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":7721,"thread_start_us":133381,"threads_started":2}
I20260709 13:04:31.789906  1590 raft_consensus.cc:2410] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 23 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:31.790617  1586 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.791695  9123 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.792040  9097 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:31.795967  1592 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 13:04:31.767398  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2705) took 1908 ms. Trace:
I20260709 13:04:31.724460  1741 raft_consensus.cc:2376] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20. Re-sending same reply.
I20260709 13:04:31.798334  1740 rpcz_store.cc:276] 0709 13:04:29.858775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.765831 (+1907056us) service_pool.cc:224] Handling call
0709 13:04:31.766376 (+   545us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.767114 (+   738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.767337 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.725337  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.727881  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:31.735566  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 82212.6us
I20260709 13:04:31.799878  1740 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.743742  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.745493  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.746165  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.802448  1666 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:31.750111  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.750797  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.760871  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1291) took 1929 ms. Trace:
I20260709 13:04:31.803694  1739 rpcz_store.cc:276] 0709 13:04:29.831844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.759948 (+1928104us) service_pool.cc:224] Handling call
0709 13:04:31.760378 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.760563 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.760590 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.760635 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:31.760665 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.760718 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.760815 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.805282  1739 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [19.10-20.11]
I20260709 13:04:31.805867  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2713) took 1919 ms. Trace:
I20260709 13:04:31.806140  1739 rpcz_store.cc:276] 0709 13:04:29.886953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.804659 (+1917706us) service_pool.cc:224] Handling call
0709 13:04:31.805029 (+   370us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:31.805625 (+   596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.805816 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.807300  1739 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: [22.11-22.11]
I20260709 13:04:31.807823  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3469) took 1908 ms. Trace:
I20260709 13:04:31.787184  9149 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:31.736871  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:31.610431  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:31.809484  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:31.810304  1591 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:31.810947  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2703) took 2000 ms. Trace:
I20260709 13:04:31.808085  1739 rpcz_store.cc:276] 0709 13:04:29.899033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.806784 (+1907751us) service_pool.cc:224] Handling call
0709 13:04:31.807066 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.807617 (+   551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.807780 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.809662  1733 raft_consensus.cc:2393] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:31.811378  1734 rpcz_store.cc:276] 0709 13:04:29.810223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.732972 (+1922749us) service_pool.cc:224] Handling call
0709 13:04:31.733241 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.810614 (+ 77373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.810886 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.815553  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2712) took 1918 ms. Trace:
I20260709 13:04:31.815970  1740 rpcz_store.cc:276] 0709 13:04:29.883658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.799351 (+1915693us) service_pool.cc:224] Handling call
0709 13:04:31.799701 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.801679 (+  1978us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.801880 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.817493  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3474) took 1916 ms. Trace:
I20260709 13:04:31.817848  1740 rpcz_store.cc:276] 0709 13:04:29.901040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.816703 (+1915663us) service_pool.cc:224] Handling call
0709 13:04:31.816971 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.817166 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.817194 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.817235 (+    41us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:31.817267 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.817320 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.817439 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.819845  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.820640  1665 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:31.820998  1734 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:31.821635  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3473) took 1921 ms. Trace:
I20260709 13:04:31.821978  1734 rpcz_store.cc:276] 0709 13:04:29.900610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.812165 (+1911555us) service_pool.cc:224] Handling call
0709 13:04:31.820668 (+  8503us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.821367 (+   699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.821570 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.825785  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.827268  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.830454  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.833777  1741 raft_consensus.cc:1240] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 13:04:31.834558  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2708) took 1958 ms. Trace:
I20260709 13:04:31.830727  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.835008  1741 rpcz_store.cc:276] 0709 13:04:29.875736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.799196 (+1923460us) service_pool.cc:224] Handling call
0709 13:04:31.833443 (+ 34247us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.834216 (+   773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.834483 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.836248  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.840555  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.843218  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.845670  1666 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:31.851895  2281 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.040s (SENT)
I20260709 13:04:31.856376  1585 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:31.876964  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.878930  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.879654  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.880503  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.880663  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.883198  1664 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:31.884044  1733 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:31.884167  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.776546  1483 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 9 to 16, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:31.887989  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2714) took 1980 ms. Trace:
I20260709 13:04:31.888410  1733 rpcz_store.cc:276] 0709 13:04:29.907266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.839503 (+1932237us) service_pool.cc:224] Handling call
0709 13:04:31.883854 (+ 44351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.887714 (+  3860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.887930 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.890347  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:31.886273  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 175017us
I20260709 13:04:31.898262  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.749400  8657 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.901822  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1294) took 1388 ms. Trace:
I20260709 13:04:31.902604  8657 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:31.904922  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1295) took 1391 ms. Trace:
I20260709 13:04:31.893858  8733 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:31.905346  1735 rpcz_store.cc:276] 0709 13:04:30.513230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.903927 (+1390697us) service_pool.cc:224] Handling call
0709 13:04:31.904261 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.904470 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.904500 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.904548 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:31.904595 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.904650 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.904845 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:31.906208  9222 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 13:04:31.906282  8733 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:31.909870  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.909332  2318 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.823194  9068 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 13:04:31.915701  9068 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:31.916066  9068 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:31.894084  9159 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 13:04:31.918087  1661 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:31.919047  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:31.922408  1592 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:31.839911  9128 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 13:04:31.840227  9158 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 13:04:31.850093  9188 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 13:04:31.902294  1733 rpcz_store.cc:276] 0709 13:04:30.512953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.889362 (+1376409us) service_pool.cc:224] Handling call
0709 13:04:31.901133 (+ 11771us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.901350 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.901375 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.901419 (+    44us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:31.901452 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.901540 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.901654 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.925590  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:31.926548  1661 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.916985  9068 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:31.921830  1585 raft_consensus.cc:1240] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:31.837138  9077 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 13:04:31.919756  9189 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:31.887897  8791 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:31.920310  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3478) took 1406 ms. Trace:
I20260709 13:04:31.953833  1735 rpcz_store.cc:276] 0709 13:04:30.525626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.906473 (+1380847us) service_pool.cc:224] Handling call
0709 13:04:31.906751 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.906950 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.906976 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.907016 (+    40us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:31.907058 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.907121 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:31.920246 (+ 13125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:31.957999  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.964524  1735 raft_consensus.cc:2376] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22. Re-sending same reply.
I20260709 13:04:31.966008  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.966645  1735 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:31.967466  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:31.931255  1705 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.735s (SENT)
I20260709 13:04:31.969573  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.936331  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.970546  1661 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:31.970664  1665 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:31.900595  1588 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:31.969597  1735 raft_consensus.cc:2368] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:31.974828  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.975674  1735 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.983896  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.985419  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:31.976066  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:31.936153  9158 consensus_queue.cc:1243] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:31.992827  1592 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
W20260709 13:04:32.002228  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 70990.6us
W20260709 13:04:32.003026  1705 leader_election.cc:336] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.739s (SENT)
I20260709 13:04:32.003453  1705 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:31.962556  9188 consensus_queue.cc:1243] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:32.008917  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:31.974025  8733 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:32.046123  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 155821us
I20260709 13:04:32.046934  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:32.051234  1588 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:32.052850  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:32.053912  1735 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:32.054155  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:32.054559  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.054993  1588 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:32.055231  1735 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:32.055972  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3480) took 1507 ms. Trace:
I20260709 13:04:32.056483  1735 rpcz_store.cc:276] 0709 13:04:30.548454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.054650 (+1506196us) service_pool.cc:224] Handling call
0709 13:04:32.054915 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:32.055606 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.055907 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:32.053030  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:32.052889  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:31.924777  9149 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.060225  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:32.064530  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61528.9us
I20260709 13:04:32.066164  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.061172  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:32.059175  1708 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:32.062721  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:32.063323  1585 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:32.062728  1592 raft_consensus.cc:1275] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:32.068615  1588 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:31.998291  9068 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:32.069691  1585 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: [20.10-22.11]
I20260709 13:04:32.070101  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:32.071785  1556 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.813s (SENT)
I20260709 13:04:32.072678  1735 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:32.067830  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:32.076581  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.073519  1664 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:32.077467  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3481) took 1520 ms. Trace:
I20260709 13:04:32.078070  1735 rpcz_store.cc:276] 0709 13:04:30.557073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.057334 (+1500261us) service_pool.cc:224] Handling call
0709 13:04:32.072286 (+ 14952us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:32.077082 (+  4796us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.077392 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:32.086858  1585 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:31.987697  8792 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.005796  9159 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:32.106218  1592 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:32.004894  9176 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.080s
I20260709 13:04:32.114198  1735 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:32.109862  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3475) took 2208 ms. Trace:
W20260709 13:04:32.124967  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58299.7us
I20260709 13:04:32.127494  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:31.991000  8855 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 13:04:32.129029  1735 raft_consensus.cc:1275] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:32.130159  1588 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:32.131631  1735 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 13:04:32.062520  8791 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:32.122844  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 133221us
I20260709 13:04:32.122284  1592 raft_consensus.cc:3055] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:32.124617  1741 rpcz_store.cc:276] 0709 13:04:29.901338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.836098 (+1934760us) service_pool.cc:224] Handling call
0709 13:04:31.880120 (+ 44022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.880406 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:31.973377 (+ 92971us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.100927 (+127550us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.109297 (+  8370us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:32.109379 (+    82us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:32.109462 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.109598 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.109794 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":215012,"threads_started":4}
I20260709 13:04:32.068050  9186 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:04:32.157722  1592 raft_consensus.cc:740] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:32.139210  1556 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.949s (SENT)
I20260709 13:04:32.068185  9123 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:32.158867  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.044507  8861 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.085s
I20260709 13:04:32.134028  1735 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:32.051448  9189 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 13:04:32.163187  1664 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:32.114466  9148 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:32.182103  1664 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.183358  1664 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.184062  8791 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:32.184680  8791 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:32.184770  1664 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:32.187613  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.188426  1666 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:32.188354  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.189872  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.190603  8792 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:32.191120  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.193297  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.164088  1592 consensus_queue.cc:260] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.174664  9244 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:32.194455  1733 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 13:04:32.175206  9223 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20260709 13:04:32.176070  9229 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 13:04:32.192782  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:32.197371  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 335937us
W20260709 13:04:32.199214  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 140112us
I20260709 13:04:32.200232  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:32.135535  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:32.199906  2281 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.856s (SENT)
I20260709 13:04:32.201994  2281 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.202351  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:32.124675  9149 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:32.203797  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64640.7us
I20260709 13:04:32.208913  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.105994  8733 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:32.074136  8677 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.124s
W20260709 13:04:32.209470  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73897.3us
I20260709 13:04:32.295747  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3471) took 2396 ms. Trace:
I20260709 13:04:32.289970  9167 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:32.298725  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:32.297699  1739 rpcz_store.cc:276] 0709 13:04:29.899823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.811801 (+1911978us) service_pool.cc:224] Handling call
0709 13:04:31.812170 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.832127 (+ 19957us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.832175 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:31.844608 (+ 12433us) log.cc:844] Serialized 26 byte log entry
0709 13:04:32.082228 (+237620us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:32.082291 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.082394 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:32.295466 (+213072us) raft_consensus.cc:1685] finished
0709 13:04:32.295487 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.295659 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1220,"raft.queue_time_us":21111,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":34958,"thread_start_us":187641,"threads_started":1,"wal-append.queue_time_us":378653}
I20260709 13:04:32.299942  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.222656  1593 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:32.245718  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1290) took 2425 ms. Trace:
I20260709 13:04:32.301290  1737 rpcz_store.cc:276] 0709 13:04:29.820714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.754864 (+1934150us) service_pool.cc:224] Handling call
0709 13:04:31.755277 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.800117 (+ 44840us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.800147 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:31.800762 (+   615us) log.cc:844] Serialized 26 byte log entry
0709 13:04:32.050147 (+249385us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:32.050203 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.050314 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:32.245434 (+195120us) raft_consensus.cc:1685] finished
0709 13:04:32.245458 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.245641 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44397,"raft.queue_time_us":45657,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":28552,"thread_start_us":169841,"threads_started":1,"wal-append.queue_time_us":410366}
I20260709 13:04:32.088675  1483 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 12, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 12 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:32.159819  9097 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:32.231225  9236 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:32.304507  9236 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.304945  1483 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 7 to 11, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 11 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:32.161477  8765 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:32.306958  1483 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 13, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 13 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:32.274096  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1284) took 2896 ms. Trace:
I20260709 13:04:32.202602  9123 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.238973  8791 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.8, Last appended by leader: 8, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.332058  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.208453  9128 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:32.210451  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:32.334725  1660 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:32.229833  1664 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 11. (term mismatch)
I20260709 13:04:32.209767  8955 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:32.310106  8733 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:32.310691  9236 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:32.310878  1585 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:32.320775  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.321466  1593 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:32.288723  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2702) took 2523 ms. Trace:
I20260709 13:04:32.323021  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:32.325568  1662 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:32.322378  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.325969  1736 rpcz_store.cc:276] 0709 13:04:29.377729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.607187 (+2229458us) service_pool.cc:224] Handling call
0709 13:04:31.607493 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.607917 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.607950 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:31.611401 (+  3451us) log.cc:844] Serialized 26 byte log entry
0709 13:04:31.704291 (+ 92890us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:31.704337 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:31.704406 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:32.273775 (+569369us) raft_consensus.cc:1685] finished
0709 13:04:32.273809 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.274004 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3399,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":92,"thread_start_us":92465,"threads_started":1,"wal-append.queue_time_us":265717}
W20260709 13:04:32.210783  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50390.5us
I20260709 13:04:32.330103  1659 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 13:04:32.262852  9240 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.137s
W20260709 13:04:32.327980  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 79689.9us
I20260709 13:04:32.332237  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:32.266188  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.332787  1661 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
W20260709 13:04:32.333628  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:32.245136  9193 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.841s	user 0.006s	sys 0.009s
I20260709 13:04:32.238869  9021 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:32.233062  8792 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 13:04:32.335832  1664 pending_rounds.cc:85] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:04:32.335423  1668 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced"
dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:52744
I20260709 13:04:32.468940  1664 pending_rounds.cc:107] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 22.11
I20260709 13:04:32.469169  1668 raft_consensus.cc:606] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:32.469429  1664 raft_consensus.cc:2889] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:32.404352  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.325851  9186 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:32.327972  9079 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:32.454358  1585 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:32.239234  9227 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.209873  9158 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:32.414621  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3560) took 2224 ms. Trace:
I20260709 13:04:32.478298  1584 rpcz_store.cc:276] 0709 13:04:30.189881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.150159 (+960278us) service_pool.cc:224] Handling call
0709 13:04:31.150523 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.253187 (+102664us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:31.320132 (+ 66945us) log.cc:844] Serialized 14 byte log entry
0709 13:04:31.329764 (+  9632us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:31.329834 (+    70us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:31.329884 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.414312 (+1084428us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400431a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:32.414368 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.414548 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8437,"prepare.queue_time_us":29289,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":170,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":235444608,"thread_start_us":57526,"threads_started":2,"wal-append.queue_time_us":9381,"wal-append.run_cpu_time_us":2470,"wal-append.run_wall_time_us":1136156}
I20260709 13:04:32.455721  1738 rpcz_store.cc:276] 0709 13:04:29.765415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.732545 (+1967130us) service_pool.cc:224] Handling call
0709 13:04:31.732761 (+   216us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.733147 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:31.933342 (+200195us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.217529 (+284187us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:32.232068 (+ 14539us) log.cc:844] Serialized 26 byte log entry
0709 13:04:32.232255 (+   187us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:32.232494 (+   239us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.241358 (+  8864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.241447 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:32.288373 (+ 46926us) raft_consensus.cc:1685] finished
0709 13:04:32.288402 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.288614 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35662,"prepare.queue_time_us":328924,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":100,"raft.queue_time_us":165589,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":241,"thread_start_us":393644,"threads_started":2,"wal-append.queue_time_us":281055}
I20260709 13:04:32.464147  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2704) took 2637 ms. Trace:
I20260709 13:04:32.480657  1742 rpcz_store.cc:276] 0709 13:04:29.826958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.757545 (+1930587us) service_pool.cc:224] Handling call
0709 13:04:31.757919 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:31.801843 (+ 43924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:31.801875 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:31.838066 (+ 36191us) log.cc:844] Serialized 26 byte log entry
0709 13:04:31.990067 (+152001us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:32.129649 (+139582us) log.cc:844] Serialized 14 byte log entry
0709 13:04:32.406653 (+277004us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:04:32.429805 (+ 23152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:32.429938 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:32.430014 (+    76us) raft_consensus.cc:1685] finished
0709 13:04:32.463953 (+ 33939us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:32.464093 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43446,"prepare.queue_time_us":183786,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":142494,"raft.run_cpu_time_us":1577,"raft.run_wall_time_us":187123,"spinlock_wait_cycles":3378048,"thread_start_us":502679,"threads_started":3,"wal-append.queue_time_us":175736}
W20260709 13:04:32.430299  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 201076us
I20260709 13:04:32.469630  1668 raft_consensus.cc:614] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 13:04:32.435086  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98130.6us
W20260709 13:04:32.472451  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.475935  9079 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:32.462270  1660 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
W20260709 13:04:32.476395  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 410271us
I20260709 13:04:32.384298  9243 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.473827  1585 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.501556  1585 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:32.473807  9251 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.497819  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.468335  9240 consensus_queue.cc:1243] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:32.397102  9250 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.090s
I20260709 13:04:32.337947  9128 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.502779  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.464506  9021 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:32.390749  8677 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.106s
I20260709 13:04:32.509192  9243 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.518610  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:32.508646  9190 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.018s	sys 0.015s
I20260709 13:04:32.525681  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.467729  9207 consensus_queue.cc:1059] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 13:04:32.497702  9079 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:32.504309  8765 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.535898  1585 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 13:04:32.537388  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:32.534267  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116639us
I20260709 13:04:32.475756  9225 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 13:04:32.418368  9123 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.576119  1585 pending_rounds.cc:85] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 13
I20260709 13:04:32.578220  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:32.589156  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 63541.1us
W20260709 13:04:32.574479  9190 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.947s	user 0.022s	sys 0.019s
I20260709 13:04:32.503489  9186 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:04:32.531957  9240 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:32.569095  9243 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:32.575291  9247 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.593146  1585 pending_rounds.cc:107] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 13:04:32.536053  9021 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:32.531806  1632 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.502002  9239 consensus_queue.cc:1059] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.089s
I20260709 13:04:32.538530  9236 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.591691  9032 raft_consensus.cc:993] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: : Instructing follower df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:04:32.592612  1556 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.531057  9253 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:32.614571  1585 raft_consensus.cc:2889] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:32.597855  9246 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:32.514351  9248 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 13:04:32.536130  9254 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.104s
W20260709 13:04:32.635229  9252 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 13:04:32.635879  9254 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:32.636425  9254 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.605181  9227 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 13:04:32.595712  8792 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 13:04:32.567019  8861 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
W20260709 13:04:32.597994  9257 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.088s	user 0.004s	sys 0.000s
I20260709 13:04:32.593725  9244 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:32.571658  9226 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.067s
W20260709 13:04:32.621729  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111433us
W20260709 13:04:32.626161  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48758.9us
I20260709 13:04:32.656886  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:32.657764  9260 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.658576  9164 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:32.659053  9164 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:32.659863  9164 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.628882  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:32.596781  8733 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.596187  9259 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.027s
W20260709 13:04:32.618196  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 86436.9us
I20260709 13:04:32.596913  9167 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:32.638536  9207 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:32.606014  8955 consensus_queue.cc:1059] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 13:04:32.645768  9254 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.625613  9243 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 13:04:32.658301  9260 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.661154  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:32.694509  1738 raft_consensus.cc:2376] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19. Re-sending same reply.
I20260709 13:04:32.694680  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:32.515995  9251 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:32.684686  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55908.5us
I20260709 13:04:32.684645  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:32.661432  9164 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:32.701135  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 166885us
I20260709 13:04:32.631958  9239 consensus_queue.cc:1243] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 21.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:32.690321  2309 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 13:04:32.646365  9068 raft_consensus.cc:1081] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Signalling peer df6a18af06d543b2931837f5e6bab95f to start an election
I20260709 13:04:32.683331  1585 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:32.697840  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:32.704787  9264 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:32.705389  9264 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.707703  9264 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:32.708230  9264 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:32.708586  9264 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:32.708961  9264 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:32.661343  9249 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:32.630702  9255 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:32.710469  9255 raft_consensus.cc:697] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.711614  9255 consensus_queue.cc:237] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.716207  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.717141  9250 raft_consensus.cc:2764] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:04:32.722630  8731 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:32.723647  9250 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:32.724146  9250 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.724201  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:32.725490  9250 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.727274  9250 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:32.727638  8731 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:32.728243  8731 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.729166  8731 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.735913  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:32.689558  9189 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:32.747577  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:32.747577  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:32.749536  8910 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:32.750465  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.752013  1556 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:32.748869  2312 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:32.755955  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.756640  1664 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 13:04:32.758661  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:32.759963  1665 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:32.761162  1663 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:32.624097  9247 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.761571  1663 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:32.770107  2310 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:32.635874  8765 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.788121  1661 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 13:04:32.790714  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:32.800191  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 200151us
I20260709 13:04:32.769181  9167 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:32.802439  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.768201  9249 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:32.744093  9207 consensus_queue.cc:1059] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 13:04:32.769095  9248 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:32.769354  9189 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:32.779078  9190 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.814193  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:32.815994  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.799098  9254 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:32.798259  8910 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:32.818934  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83097us
I20260709 13:04:32.803064  9128 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:32.640769  1483 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 9 to 16, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 16 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:32.831264  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.832973  9244 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:32.833544  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.813741  9248 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:32.836894  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93565.6us
I20260709 13:04:32.834472  9244 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.819720  8791 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:32.839341  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:32.776796  9264 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:32.820175  9267 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:32.820287  9032 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 13:04:32.837646  9248 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:32.891145  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 57668.7us
I20260709 13:04:32.897003  1659 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:32.901819  1662 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:32.898074  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:32.898414  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:32.899137  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:32.912547  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.826905  8910 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:32.914264  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 98840us
I20260709 13:04:32.871775  9267 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:32.924750  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:32.870218  9236 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:32.928323  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 256877us
W20260709 13:04:32.928970  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81019.5us
I20260709 13:04:32.932745  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:32.969051  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:32.973002  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 60581.2us
I20260709 13:04:32.974113  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:32.991252  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:32.991115  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 151851us
W20260709 13:04:32.993445  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60799.3us
I20260709 13:04:32.992098  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:32.990309  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:32.923545  9253 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:32.926543  9207 consensus_queue.cc:1243] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 23.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:32.845793  9190 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.084s
I20260709 13:04:33.060899  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1209) took 1609 ms. Trace:
I20260709 13:04:32.930764  9267 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:33.059587  1664 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:32.914444  9255 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.134s
I20260709 13:04:33.061339  1589 rpcz_store.cc:276] 0709 13:04:31.451864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.619180 (+167316us) service_pool.cc:224] Handling call
0709 13:04:31.619463 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.716788 (+ 97325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.060623 (+1343835us) spinlock_profiling.cc:243] Waited 96.7 ms on lock 0x7b540031d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:33.060834 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222331776}
I20260709 13:04:33.065745  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.070647  1585 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 13:04:32.868842  9189 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:32.924093  9249 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.092353  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:33.097124  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 123079us
I20260709 13:04:33.098408  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:33.112473  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 117220us
I20260709 13:04:32.971968  9236 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.121568  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:33.090646  8731 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:32.974867  9266 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.149s
I20260709 13:04:33.088896  9260 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.128598  2309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:33.024209  9272 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.153s
I20260709 13:04:33.062613  9274 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:33.133577  2309 raft_consensus.cc:606] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:33.137671  2309 raft_consensus.cc:614] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:33.140156  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:33.142604  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44284.3us
I20260709 13:04:33.142649  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:33.143846  2309 raft_consensus.cc:2410] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 13:04:33.143707  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.125698  9255 consensus_queue.cc:1243] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 13:04:33.145988  9265 log.cc:927] Time spent T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.312s	user 0.000s	sys 0.001s
I20260709 13:04:33.130800  9262 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:33.147254  9268 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 13:04:33.147794  9276 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:33.148411  9276 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.150854  9245 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:33.151017  9276 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:33.151888  9245 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:33.152334  9262 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:33.153473  9262 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.131888  9188 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:33.153776  1483 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:33.158823  9188 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:33.159464  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167413us
I20260709 13:04:33.159859  9188 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.160414  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:33.161264  9236 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.154168  9283 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:33.096274  9186 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 13:04:33.129702  9251 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.192871  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.125837  9250 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 13:04:33.192505  9274 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:33.205405  9274 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.207430  9272 consensus_queue.cc:1243] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:33.162381  9283 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:33.189299  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.163985  9260 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 13:04:33.162835  9261 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.200739  8517 raft_consensus.cc:3055] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:33.201144  9269 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:33.137810  9247 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.159893  1483 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 14, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:04:33.203372  9271 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 13:04:33.164469  9244 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:33.209857  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 46134.8us
I20260709 13:04:33.212805  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1094) took 1537 ms. Trace:
I20260709 13:04:33.213647  1588 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:33.230700  1588 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:33.126976  9277 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
W20260709 13:04:33.221921  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 231676us
I20260709 13:04:33.226302  1584 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 8. Preceding OpId from leader: term: 20 index: 9. (index mismatch)
I20260709 13:04:33.181197  8731 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:04:33.229691  2318 rpcz_store.cc:276] 0709 13:04:31.675359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:31.676073 (+   714us) service_pool.cc:224] Handling call
0709 13:04:31.701671 (+ 25598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:31.911750 (+210079us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.212547 (+1300797us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400572e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:33.212741 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476925824}
I20260709 13:04:33.215170  8657 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:33.239464  2312 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
mode: GRACEFUL
new_leader_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:59800
I20260709 13:04:33.240289  2312 raft_consensus.cc:606] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Received request to transfer leadership to TS df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:33.240727  2312 raft_consensus.cc:614] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 13:04:33.209936  9289 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:33.211411  9283 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:33.219286  8517 raft_consensus.cc:740] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:33.195256  8801 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 13:04:33.217430  9261 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.255668  9291 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:33.256191  9291 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:33.243398  9190 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:33.257081  9291 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.257793  9190 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:33.258479  9291 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:33.258937  9190 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.239548  8657 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:33.260893  9190 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:33.262072  8517 consensus_queue.cc:260] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.262279  9186 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:33.263180  8517 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:33.264444  9186 consensus_queue.cc:1048] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:04:33.266454  9021 consensus_meta.cc:287] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.005s
I20260709 13:04:33.266901  9021 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.236672  9247 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:33.270025  9247 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.272766  9247 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:33.273192  9244 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 13:04:33.273808  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.275367  9254 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:33.269248  9021 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.275631  8910 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.323916  1584 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:33.277658  9286 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:33.280138  1483 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 4 to 12, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 12 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:33.278996  9224 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:33.245110  8953 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:33.227506  9269 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.284058  9267 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.243579  9263 consensus_queue.cc:1048] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 13:04:33.223284  9264 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:33.294184  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.295616  2310 raft_consensus.cc:1240] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:33.278131  9288 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:33.302433  8657 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.305971  1588 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.309926  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:33.324940  9286 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:33.278033  9164 consensus_meta.cc:287] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.532s	user 0.012s	sys 0.001s
W20260709 13:04:33.330061  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 186388us
I20260709 13:04:33.393038  1705 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:33.333606  9224 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:33.427065  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.378319  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 249404us
I20260709 13:04:33.433777  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:33.437882  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.425904  8657 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:33.413833  9106 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.443955  9300 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:33.444619  9300 raft_consensus.cc:697] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:33.445885  9300 consensus_queue.cc:237] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:33.449424  9090 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
W20260709 13:04:33.454035  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.454578  9269 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.456235  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.457147  1587 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:33.458381  1660 raft_consensus.cc:1240] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25. Ops: [24.12-24.12]
I20260709 13:04:33.459707  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.461962  9224 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.462868  1584 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:04:33.463256  9224 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:33.466337  9246 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:04:33.466682  9267 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.466806  9246 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:33.467737  9246 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.468896  9246 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:33.432395  9266 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.469837  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.470309  9266 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.471272  9267 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:33.471545  1662 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:33.472059  1662 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:33.472833  9266 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.473440  9158 raft_consensus.cc:3055] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:33.473894  9158 raft_consensus.cc:740] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:33.473807  9106 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.475037  9158 consensus_queue.cc:260] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.476145  9158 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:33.476410  9106 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.478116  9283 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.478520  9264 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.479202  9264 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.480341  9283 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.480435  1592 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:33.439096  9257 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 13:04:33.481877  9264 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.433408  9164 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.482942  9164 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.484925  9164 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.485229  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.414062  9284 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:33.486066  9284 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.488207  9284 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.489194  9267 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:33.440057  9288 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 13:04:33.492967  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.493130  1663 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.494557  1663 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:33.495204  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.501816  1592 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:33.502800  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3574) took 1416 ms. Trace:
I20260709 13:04:33.503249  1592 rpcz_store.cc:276] 0709 13:04:32.086610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.121700 (+ 35090us) service_pool.cc:224] Handling call
0709 13:04:32.122117 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:32.307967 (+185850us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b58005d3bd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:04:32.660620 (+352653us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54002b1220. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52a1d 00007fcdedf391a4 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:04:33.480076 (+819456us) spinlock_profiling.cc:243] Waited 83.4 ms on lock 0x7b54002b1220. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52a1d 00007fcdedf391a4 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419
0709 13:04:33.502438 (+ 22362us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.502719 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":619830656}
I20260709 13:04:33.506494  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:33.508430  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 262889us
I20260709 13:04:33.372385  8953 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.510803  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.472874  1662 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.514262  9267 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:33.514930  9290 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:33.515214  8635 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:33.515307  9298 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.076s
I20260709 13:04:33.515735  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.516398  9264 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 13:04:33.516664  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.518419  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.523253  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.527199  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.530059  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:33.573618  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:33.536758  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.574862  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.562296  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:33.576050  2309 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:33.563974  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:33.572727  1590 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:33.442384  9244 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:33.566821  1662 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:33.582353  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 59180.5us
I20260709 13:04:33.583257  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:33.441461  9247 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:33.586298  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.010s	sys 0.007s
W20260709 13:04:33.586766  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.010s	sys 0.007s
W20260709 13:04:33.587314  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.578948  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:33.568711  1664 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:33.469885  9286 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.528429  9253 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:33.571861  9257 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 13:04:33.592301  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:33.668327  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 76134.7us
W20260709 13:04:33.668797  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 69708.8us
W20260709 13:04:33.569916  9248 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.642s	user 0.010s	sys 0.001s
W20260709 13:04:33.593432  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.000s	sys 0.019s
W20260709 13:04:33.669981  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.000s	sys 0.019s
I20260709 13:04:33.669796  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:33.670419  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 745551us
I20260709 13:04:33.603873  1590 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:33.676661  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.718735  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.582221  9244 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:33.521845  8798 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
W20260709 13:04:33.742197  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.740957  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 140293us
W20260709 13:04:33.765744  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.766894  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 95398.4us
W20260709 13:04:33.774997  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 105285us
I20260709 13:04:33.777237  1662 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:33.779862  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.780556  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.781062  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:33.786271  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.667845  9290 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.602401  9267 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:33.597013  9295 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:33.684058  9236 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:33.766973  9244 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:33.798491  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.798200  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:33.807261  1663 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:33.764621  9282 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:33.810541  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2687) took 1051 ms. Trace:
I20260709 13:04:33.790033  9247 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:33.816349  1663 rpcz_store.cc:276] 0709 13:04:32.759625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.760783 (+  1158us) service_pool.cc:224] Handling call
0709 13:04:32.761046 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.492741 (+731695us) spinlock_profiling.cc:243] Waited 21.2 ms on lock 0x7b4c00164c80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:04:33.809917 (+317176us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.810481 (+   564us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48861952}
I20260709 13:04:33.739791  9248 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:33.827708  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:33.814011  9079 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.224s	user 0.016s	sys 0.000s
I20260709 13:04:33.840705  9079 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.841799  9248 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:33.826921  9266 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.843751  1667 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:33.864202  9266 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.810961  8792 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.808707  9236 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:33.850353  9309 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 13:04:33.851001  9106 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.851292  9251 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:33.849997  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:33.852012  8953 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.852186  1557 leader_election.cc:400] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:33.883746  1557 leader_election.cc:403] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:33.883890  1661 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 13:04:33.787321  9280 consensus_queue.cc:1059] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 13:04:33.868656  1593 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 13:04:33.847532  9079 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.874228  1591 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:04:33.886929  1591 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:33.874495  9176 raft_consensus.cc:3055] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Stepping down as leader of term 24
W20260709 13:04:33.857391  9097 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
W20260709 13:04:33.856740  9225 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
I20260709 13:04:33.897397  9284 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.884316  1661 pending_rounds.cc:85] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 11
I20260709 13:04:33.881302  9106 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.801488  9290 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.898005  1661 pending_rounds.cc:107] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 13:04:33.897958  9284 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:33.889184  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:33.833288  9247 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:33.899904  9290 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.900307  9284 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.899974  9247 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.894569  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:33.901713  1738 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 13:04:33.902889  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:33.903550  1736 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:33.894970  1591 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.904742  1591 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:33.891492  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:33.896539  9300 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.906795  9300 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.906400  1483 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 5 to 9, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 9 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:33.896927  8733 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:33.909237  1483 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:33.896811  9251 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.912189  9251 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.897224  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:33.914108  9305 raft_consensus.cc:3055] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:33.909843  8733 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.889583  9159 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:33.915551  9159 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:33.916004  9248 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:33.916489  8733 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:33.816381  9282 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:33.892083  9176 raft_consensus.cc:740] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:33.917212  9282 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:33.917965  9238 log.cc:927] Time spent T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.739s	user 0.015s	sys 0.000s
I20260709 13:04:33.918047  9176 consensus_queue.cc:260] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:33.919154  9176 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:33.893847  9267 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 13:04:33.924273  9236 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:33.924552  9106 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:33.898492  1661 raft_consensus.cc:2889] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:33.914556  9305 raft_consensus.cc:740] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:33.930595  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.930826  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.930625  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.930898  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.933768  1585 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:33.933768  1592 raft_consensus.cc:2410] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 13:04:33.787395  9245 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 13:04:33.884526  1663 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:33.925153  9106 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:33.910777  1483 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 6 to 13. New cstate: current_term: 13 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:33.971122  1737 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:33.972952  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.984669  1660 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:33.984853  1740 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:33.974536  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:33.975952  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:33.986452  1739 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:33.980890  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:33.982201  1734 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:33.994304  1733 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 13:04:34.020758  1736 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:33.880218  8792 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:34.029878  1592 raft_consensus.cc:2393] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:33.975950  1591 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 13:04:34.036119  1591 pending_rounds.cc:85] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 13
I20260709 13:04:34.036721  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:34.047638  1738 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:34.041139  1737 raft_consensus.cc:1240] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 13:04:34.034029  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:34.045758  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.046046  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.050624  1591 pending_rounds.cc:107] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 13:04:34.051388  1591 raft_consensus.cc:2889] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:34.050145  1734 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:33.980005  9159 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:34.043108  9158 consensus_meta.cc:287] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.567s	user 0.005s	sys 0.010s
I20260709 13:04:34.059549  1737 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [24.13-25.14]
W20260709 13:04:34.058678  9179 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 13:04:34.042228  9300 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:34.068882  1733 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 13:04:34.068203  1738 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:34.073289  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.070920  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3566) took 2019 ms. Trace:
I20260709 13:04:34.075220  1733 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 13:04:34.076813  1734 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:34.077409  1591 rpcz_store.cc:276] 0709 13:04:32.051193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.051396 (+   203us) service_pool.cc:224] Handling call
0709 13:04:32.051708 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.874105 (+1822397us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54002b5d20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfdf1c0 00007fcdedfe0231 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c
0709 13:04:34.053318 (+179213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.070861 (+ 17543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281378688}
I20260709 13:04:34.078229  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.078903  1737 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:34.082055  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.082815  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.082834  1738 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:34.086870  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.087571  1737 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:34.088829  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.089012  1663 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:34.089562  1738 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:34.093225  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.112442  9281 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:34.113067  9281 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:34.114037  9281 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.115336  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.115237  9281 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:34.117527  9310 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 13:04:34.122622  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:34.089761  9278 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.943s	user 0.008s	sys 0.003s
I20260709 13:04:34.107818  1737 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:34.133069  1737 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [19.10-20.11]
I20260709 13:04:34.133478  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:34.133911  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.134193  1663 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:34.135211  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:34.135934  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:34.139019  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:34.146917  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:34.147588  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:34.148321  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:34.081564  9318 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.151811  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:34.153594  9090 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:34.155146  9090 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
W20260709 13:04:34.155280  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.155678  9090 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:34.156653  9090 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.158556  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2685) took 1400 ms. Trace:
I20260709 13:04:34.159129  1666 rpcz_store.cc:276] 0709 13:04:32.757720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.757868 (+   148us) service_pool.cc:224] Handling call
0709 13:04:32.758282 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:33.199366 (+441084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:33.199455 (+    89us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:33.222379 (+ 22924us) log.cc:844] Serialized 26 byte log entry
0709 13:04:33.761986 (+539607us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:33.762032 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.762110 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:34.158240 (+396130us) raft_consensus.cc:1685] finished
0709 13:04:34.158271 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:34.158491 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8666,"raft.queue_time_us":50299,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":5965,"thread_start_us":870044,"threads_started":2,"wal-append.queue_time_us":820580}
I20260709 13:04:34.160756  9318 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.161952  1736 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:34.163103  9318 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:34.164268  1661 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:34.173300  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.145080  9158 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:34.072732  9305 consensus_queue.cc:260] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.186268  1740 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:34.177083  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.142578  9248 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 13:04:34.180359  9316 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:04:34.181170  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:34.188191  1589 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:34.142272  9167 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 13:04:34.182799  1588 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:34.150291  9280 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 13:04:34.177979  9243 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.184258  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.185367  1738 raft_consensus.cc:1275] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:34.188295  1735 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:34.150189  9251 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 13:04:34.207086  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.227552  1735 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:34.231048  1588 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:34.199937  1739 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:34.165489  9106 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:34.207450  1736 raft_consensus.cc:2393] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 15: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:34.285115  9189 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:34.322425  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.322665  9189 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 13:04:34.323164  1736 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:34.324131  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:34.207834  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:34.214236  9243 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.148468  9284 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 13:04:34.215267  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:34.191882  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.239209  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3575) took 2132 ms. Trace:
I20260709 13:04:34.369472  2281 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.369657  1738 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:34.369789  1588 rpcz_store.cc:276] 0709 13:04:32.107068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.131209 (+ 24141us) service_pool.cc:224] Handling call
0709 13:04:32.131477 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.213523 (+1082046us) spinlock_profiling.cc:243] Waited 1.98 ms on lock 0x7b54002b4420. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfdf1c0 00007fcdedfe0231 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c
0709 13:04:33.305534 (+ 92011us) spinlock_profiling.cc:243] Waited 3.23 ms on lock 0x7b4c00479b40. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:04:34.182552 (+877018us) spinlock_profiling.cc:243] Waited 679 ms on lock 0x7b58005a5cd0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf390aa 00007fcdedfbb447 00007fcdedfd3fe7 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416 00007fcdedfdcd8f 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4
0709 13:04:34.238852 (+ 56300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.239137 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1572854016}
I20260709 13:04:34.229048  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:34.298965  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.203809  9313 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 13:04:34.198467  9318 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:34.342676  1736 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:34.341727  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:34.342391  1741 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:34.341948  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:34.396157  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:34.371273  1735 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
W20260709 13:04:34.373673  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 181810us
W20260709 13:04:34.323648  8855 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
I20260709 13:04:34.178740  9158 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:34.366359  9244 consensus_meta.cc:287] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.541s	user 0.011s	sys 0.000s
I20260709 13:04:34.429860  1738 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 26.
I20260709 13:04:34.201018  9167 consensus_queue.cc:1243] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:04:34.400646  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.402591  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.402812  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:34.323360  9189 consensus_queue.cc:1243] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 13:04:34.367971  9097 consensus_queue.cc:1048] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 13:04:34.395972  1739 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:34.390734  9127 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:34.439700  1735 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:34.320966  9106 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:34.440961  1736 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:34.398084  9305 raft_consensus.cc:3060] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:34.401799  9224 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.938s	user 0.005s	sys 0.010s
I20260709 13:04:34.342131  8792 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:34.341358  9159 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.434301  9225 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:34.431042  9266 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:34.397722  9300 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:34.443423  9127 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.598176  9127 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:34.599922  9320 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 13:04:34.448915  9258 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:34.601845  9258 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:34.602440  9258 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:34.447150  9236 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:34.451153  9106 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:34.609069  9243 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:34.585335  9159 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.456346  9261 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:34.593142  9225 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:34.618014  9225 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:34.443933  9244 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:34.470992  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.399708  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 251402us
W20260709 13:04:34.733242  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.733736  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.399053  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 193599us
I20260709 13:04:34.735505  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3583) took 1277 ms. Trace:
W20260709 13:04:34.406890  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65258.9us
I20260709 13:04:34.775290  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2694) took 1560 ms. Trace:
I20260709 13:04:34.875218  1659 rpcz_store.cc:276] 0709 13:04:33.214981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:33.215104 (+   123us) service_pool.cc:224] Handling call
0709 13:04:33.217768 (+  2664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:33.415527 (+197759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:33.415577 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:33.415646 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:33.415705 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.774479 (+1358774us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54000a9bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:34.774568 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:34.774930 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":453967744}
W20260709 13:04:34.782862  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2681) took 2459 ms (client timeout 2017 ms). Trace:
W20260709 13:04:34.877679  1667 rpcz_store.cc:269] 0709 13:04:32.322887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:32.331994 (+  9107us) service_pool.cc:224] Handling call
0709 13:04:34.782205 (+2450211us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540013e7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:34.782545 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3201715584}
W20260709 13:04:34.445961  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45432.3us
I20260709 13:04:34.803042  1734 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:34.448534  1735 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 13:04:34.880887  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:34.823004  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:34.450171  2281 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:34.496214  8955 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.446254  9158 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:35.012365  1584 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:34.428062  8798 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.616611  9159 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:34.467480  9239 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.175s
I20260709 13:04:34.629324  9244 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:34.829245  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:35.016671  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.646447  9246 consensus_meta.cc:287] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.177s	user 0.014s	sys 0.002s
I20260709 13:04:35.020785  9128 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.021884  9128 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.675611  9258 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.632303  9261 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:34.678017  9262 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.519s	user 0.008s	sys 0.019s
W20260709 13:04:34.656215  9282 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.738s	user 0.013s	sys 0.000s
W20260709 13:04:35.074138  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 193708us
I20260709 13:04:35.076064  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:34.673460  9314 log.cc:927] Time spent T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.545s	user 0.002s	sys 0.002s
W20260709 13:04:34.621922  9322 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.192s	user 0.001s	sys 0.001s
I20260709 13:04:35.097153  9148 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:35.098237  9148 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:35.098273  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.098873  9148 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:34.864078  1706 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:34.865402  9335 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.866396  9303 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.883455  9284 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.884336  9164 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.885057  9179 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.885911  8855 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.456260  1738 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:34.891454  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:34.892707  9324 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:34.899045  9323 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 13:04:34.918566  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:34.948637  9283 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:34.950254  8731 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.951848  9079 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:04:34.956951  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81896.5us
W20260709 13:04:34.994570  1660 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.000s	sys 0.012s
W20260709 13:04:34.995043  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:34.995348  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45116.4us
I20260709 13:04:34.995263  8955 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:34.998061  9295 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:34.997710  9302 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 13:04:35.002650  9269 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.038021  1742 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 26.
I20260709 13:04:35.044986   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 13:04:35.042654  1734 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:04:35.044406  8798 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.052804  9207 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.052956  8792 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:34.855747  1587 rpcz_store.cc:276] 0709 13:04:33.458289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:33.459630 (+  1341us) service_pool.cc:224] Handling call
0709 13:04:33.459972 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:33.460330 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:33.460367 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:33.460884 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 13:04:33.762350 (+301466us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:33.762392 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:33.762466 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:34.734680 (+972214us) raft_consensus.cc:1685] finished
0709 13:04:34.734764 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:34.735213 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83,"raft.queue_time_us":1233,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":177,"thread_start_us":301026,"threads_started":1,"wal-append.queue_time_us":420985}
I20260709 13:04:35.058493  9250 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:35.148630  9250 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.059330  9261 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:35.151244  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 260145us
I20260709 13:04:35.151342  8955 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:35.152092  8955 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.152302  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.153476  9261 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.153796  9283 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:35.094534  9262 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.935s	user 0.009s	sys 0.020s
W20260709 13:04:35.150548  1741 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.755s	user 0.006s	sys 0.015s
I20260709 13:04:35.156946  8731 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:35.096274  9190 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.795s	user 0.013s	sys 0.010s
I20260709 13:04:35.158700  1741 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:35.158804  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:35.159777  8731 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.160519  8731 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:35.161923  9090 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:35.162452  9090 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.162909  9090 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:35.154261  9261 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.169965  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.170895  1665 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:35.171170  9324 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:35.171602  9324 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:35.108851  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 89750.8us
I20260709 13:04:35.171989  9324 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.074149  9282 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.173624  1734 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:04:35.174090  9272 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:35.175418  1734 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:35.176033  9282 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.095429  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:35.113830  8855 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.114089  9335 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.112238  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.140643  1585 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:35.113343  9179 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.152884  8955 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.145406  9344 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:35.146767  9269 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.156409  9283 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.157413  9262 raft_consensus.cc:2804] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 13:04:35.167068  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 303622us
I20260709 13:04:35.112154  9164 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.113240  9128 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.161098  1632 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:35.177912  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:35.145123  9284 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.167961  9261 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.198994  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3383) took 1054 ms. Trace:
I20260709 13:04:35.200196  9335 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.200651  8855 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.138784  9303 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.201593  1662 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:04:35.196501  1667 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
W20260709 13:04:35.208070  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 112871us
I20260709 13:04:35.222283  2310 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 13:04:35.209956  1668 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.151356  9250 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.200686  9318 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:35.246058  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68300.3us
I20260709 13:04:35.252823   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:04:35.254973  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.253793  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3513) took 1064 ms. Trace:
I20260709 13:04:35.179358  9207 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.266050  1738 rpcz_store.cc:276] 0709 13:04:34.189366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.443588 (+254222us) service_pool.cc:224] Handling call
0709 13:04:34.455516 (+ 11928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.223378 (+767862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.253574 (+ 30196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.225700  1660 rpcz_store.cc:276] 0709 13:04:34.139015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.139156 (+   141us) service_pool.cc:224] Handling call
0709 13:04:34.150150 (+ 10994us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:34.150555 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:34.398851 (+248296us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.153624 (+754773us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:35.154447 (+   823us) log.cc:844] Serialized 26 byte log entry
0709 13:04:35.154654 (+   207us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:35.154977 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.155240 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.155434 (+   194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.178709 (+ 23275us) raft_consensus.cc:1685] finished
0709 13:04:35.178729 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.193642 (+ 14913us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":472243,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":5934,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":253,"thread_start_us":977545,"threads_started":2,"wal-append.queue_time_us":577519}
W20260709 13:04:35.285163  9356 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 13:04:35.255355  9262 raft_consensus.cc:697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:35.256219  9335 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:35.287257  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 126315us
I20260709 13:04:35.277424  8731 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.288060  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3528) took 1090 ms. Trace:
I20260709 13:04:35.287405  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.292029  9128 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:35.292238  9258 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:35.294082  9349 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:35.319834  9250 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:35.295948  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3529) took 1098 ms. Trace:
I20260709 13:04:35.322458  1738 rpcz_store.cc:276] 0709 13:04:34.197441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.294583 (+1097142us) service_pool.cc:224] Handling call
0709 13:04:35.295100 (+   517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.295545 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.295570 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.295635 (+    65us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:35.295678 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.295741 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.295877 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:35.297742  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 108142us
W20260709 13:04:35.299536  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.293695  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:35.308018  1734 rpcz_store.cc:276] 0709 13:04:34.197177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.178646 (+981469us) service_pool.cc:224] Handling call
0709 13:04:35.254804 (+ 76158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.287620 (+ 32816us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.287664 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.287712 (+    48us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:35.287757 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.287819 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.287991 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:35.335142  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 56872.8us
I20260709 13:04:35.323815  1738 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "219129ac643e4650b611a85f565f5990"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
 from {username='slave'} at 127.0.0.1:42028
W20260709 13:04:35.313606  9127 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.715s	user 0.017s	sys 0.008s
W20260709 13:04:35.289000  9225 consensus_meta.cc:287] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.009s
I20260709 13:04:35.321043  9227 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 13:04:35.374254  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3514) took 1184 ms. Trace:
W20260709 13:04:35.311774  9106 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.705s	user 0.012s	sys 0.003s
I20260709 13:04:35.374945  1739 rpcz_store.cc:276] 0709 13:04:34.189990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.438239 (+248249us) service_pool.cc:224] Handling call
0709 13:04:34.438650 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:34.439104 (+   454us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:35.089761 (+650657us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.282539 (+192778us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:35.283405 (+   866us) log.cc:844] Serialized 26 byte log entry
0709 13:04:35.283624 (+   219us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:35.283656 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.283751 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.373801 (+ 90050us) raft_consensus.cc:1685] finished
0709 13:04:35.373831 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.374010 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":386462,"prepare.queue_time_us":441058,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":86,"raft.queue_time_us":245,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":6716,"thread_start_us":345562,"threads_started":2,"wal-append.queue_time_us":260059}
I20260709 13:04:35.376214  9106 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.377646  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.378782  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1306) took 1168 ms. Trace:
I20260709 13:04:35.307727  9283 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:04:35.379257  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3521) took 1140 ms. Trace:
I20260709 13:04:35.307603  9207 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.285480  8798 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.298564  9179 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.379271  1739 rpcz_store.cc:276] 0709 13:04:34.210152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.377224 (+1167072us) service_pool.cc:224] Handling call
0709 13:04:35.377841 (+   617us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.378174 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.378204 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.378278 (+    74us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:35.378505 (+   227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.378593 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.378721 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.379817  1587 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:35.299253  9335 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:35.359046  9176 consensus_meta.cc:287] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 1.437s	user 0.022s	sys 0.003s
I20260709 13:04:35.295163  8955 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.254462  8792 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.381959  1739 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.382740  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1307) took 1172 ms. Trace:
I20260709 13:04:35.383100  1739 rpcz_store.cc:276] 0709 13:04:34.210454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.380900 (+1170446us) service_pool.cc:224] Handling call
0709 13:04:35.381641 (+   741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.382465 (+   824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.382684 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.383916  8792 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.299121  9164 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.362771  9261 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:35.364023  9337 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.238s	user 0.003s	sys 0.000s
I20260709 13:04:35.384963  1589 raft_consensus.cc:1275] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
W20260709 13:04:35.361079  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 90742.8us
I20260709 13:04:35.385306  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1309) took 1174 ms. Trace:
I20260709 13:04:35.385880  1739 rpcz_store.cc:276] 0709 13:04:34.211068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.384103 (+1173035us) service_pool.cc:224] Handling call
0709 13:04:35.384556 (+   453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.384919 (+   363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.384949 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.385022 (+    73us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:35.385069 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.385128 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.385245 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.386668  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 723) took 1614 ms. Trace:
I20260709 13:04:35.386919  9303 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.318090  9128 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.387672  9248 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:35.388191  9128 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:35.387115  2318 rpcz_store.cc:276] 0709 13:04:33.772378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:33.772846 (+   468us) service_pool.cc:224] Handling call
0709 13:04:33.773241 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:33.773700 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:33.773730 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:34.600830 (+827100us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b2c00070ac8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:34.601381 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 13:04:34.918639 (+317258us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:34.918722 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.918855 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.347367 (+428512us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:35.386209 (+ 38842us) raft_consensus.cc:1685] finished
0709 13:04:35.386235 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.386397 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":476,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":6187,"spinlock_wait_cycles":25949696,"thread_start_us":237068,"threads_started":1,"wal-append.queue_time_us":382053}
I20260709 13:04:35.309934  9288 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.307514  8731 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.317046  9291 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 13:04:35.286130  9284 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.367998  9225 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.368917  1734 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:35.370302  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3510) took 1182 ms. Trace:
I20260709 13:04:35.310884  9262 consensus_queue.cc:237] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.383446  9335 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.388584  9248 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.367652  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.358275  1738 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:04:35.413802  1738 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:35.414391  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1304) took 1195 ms. Trace:
I20260709 13:04:35.414804  1734 rpcz_store.cc:276] 0709 13:04:34.209488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.343079 (+1133591us) service_pool.cc:224] Handling call
0709 13:04:35.368604 (+ 25525us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.403610 (+ 35006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.403929 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.391813  8731 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.379808  1737 rpcz_store.cc:276] 0709 13:04:34.191832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.892083 (+700251us) service_pool.cc:224] Handling call
0709 13:04:34.892789 (+   706us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:34.933232 (+ 40443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:34.933271 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:34.999162 (+ 65891us) log.cc:844] Serialized 52 byte log entry
0709 13:04:35.040338 (+ 41176us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:35.071365 (+ 31027us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.071753 (+   388us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.071852 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.074475 (+  2623us) spinlock_profiling.cc:243] Waited 22.4 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:04:35.299490 (+225015us) raft_consensus.cc:1685] finished
0709 13:04:35.331786 (+ 32296us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.332041 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27946,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":10777,"raft.queue_time_us":9726,"raft.run_cpu_time_us":2013,"raft.run_wall_time_us":166088,"spinlock_wait_cycles":51504896,"thread_start_us":40515,"threads_started":1,"wal-append.queue_time_us":91216}
I20260709 13:04:35.392905  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:35.288834  9269 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.393880  9148 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.395857  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3522) took 1203 ms. Trace:
W20260709 13:04:35.418216  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61061.3us
I20260709 13:04:35.367091  9295 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:35.389640  9288 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.379048  9106 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:35.415946  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49076.5us
W20260709 13:04:35.437947  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 162066us
I20260709 13:04:35.440124  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1313) took 1124 ms. Trace:
I20260709 13:04:35.441010  9288 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.404258  1740 rpcz_store.cc:276] 0709 13:04:34.187317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.201382 (+ 14065us) service_pool.cc:224] Handling call
0709 13:04:34.201887 (+   505us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:34.202432 (+   545us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:34.822057 (+619625us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b1800347800. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdeaef2a40 00007fcdeaeed4e4 00007fcdeaeed455 00007fcdf19d53c7 00007fcdedfe79a7 00007fcdedff70bf 00007fcdedff7038
0709 13:04:34.977446 (+155389us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.189354 (+211908us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.192609 (+  3255us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:35.196132 (+  3523us) log.cc:844] Serialized 52 byte log entry
0709 13:04:35.196323 (+   191us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:35.310418 (+114095us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b1800347800. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:35.369434 (+ 59016us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.369795 (+   361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.369895 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.369949 (+    54us) raft_consensus.cc:1685] finished
0709 13:04:35.369959 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.370093 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":532402,"prepare.queue_time_us":901905,"prepare.run_cpu_time_us":157,"prepare.run_wall_time_us":161,"raft.queue_time_us":63246,"raft.run_cpu_time_us":660,"raft.run_wall_time_us":34570,"spinlock_wait_cycles":15369728,"thread_start_us":126417,"threads_started":2,"wal-append.queue_time_us":111884}
I20260709 13:04:35.426302  9291 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
W20260709 13:04:35.449684  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 82178.1us
I20260709 13:04:35.443197  9148 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.416421  9248 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.419247  9283 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:35.440755  1737 rpcz_store.cc:276] 0709 13:04:34.315759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.418382 (+1102623us) service_pool.cc:224] Handling call
0709 13:04:35.439165 (+ 20783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.439640 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.439683 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.439746 (+    63us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:35.439803 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.439877 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.440017 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.460208  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1312) took 1145 ms. Trace:
I20260709 13:04:35.460088  9283 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.460548  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.460734  1734 rpcz_store.cc:276] 0709 13:04:34.315337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.415608 (+1100271us) service_pool.cc:224] Handling call
0709 13:04:35.455292 (+ 39684us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:35.459795 (+  4503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.459828 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:35.459875 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:35.459922 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.459999 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.460143 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.461941  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.462343  9225 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.462357  9284 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:35.463016  9106 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:35.463158  9284 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.463505  9106 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.463604  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.464104  9106 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.464453  1593 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:35.438050  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:35.450457  9335 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:35.400681  9127 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.418570  1735 rpcz_store.cc:276] 0709 13:04:34.192088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.968668 (+776580us) service_pool.cc:224] Handling call
0709 13:04:35.058477 (+ 89809us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:35.059473 (+   996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.059510 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:04:35.060822 (+  1312us) log.cc:844] Serialized 78 byte log entry
0709 13:04:35.132086 (+ 71264us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:35.195201 (+ 63115us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.322945 (+127744us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b18003bce60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219
0709 13:04:35.357753 (+ 34808us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b18003bce60. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeae6ac0a 00007fcdf45aa219 00007fcdf1eb158f 00007fcdeaef0571 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd
0709 13:04:35.394250 (+ 36497us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.395246 (+   996us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.395356 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.395421 (+    65us) raft_consensus.cc:1685] finished
0709 13:04:35.395440 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.395634 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":206181,"prepare.queue_time_us":254004,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":94,"raft.queue_time_us":2781,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":6611,"spinlock_wait_cycles":29667456,"thread_start_us":70684,"threads_started":1,"wal-append.queue_time_us":316706}
I20260709 13:04:35.486198  1738 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.488538  1738 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.489042  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:42028 (request call id 1303) took 1290 ms. Trace:
W20260709 13:04:35.489945  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 983919us
I20260709 13:04:35.491402  2312 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:35.463723  9284 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.492151  9090 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.458097  9269 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:35.494177  9269 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.462577  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.489347  1738 rpcz_store.cc:276] 0709 13:04:34.208168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.323527 (+1115359us) service_pool.cc:224] Handling call
0709 13:04:35.488957 (+165430us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:35.505976  9363 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.507020  9363 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.507916  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.511318  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:35.512028  2315 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:35.512712  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 492772us
I20260709 13:04:35.516095  9363 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.516433  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1098) took 1994 ms. Trace:
I20260709 13:04:35.451278  9128 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.508227  1586 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:04:35.516950  2310 rpcz_store.cc:276] 0709 13:04:33.522118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:33.523225 (+  1107us) service_pool.cc:224] Handling call
0709 13:04:33.575081 (+ 51856us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:35.249549 (+1674468us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.516042 (+266493us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b5400724054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:35.516342 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3686495104}
W20260709 13:04:35.525956  9369 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.145s	user 0.002s	sys 0.000s
I20260709 13:04:35.525665  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:35.526129  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.526657  2314 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.530285  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.532444  9128 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.538053  9148 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.538654  2314 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:35.538777  9148 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.502194  9164 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:35.540738  9164 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:35.541129  9164 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:35.541993  9148 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.542975  9158 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.544301  1584 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:35.546237  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.547003  1584 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
I20260709 13:04:35.547448  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:35.548225  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.550518  9327 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:35.501569  9250 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:35.550938  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.552376  9106 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.554577  9335 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.554854  8791 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.552399  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.564276  8791 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.565833  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:35.567127  8791 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.567783  9148 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.556219  9335 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.568256  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.568562  9148 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.569407  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.571076  1591 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 13:04:35.573096  1738 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20260709 13:04:35.573582  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.574189  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.574904  2317 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:35.581765  9106 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.581933  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:35.550971  9327 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:35.554931  9318 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:35.564831  9352 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.470s	user 0.000s	sys 0.003s
I20260709 13:04:35.577162  9158 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.596890  2318 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 13:04:35.599435  2318 pending_rounds.cc:85] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Aborting all ops after (but not including) 12
I20260709 13:04:35.599917  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.599761  2318 pending_rounds.cc:107] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 13:04:35.602200  2318 raft_consensus.cc:2889] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:35.602461  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80685.1us
I20260709 13:04:35.609418  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.590401  9327 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:35.585691  9335 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.598536  9128 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:35.612284  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.623247  9318 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.610248  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.649569  2315 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:35.627974  9164 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.629542  9284 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.588636  9190 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:35.633666  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1310) took 1422 ms. Trace:
I20260709 13:04:35.633916  1584 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:35.634688  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:35.635550  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3523) took 1411 ms. Trace:
I20260709 13:04:35.604137  9303 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:35.642485  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.643381  9302 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.643966  9225 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.645404  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.607744  9250 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.645061  9282 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:35.612416  9269 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.655727  1739 rpcz_store.cc:276] 0709 13:04:34.211357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.387063 (+1175706us) service_pool.cc:224] Handling call
0709 13:04:35.387494 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:35.389558 (+  2064us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.389595 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:35.390223 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 13:04:35.457472 (+ 67249us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:35.482665 (+ 25193us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.482909 (+   244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.482968 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.633369 (+150401us) raft_consensus.cc:1685] finished
0709 13:04:35.633388 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.633563 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26106,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":1988,"raft.queue_time_us":97,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":17113,"thread_start_us":66663,"threads_started":1,"wal-append.queue_time_us":240674}
I20260709 13:04:35.663723  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.645430  9128 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.666337  1593 raft_consensus.cc:1240] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:35.665659  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:35.672387  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.675885  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.676666  2315 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:35.677201  2315 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:35.680541  1734 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:35.687068  1734 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:35.691778  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.699720  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.672821  9128 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:35.702399  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 67817.3us
I20260709 13:04:35.702478  2312 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:35.656352  1736 rpcz_store.cc:276] 0709 13:04:34.192323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.989097 (+796774us) service_pool.cc:224] Handling call
0709 13:04:35.065779 (+ 76682us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:35.151438 (+ 85659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.151475 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:35.152278 (+   803us) log.cc:844] Serialized 52 byte log entry
0709 13:04:35.192139 (+ 39861us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:35.198633 (+  6494us) log.cc:844] Serialized 14 byte log entry
0709 13:04:35.199481 (+   848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:35.587550 (+388069us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54000df7d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:35.587606 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.587656 (+    50us) raft_consensus.cc:1685] finished
0709 13:04:35.603838 (+ 16182us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.604022 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6023,"prepare.queue_time_us":173354,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":2228,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":263,"spinlock_wait_cycles":195466624,"thread_start_us":39386,"threads_started":1,"wal-append.queue_time_us":165858}
I20260709 13:04:35.703066  2317 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:04:35.711494  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.712607  2314 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:35.713162  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:35.714103  2281 leader_election.cc:336] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.220s (SENT)
I20260709 13:04:35.714684  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:35.715728  9262 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:04:35.715840  1632 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 2.031s (SENT) [suppressed 74 similar messages]
I20260709 13:04:35.716826  9227 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:35.707265  9090 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.717371  9227 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.718106  9227 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.717841  9282 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.679411  9225 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.727998  9225 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.683854  9190 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.681700  9127 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.726258  9262 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:35.731189  9190 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.729483  9227 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.734683  9227 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:35.737092  9305 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:35.737655  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:35.738137  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.738569  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.738772  9274 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.072s
I20260709 13:04:35.739543  9269 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.687916  8792 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:35.717391  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3517) took 1455 ms. Trace:
I20260709 13:04:35.749963  1666 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 9. (term mismatch)
I20260709 13:04:35.750510  1666 pending_rounds.cc:85] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 8
I20260709 13:04:35.750849  1666 pending_rounds.cc:107] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.9
I20260709 13:04:35.750702  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:35.751222  1666 raft_consensus.cc:2889] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:35.752002  9227 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:35.754093  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.755035  2315 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:35.762185  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.752589  9227 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:35.766670  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:35.770089  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.747907  9305 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:35.773290  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.688235  9250 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.774499  1663 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:35.748760  1733 rpcz_store.cc:276] 0709 13:04:34.190831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.827435 (+636604us) service_pool.cc:224] Handling call
0709 13:04:34.828767 (+  1332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:34.864211 (+ 35444us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:34.864250 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:34.865153 (+   903us) log.cc:844] Serialized 26 byte log entry
0709 13:04:34.979497 (+114344us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:34.979779 (+   282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:34.980234 (+   455us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:35.075373 (+ 95139us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b4800062358. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:04:35.563959 (+488586us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:35.645768 (+ 81809us) raft_consensus.cc:1685] finished
0709 13:04:35.645905 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:35.646072 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13814,"raft.queue_time_us":15542,"raft.run_cpu_time_us":2026,"raft.run_wall_time_us":209775,"spinlock_wait_cycles":237106816,"thread_start_us":113313,"threads_started":1,"wal-append.queue_time_us":189477}
I20260709 13:04:35.704425  9303 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.791671  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.694047  9302 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.764361  8855 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:35.779490  9227 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:35.783720  9305 raft_consensus.cc:3060] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:35.807704  1736 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:04:35.808493  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.811700  2315 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:35.784906  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.808586  8731 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 13:04:35.775422  9248 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.818488  9248 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.819352  1586 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:35.806121  9274 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 13:04:35.821995  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:35.822705  1661 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:35.835098  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.836395  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.858094  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.859396  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.860234  9250 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.860704  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.866665  9148 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:35.867633  1660 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 13:04:35.868129  1660 pending_rounds.cc:85] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:04:35.868455  1660 pending_rounds.cc:107] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 13:04:35.868755  1660 raft_consensus.cc:2889] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:35.868803  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:35.870986  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.764629  9269 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:35.840955  9274 consensus_queue.cc:1048] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:35.809337  8855 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.883524  9274 consensus_queue.cc:1243] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 24.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 13:04:35.843096  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.885181  1741 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
W20260709 13:04:35.886086  9358 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.636s	user 0.005s	sys 0.000s
I20260709 13:04:35.886332  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:35.886957  1741 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:35.845675  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.785578  9225 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.094s
I20260709 13:04:35.890504  9225 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:35.890602  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.890959  9225 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:35.891628  1593 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 22 because replica has last-logged OpId of term: 20 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
I20260709 13:04:35.845897  9164 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.847818  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.849326  9283 consensus_queue.cc:1059] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:04:35.850055  1742 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1297) took 3196 ms (client timeout 2220 ms). Trace:
I20260709 13:04:35.850116  9282 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.850883  9090 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:35.852582  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.840092  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.911580  1589 raft_consensus.cc:2410] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:35.765865  8792 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.915333  8792 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.915521  1590 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:35.878872  1740 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:35.887687  2317 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:35.891889  9225 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:35.917531  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:35.917796  9225 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.918749  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.919953  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.920517  1740 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:35.921715  9283 consensus_queue.cc:1243] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer 316f3bf18ed94e32b238130a6bfedf1a log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 13:04:35.909657  1742 rpcz_store.cc:269] 0709 13:04:32.653934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:34.822052 (+2168118us) service_pool.cc:224] Handling call
0709 13:04:35.849832 (+1027780us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54003fa2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:35.849965 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215462272}
I20260709 13:04:35.900985  1661 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:35.905102  1593 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:35.921294  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.930840  1585 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 20 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 13:04:35.939961  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:35.930079  1588 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:35.943372  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:35.946231  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.947077  1741 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 19.
W20260709 13:04:35.950327  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:35.951277  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:35.947057  1739 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:35.953742  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:35.905422  8855 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:35.958777  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:35.964637  1661 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 13:04:35.965688  1740 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:35.970252  1663 raft_consensus.cc:2468] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:35.985271  9167 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 13:04:35.985913  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:35.986757  9158 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:35.987444  9158 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:35.989003  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.992520  2317 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:04:35.993988  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.995538  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:35.996299  1660 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:35.999009  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:36.000192  1557 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:36.000476  1557 leader_election.cc:403] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:36.004029  9274 raft_consensus.cc:3055] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:36.004706  9274 raft_consensus.cc:740] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:35.966655  9284 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:36.006421  9383 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 13:04:36.008836  9284 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:35.957787  9164 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:36.009918  9164 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.012236  9164 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:36.012828  9164 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 19
I20260709 13:04:36.020547  1736 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:35.974242  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:36.026050  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51865.5us
I20260709 13:04:36.031934  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:35.942174  9262 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.036s
I20260709 13:04:36.036041  1587 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:36.055254  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:36.056068  1661 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:36.051307  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:35.979290  8792 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.031786  9269 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:36.072646  9327 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:36.078102  9303 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:36.079262  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:36.079813  9324 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.907s	user 0.018s	sys 0.000s
I20260709 13:04:36.081168  1735 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:36.081259  1738 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 13:04:36.096802  1662 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:36.097642  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1268) took 1062 ms. Trace:
W20260709 13:04:36.113144  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61943.6us
I20260709 13:04:36.110707  1667 rpcz_store.cc:276] 0709 13:04:35.090693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.111672 (+ 20979us) service_pool.cc:224] Handling call
0709 13:04:36.054191 (+942519us) spinlock_profiling.cc:243] Waited 6.97 ms on lock 0x7b54001a77d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:36.054347 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16023296}
I20260709 13:04:36.119925  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:36.102738  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:36.108110  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 90924.3us
I20260709 13:04:36.019455  9302 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:36.036379  8731 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:36.009459  9284 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:36.133540  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:36.015493  9283 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.143s
I20260709 13:04:36.134157  1736 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
W20260709 13:04:36.140268  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61139us
I20260709 13:04:36.140576  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:36.137673  8731 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:36.162490  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:36.110128  8792 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.140074  9269 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:04:36.170368  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 454451us
I20260709 13:04:36.120018  9262 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 13:04:36.014276  9274 consensus_queue.cc:260] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.143082  9302 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:36.119277  9324 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.133214  8855 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:04:36.148255  9090 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 13:04:36.172712  8731 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:36.201211  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:36.171303  9176 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:36.174993  9269 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:36.204363  1588 pending_rounds.cc:85] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:04:36.166507  9225 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 13:04:36.211122  1588 pending_rounds.cc:107] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260709 13:04:36.216660  1588 raft_consensus.cc:2889] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:36.220691  9288 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:36.221156  9288 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:36.221771  9251 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:36.222290  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:36.222991  2318 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:36.222184  9288 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.224296  9288 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:36.224892  1592 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 13:04:36.222227  9251 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:36.227007  9251 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.234742  9281 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:36.237923  9190 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:36.238519  9190 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:36.239503  9190 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:36.240674  9190 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:36.243795  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:36.244508  2317 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 26.
I20260709 13:04:36.244712  9335 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:36.245476  9335 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.246490  9327 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:36.191279  9302 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:36.224552  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:36.258416  2318 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:36.216840  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:36.253394  1740 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:36.215077  9269 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:36.202214  9375 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 13:04:36.265709  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:36.197363  9291 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 13:04:36.215562  9272 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:36.270152  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53412.5us
I20260709 13:04:36.290331  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:36.272289  9327 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:36.291270  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 317877us
W20260709 13:04:36.293324  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68856.3us
I20260709 13:04:36.298520  2318 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:36.300752  1593 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:36.210718  9176 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.222437  9266 consensus_queue.cc:1059] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 13:04:36.270838  9282 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:36.288882  8855 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:36.312196  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:36.319873  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.330945  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.332224  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.322954  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 116975us
I20260709 13:04:36.337275  1592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3605) took 1286 ms. Trace:
I20260709 13:04:36.296762  9272 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:36.329646  1589 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
W20260709 13:04:36.365801  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 54808us
I20260709 13:04:36.310122  9106 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:36.338079  1592 rpcz_store.cc:276] 0709 13:04:35.051198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.171377 (+120179us) service_pool.cc:224] Handling call
0709 13:04:35.171785 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.225329 (+1053544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:36.321154 (+ 95825us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400443254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:36.337158 (+ 16004us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2395561728}
I20260709 13:04:36.384748  9148 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:36.385723  9148 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.278388  9305 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.319028  9225 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:36.424491  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:36.425419  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:36.425956  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 103461us
I20260709 13:04:36.399355  9318 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:36.377336  9248 raft_consensus.cc:3055] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:36.426818  9282 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:36.451241  9388 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 13:04:36.361764  9272 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:36.305104  9227 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:36.318460  9079 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:36.384732  9335 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:36.483502  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 83600us
I20260709 13:04:36.370826  9323 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
W20260709 13:04:36.487267  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 289195us
I20260709 13:04:36.461423  9225 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:36.322214  9284 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:36.468137  9248 raft_consensus.cc:740] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:36.528671  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46984.4us
W20260709 13:04:36.572398  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51088.8us
I20260709 13:04:36.475241  9318 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:36.579452  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49930.5us
I20260709 13:04:36.644178  1705 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:36.456928  9302 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.646461  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:36.730959  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 84550.4us
I20260709 13:04:36.636682  9324 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.049842  8855 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:36.731962  9164 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.719s	user 0.014s	sys 0.000s
I20260709 13:04:36.650295  9318 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.656726  9251 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.644755  9079 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:36.904273  1741 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.000s	sys 0.057s
W20260709 13:04:37.082231  1741 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.000s	sys 0.057s
I20260709 13:04:36.732456  9127 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.196s
W20260709 13:04:36.905622  1740 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.012s	sys 0.030s
W20260709 13:04:37.088056  1740 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.013s	sys 0.030s
W20260709 13:04:36.909881  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 267758us
W20260709 13:04:36.953598  9386 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.485s	user 0.000s	sys 0.001s
I20260709 13:04:36.855687  9248 consensus_queue.cc:260] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:36.895525  9225 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:36.845839  1739 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.017s	sys 0.026s
W20260709 13:04:36.866351  1733 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.020s	sys 0.033s
W20260709 13:04:36.890262  2280 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.016s	sys 0.026s
I20260709 13:04:37.047551  9259 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:04:37.126169  1739 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.855s	user 0.032s	sys 0.032s
I20260709 13:04:37.125588  9295 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.003s
W20260709 13:04:37.128808  2280 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.016s	sys 0.042s
W20260709 13:04:37.129298  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 795443us
W20260709 13:04:37.126128  1733 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.852s	user 0.021s	sys 0.051s
W20260709 13:04:37.048080  9128 consensus_meta.cc:287] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.325s	user 0.014s	sys 0.019s
I20260709 13:04:36.916703  9375 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.127s
I20260709 13:04:37.131606  2280 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:36.528678  9090 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.129s
I20260709 13:04:36.863672  9097 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 13:04:37.053000  1631 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.072s	sys 0.085s
W20260709 13:04:37.142827  1631 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.073s	sys 0.086s
W20260709 13:04:37.143239  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:37.060403  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 89137.2us
I20260709 13:04:37.072961  1705 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:37.091405  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3540) took 1390 ms. Trace:
I20260709 13:04:37.056381  8855 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.101735  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3615) took 1443 ms. Trace:
W20260709 13:04:37.110492  9392 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.444s	user 0.006s	sys 0.000s
I20260709 13:04:37.116103  9251 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.118566  9318 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.163239  1585 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:04:37.118563  1590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3629) took 1296 ms. Trace:
I20260709 13:04:37.120271  9079 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:37.133688  1707 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.027s	sys 0.017s
I20260709 13:04:37.049934  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.082633  9164 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
W20260709 13:04:37.174361  1707 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.028s	sys 0.017s
I20260709 13:04:37.180130  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:37.128738  1737 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.004s	sys 0.009s
W20260709 13:04:37.149212  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.142438  9097 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:37.161823  1592 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:37.176689  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:37.152228  1584 rpcz_store.cc:276] 0709 13:04:35.658113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.668666 (+ 10553us) service_pool.cc:224] Handling call
0709 13:04:35.698803 (+ 30137us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:35.744392 (+ 45589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.744446 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:35.757189 (+ 12743us) log.cc:844] Serialized 26 byte log entry
0709 13:04:36.167313 (+410124us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:36.167392 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:36.167507 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.091005 (+923498us) raft_consensus.cc:1685] finished
0709 13:04:37.091041 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.101618 (+ 10577us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3483,"raft.queue_time_us":709,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10043,"thread_start_us":380814,"threads_started":1,"wal-append.queue_time_us":541070}
I20260709 13:04:37.161578  9411 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
W20260709 13:04:37.159268  2280 connection.cc:256] RPC call timeout handler was delayed by -1.3220524489999548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 13:04:37.162305  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.162523  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.144155  1734 rpcz_store.cc:276] 0709 13:04:35.700933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.801333 (+100400us) service_pool.cc:224] Handling call
0709 13:04:35.834031 (+ 32698us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:35.834574 (+   543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.834606 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:35.835341 (+   735us) log.cc:844] Serialized 52 byte log entry
0709 13:04:36.107581 (+272240us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:36.628573 (+520992us) log.cc:844] Serialized 14 byte log entry
0709 13:04:36.629002 (+   429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:36.629140 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.091165 (+462025us) raft_consensus.cc:1685] finished
0709 13:04:37.091188 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.091338 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54681,"prepare.queue_time_us":48044,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":3947,"raft.queue_time_us":104734,"raft.run_cpu_time_us":8694,"raft.run_wall_time_us":308032,"thread_start_us":698675,"threads_started":2,"wal-append.queue_time_us":435690}
I20260709 13:04:37.163782  1585 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:37.185400  1586 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:37.186868  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:37.183813  1737 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.876s	user 0.005s	sys 0.009s
I20260709 13:04:37.191378  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.197996  1662 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:37.205300  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.153146  9302 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.166935  1590 rpcz_store.cc:276] 0709 13:04:35.822524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.916353 (+ 93829us) service_pool.cc:224] Handling call
0709 13:04:35.916658 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:35.940283 (+ 23625us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:36.263691 (+323408us) log.cc:844] Serialized 14 byte log entry
0709 13:04:36.800501 (+536810us) log.cc:844] Serialized 14 byte log entry
0709 13:04:37.049141 (+248640us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.066335 (+ 17194us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.066567 (+   232us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:37.066618 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.066726 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.118245 (+ 51519us) raft_consensus.cc:1685] finished
0709 13:04:37.118280 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.118470 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57712,"prepare.queue_time_us":27882,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":129,"raft.queue_time_us":79682,"raft.run_cpu_time_us":548,"raft.run_wall_time_us":39847,"spinlock_wait_cycles":310912,"thread_start_us":723624,"threads_started":2,"wal-append.queue_time_us":815705}
I20260709 13:04:37.080340   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:04:37.211040  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.168036  9148 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:37.206648  2317 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.006s	sys 0.013s
W20260709 13:04:37.216326  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.216430  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.216506  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.185873  9225 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:37.216645  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:37.217911  2317 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.007s	sys 0.013s
I20260709 13:04:37.183010  9418 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:37.228379  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:37.173784  9410 raft_consensus.cc:2764] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 13:04:37.227720  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.235889  1667 raft_consensus.cc:2393] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:37.210350  9323 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:37.241753  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:37.226922  9190 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.986s	user 0.014s	sys 0.005s
W20260709 13:04:37.246234  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 189680us
I20260709 13:04:37.246627  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.258540  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3548) took 1441 ms. Trace:
I20260709 13:04:37.259378  1734 rpcz_store.cc:276] 0709 13:04:35.816943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.205629 (+1388686us) service_pool.cc:224] Handling call
0709 13:04:37.230377 (+ 24748us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.257902 (+ 27525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.257957 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:37.258035 (+    78us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:37.258085 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.258168 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.258433 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:37.226601  9305 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.143262  9128 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.225621  9079 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:37.270501  1667 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:37.244349  9248 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.276922  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.282887  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.284303  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.235911  9227 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.227241  9401 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:37.287307  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 630716us
I20260709 13:04:37.301157  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3549) took 1484 ms. Trace:
I20260709 13:04:37.302222  1734 rpcz_store.cc:276] 0709 13:04:35.817214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.272797 (+1455583us) service_pool.cc:224] Handling call
0709 13:04:37.300439 (+ 27642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.300707 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.300745 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:37.300805 (+    60us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:37.300863 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.300932 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.301082 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:37.251174  9409 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:37.309226  9400 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:37.293313  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3543) took 1522 ms. Trace:
I20260709 13:04:37.213255  9412 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 13:04:37.297286  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2764) took 1557 ms. Trace:
I20260709 13:04:37.241425  9402 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.270416  8798 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.285883  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:37.309415  9408 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.269833  8791 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:37.314666  1735 rpcz_store.cc:276] 0709 13:04:35.770630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.175887 (+405257us) service_pool.cc:224] Handling call
0709 13:04:36.190885 (+ 14998us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:37.045704 (+854819us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.045738 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:37.046468 (+   730us) log.cc:844] Serialized 52 byte log entry
0709 13:04:37.234056 (+187588us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.234101 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.234182 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.292932 (+ 58750us) raft_consensus.cc:1685] finished
0709 13:04:37.292964 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.293215 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":106090,"raft.queue_time_us":334589,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":62901,"thread_start_us":520486,"threads_started":2,"wal-append.queue_time_us":243499}
W20260709 13:04:37.327374  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 261576us
I20260709 13:04:37.317180  1741 rpcz_store.cc:276] 0709 13:04:35.739750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.976463 (+236713us) service_pool.cc:224] Handling call
0709 13:04:36.035953 (+ 59490us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:37.082732 (+1046779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.082772 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:37.098068 (+ 15296us) log.cc:844] Serialized 52 byte log entry
0709 13:04:37.248794 (+150726us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:37.248857 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.248977 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.296979 (+ 48002us) raft_consensus.cc:1685] finished
0709 13:04:37.297004 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.297212 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":235788,"raft.queue_time_us":348168,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"thread_start_us":930682,"threads_started":2,"wal-append.queue_time_us":184430}
I20260709 13:04:37.348876  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:37.354310  2309 raft_consensus.cc:2368] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:37.308674  9409 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.313198  9400 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:37.349191  8731 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.090s	user 0.013s	sys 0.005s
I20260709 13:04:37.319401  9415 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:37.358855  2283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.960s	user 0.010s	sys 0.037s
I20260709 13:04:37.299072  9079 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.346992  1734 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 13:04:37.389284  2283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.991s	user 0.010s	sys 0.037s
I20260709 13:04:37.390904  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:37.392144  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 993629us
I20260709 13:04:37.404227  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1316) took 1586 ms. Trace:
I20260709 13:04:37.404664  1734 rpcz_store.cc:276] 0709 13:04:35.817954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.303465 (+1485511us) service_pool.cc:224] Handling call
0709 13:04:37.338382 (+ 34917us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.398973 (+ 60591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.404168 (+  5195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:37.401111  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:37.412111  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 225340us
W20260709 13:04:37.358038  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78729.5us
I20260709 13:04:37.418767  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:37.383394  9400 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:37.378686  9227 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:37.308796  9401 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.366407  9409 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.426070  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3531) took 1844 ms. Trace:
I20260709 13:04:37.380916  9318 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.424369  2317 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1116) took 1172 ms. Trace:
I20260709 13:04:37.438879  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.458330  1741 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:37.354001  8798 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:37.472980  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.462725  1737 rpcz_store.cc:276] 0709 13:04:35.584184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.588961 (+  4777us) service_pool.cc:224] Handling call
0709 13:04:35.621335 (+ 32374us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:35.643795 (+ 22460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:35.643837 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 13:04:35.644779 (+   942us) log.cc:844] Serialized 78 byte log entry
0709 13:04:36.097496 (+452717us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:36.128910 (+ 31414us) log.cc:844] Serialized 14 byte log entry
0709 13:04:37.196473 (+1067563us) log.cc:844] Serialized 14 byte log entry
0709 13:04:37.425579 (+229106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.425694 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.425755 (+    61us) raft_consensus.cc:1685] finished
0709 13:04:37.425775 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.425985 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":406744,"prepare.queue_time_us":33668,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":79,"raft.queue_time_us":504119,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":74201,"thread_start_us":1340336,"threads_started":2,"wal-append.queue_time_us":672370}
I20260709 13:04:37.436465  9418 raft_consensus.cc:697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:37.481278  9419 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.101s	user 0.001s	sys 0.001s
W20260709 13:04:37.481766  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132964us
W20260709 13:04:37.482687  9386 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 13:04:37.356657  8791 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.489866  8855 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.356714  9402 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.569710  9437 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:37.571488  9437 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:37.571764  9402 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.572304  9437 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.572914  9437 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:37.432128  9335 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.456494  2317 rpcz_store.cc:276] 0709 13:04:36.252048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.252202 (+   154us) service_pool.cc:224] Handling call
0709 13:04:36.252519 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:36.252743 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:36.607396 (+354653us) log.cc:844] Serialized 14 byte log entry
0709 13:04:37.423704 (+816308us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:37.423884 (+   180us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.423979 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.424080 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.424278 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":866581,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":58495,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4464,"thread_start_us":1044709,"threads_started":3,"wal-append.queue_time_us":752540}
W20260709 13:04:37.519168  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 136810us
W20260709 13:04:37.519502  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:37.520533  9303 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.581087  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3535) took 1959 ms. Trace:
I20260709 13:04:37.520855  9128 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.462397  9415 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:37.451207  9305 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.521436  1735 raft_consensus.cc:1275] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 13:04:37.518438  9424 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.522908  9416 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.523732  9412 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 13:04:37.524524  9272 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.948s	user 0.005s	sys 0.012s
I20260709 13:04:37.525640  9430 consensus_queue.cc:1059] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:37.527019  9148 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:37.527262  9269 consensus_meta.cc:287] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.226s	user 0.013s	sys 0.001s
W20260709 13:04:37.528153  9422 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.058s	user 0.002s	sys 0.000s
I20260709 13:04:37.480593  9405 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.306910  9097 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.531455  9401 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.532452  9190 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:37.358062  9408 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.533571  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2767) took 1792 ms. Trace:
I20260709 13:04:37.534657  9423 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.537034  9176 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.537382  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2768) took 1796 ms. Trace:
I20260709 13:04:37.543871  9079 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.630195  1736 rpcz_store.cc:276] 0709 13:04:35.740933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.149695 (+408762us) service_pool.cc:224] Handling call
0709 13:04:36.171745 (+ 22050us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.661677 (+489932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.662519 (+   842us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.898421 (+235902us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.359560 (+461139us) raft_consensus.cc:1645] Marking committed up to 8
0709 13:04:37.359602 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.359687 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.537117 (+177430us) raft_consensus.cc:1685] finished
0709 13:04:37.537149 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.537311 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":136269,"raft.queue_time_us":306865,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":18989,"thread_start_us":567010,"threads_started":2,"wal-append.queue_time_us":434873}
I20260709 13:04:37.544682  8792 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.392212  8731 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.634878  8731 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.449224  1661 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:37.636231  1660 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 13:04:37.549849  9318 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.438613  9227 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:37.640525  9227 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.453310  9295 raft_consensus.cc:2804] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:37.646610  9295 raft_consensus.cc:697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:37.648008  9295 consensus_queue.cc:237] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.432379  9406 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.649884  9406 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.578760  9418 consensus_queue.cc:237] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.582445  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.652258  9401 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:37.652451  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 70124.6us
W20260709 13:04:37.583303  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 317616us
I20260709 13:04:37.520454  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:37.603899  9412 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:37.584156  1733 rpcz_store.cc:276] 0709 13:04:35.621958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.794169 (+172211us) service_pool.cc:224] Handling call
0709 13:04:35.807580 (+ 13411us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 13:04:37.130606 (+1323026us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.130650 (+    44us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 13:04:37.162200 (+ 31550us) log.cc:844] Serialized 104 byte log entry
0709 13:04:37.391082 (+228882us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.391151 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.391258 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.580721 (+189463us) raft_consensus.cc:1685] finished
0709 13:04:37.580750 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.581008 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":310608,"raft.queue_time_us":334829,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":64882,"thread_start_us":1084159,"threads_started":2,"wal-append.queue_time_us":415454}
W20260709 13:04:37.603050  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 318746us
I20260709 13:04:37.572770  9402 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
W20260709 13:04:37.609426  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 199395us
I20260709 13:04:37.607864  1662 pending_rounds.cc:85] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 7
I20260709 13:04:37.663548  1662 pending_rounds.cc:107] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 13:04:37.621245  1738 rpcz_store.cc:276] 0709 13:04:35.740654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.134125 (+393471us) service_pool.cc:224] Handling call
0709 13:04:36.140329 (+  6204us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.648839 (+508510us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.648910 (+    71us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.046841 (+397931us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.265051 (+218210us) raft_consensus.cc:1645] Marking committed up to 7
0709 13:04:37.266287 (+  1236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.266399 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.533283 (+266884us) raft_consensus.cc:1685] finished
0709 13:04:37.533317 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.533470 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":445018,"raft.queue_time_us":503848,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":313442,"thread_start_us":487111,"threads_started":2,"wal-append.queue_time_us":484086}
I20260709 13:04:37.663861  1662 raft_consensus.cc:2889] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:37.622154  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 99050.2us
I20260709 13:04:37.628293  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.628852  1663 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: []
I20260709 13:04:37.601264  9424 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:37.586053  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167972us
I20260709 13:04:37.657467  1706 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:37.597867  9303 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:37.659070  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138722us
I20260709 13:04:37.665906  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.658212  1668 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:04:37.671180  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:37.667294  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.662053  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:37.668674  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 101512us
I20260709 13:04:37.597867  9335 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.673444  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:37.674489  1593 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:37.639866  9318 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.675962  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:37.660048  9097 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:37.608563  9305 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.663278  9412 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:37.663333  9176 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.681000  2281 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:37.682660  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1246) took 1313 ms. Trace:
I20260709 13:04:37.687451  1706 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:37.665983  9430 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:37.684707  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.608947  9272 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.669723  8792 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:37.617843  9269 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.703593  1665 raft_consensus.cc:2393] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:37.696058  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.685950  9128 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:37.701824  8792 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.714135  8792 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.707540  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:37.687237  9425 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 13:04:37.708876  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:37.679512  9079 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:37.682261  9411 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election won for term 24
W20260709 13:04:37.687343  9432 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 13:04:37.609225  9416 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.723276  1632 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:37.626035  9405 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.706985  9128 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.694610  1589 rpcz_store.cc:276] 0709 13:04:36.369391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.382972 (+ 13581us) service_pool.cc:224] Handling call
0709 13:04:36.456037 (+ 73065us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.578294 (+122257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.578344 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.634853 (+ 56509us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.130330 (+495477us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:37.130385 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.130485 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.682399 (+551914us) raft_consensus.cc:1685] finished
0709 13:04:37.682426 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.682569 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44654,"raft.queue_time_us":58610,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":36819,"thread_start_us":398359,"threads_started":1,"wal-append.queue_time_us":719690}
I20260709 13:04:37.678926  9090 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:37.678905  9190 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 13:04:37.722463  9318 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:37.687130  9327 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 13:04:37.702193  9176 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.737010  9128 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:37.702785  9409 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.737653  9262 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:37.757433  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.721275  9323 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.039s
W20260709 13:04:37.769956  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83341us
I20260709 13:04:37.759295  8855 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.748052  9090 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.818640  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:37.819789  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 68935.1us
I20260709 13:04:37.820081  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:37.823724  1584 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.800944  9250 raft_consensus.cc:2804] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:37.829905  8855 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:37.831622  9335 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:37.831820  9449 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 13:04:37.832037  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:37.832679  8855 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:37.833206  9451 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:37.803880  9186 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.130s
I20260709 13:04:37.841997  9408 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.842348  9269 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.845883  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 733) took 1486 ms. Trace:
I20260709 13:04:37.846484  9409 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.822870  9318 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:37.800149  9401 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.847812  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.848399  9176 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:37.848127  9401 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:37.848590  1660 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:37.826155  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2762) took 2124 ms. Trace:
W20260709 13:04:37.826460  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 139066us
I20260709 13:04:37.826835  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.846599  9405 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:37.796800  9262 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:37.872624  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.850885  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.846345  2318 rpcz_store.cc:276] 0709 13:04:36.359855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.360031 (+   176us) service_pool.cc:224] Handling call
0709 13:04:36.387866 (+ 27835us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.669294 (+281428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.669405 (+   111us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.090504 (+421099us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.629534 (+539030us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:37.629583 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.629670 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.844158 (+214488us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:37.845419 (+  1261us) raft_consensus.cc:1685] finished
0709 13:04:37.845443 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.845778 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":503792,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":7071,"spinlock_wait_cycles":8675200,"thread_start_us":708432,"threads_started":2,"wal-append.queue_time_us":563125}
I20260709 13:04:37.874385  1665 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:37.883234  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.873039  1591 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:37.885082  2314 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 26.
I20260709 13:04:37.883919  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.851178  1742 rpcz_store.cc:276] 0709 13:04:35.701986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.962141 (+260155us) service_pool.cc:224] Handling call
0709 13:04:35.980702 (+ 18561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:36.571550 (+590848us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.571593 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:36.644059 (+ 72466us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.178669 (+534610us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:37.178736 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.178848 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:37.825831 (+646983us) raft_consensus.cc:1685] finished
0709 13:04:37.825866 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:37.826071 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":290941,"raft.queue_time_us":403191,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":78,"thread_start_us":432162,"threads_started":2,"wal-append.queue_time_us":535412}
I20260709 13:04:37.902208  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:37.911231  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.950763  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:37.883584  1593 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:37.953523  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:37.950585  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 227350us
I20260709 13:04:37.914116  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:37.948691  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.911330  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.882658  1668 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:37.954658  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:37.956162  2317 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:37.966435  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.967720  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:37.968314  1741 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:37.969051  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:37.969548  1734 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:37.878296  9262 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:37.962759  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:37.964159  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.963603  1590 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:37.967295  1737 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 26.
I20260709 13:04:37.957721  1660 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:37.975713  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:37.912254  9447 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 13:04:37.959789  2283 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:37.901394  9405 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:37.957541  1733 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:37.970535  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:37.885829  9250 raft_consensus.cc:697] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:38.007121  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50897.8us
I20260709 13:04:37.988957  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.885540  9409 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.904744  9439 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:37.955289  9411 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:37.919296  9272 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:37.985535  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:38.041273  1593 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:37.880134  9335 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:37.954270  9408 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:37.878523  9375 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.120s
I20260709 13:04:37.967244  9318 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.024515  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:37.885767  9269 consensus_queue.cc:1059] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.112s
I20260709 13:04:38.041596  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.044564  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:37.999583  8798 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.052121  9250 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:38.000025  9435 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
W20260709 13:04:38.058161  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49898.9us
I20260709 13:04:38.065202  1664 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:38.005864  9303 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.070003  1590 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.083820  1668 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:38.014752  9305 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.088461  1741 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:38.069818  1665 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
W20260709 13:04:38.057931  9248 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.758s	user 0.011s	sys 0.003s
I20260709 13:04:38.090979  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.001925  9323 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.100274  9323 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.103376  9323 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.088961  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.091882  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.094097  9128 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.097877  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:38.078981  9443 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 13:04:38.119429  1660 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:38.123647  1590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:38.125141  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.125330  1593 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.125829  1668 raft_consensus.cc:2468] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:38.126101  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.127141  1660 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:38.128232  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 168530us
W20260709 13:04:38.132716  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.065174  9439 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:38.132839  1590 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
W20260709 13:04:38.135023  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 148287us
I20260709 13:04:38.131834  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:38.137846  1593 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:38.139966  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.141173  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.145749  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.037737  9405 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.127585  9269 raft_consensus.cc:2804] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:38.073441  9408 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.150997  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.065639  9335 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.116750  9402 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:38.063467  9409 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.159192  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.168738  1593 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:38.081290  9318 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.156934  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.161548  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.162810  9090 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:38.163333  9400 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.617s	user 0.007s	sys 0.010s
I20260709 13:04:38.163393  1665 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:38.163832  9407 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.164021  9269 raft_consensus.cc:697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:38.164351  9455 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.171s
I20260709 13:04:38.164973  9454 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:38.165385  9444 consensus_queue.cc:1048] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.128s
I20260709 13:04:38.165817  9466 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:38.167569  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.167038  9402 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:38.170570  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.121577  9148 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 13:04:38.175745  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3546) took 2359 ms. Trace:
I20260709 13:04:38.309230  1740 rpcz_store.cc:276] 0709 13:04:35.816364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.301716 (+485352us) service_pool.cc:224] Handling call
0709 13:04:36.302191 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:37.088565 (+786374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.088617 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.105795 (+ 17178us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.405612 (+299817us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:37.405653 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.405739 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:38.175539 (+769800us) raft_consensus.cc:1685] finished
0709 13:04:38.175562 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:38.175683 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62931,"raft.queue_time_us":362103,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":130461,"thread_start_us":965088,"threads_started":2,"wal-append.queue_time_us":453044}
I20260709 13:04:38.311614  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.313887  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.177984  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.314891  2314 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:38.187340  9407 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.315595  1740 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.317394  9407 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.176479  9457 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:38.166522  9335 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.189294  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.318137  9457 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:38.318631  9457 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:38.319350  9448 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:38.319478  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.319924  9448 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:38.186775  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:38.194653  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.197678  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.321967  1736 raft_consensus.cc:3060] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.323129  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.324369  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.325007  1662 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.202925  9402 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:04:38.177629  9302 consensus_meta.cc:287] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.948s	user 0.011s	sys 0.004s
W20260709 13:04:38.321964  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 135249us
I20260709 13:04:38.331116  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:38.301290  9412 consensus_meta.cc:287] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.596s	user 0.006s	sys 0.006s
I20260709 13:04:38.331923  9412 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.332454  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.241703  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.320907  9448 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.334043  1665 raft_consensus.cc:2393] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:38.334298  9448 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:38.334995  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.336277  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.178704  9400 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.338758  9400 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.340294  9282 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.321036  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.217571  9454 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.341846  9335 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:38.342304  9335 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:38.342695  9335 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:38.343509  9454 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.343699  1660 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:38.345538  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.345659  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.346818  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.346992  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.347715  1660 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:38.350152  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3639) took 1202 ms. Trace:
I20260709 13:04:38.350626  1587 rpcz_store.cc:276] 0709 13:04:37.147449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.147624 (+   175us) service_pool.cc:224] Handling call
0709 13:04:37.174628 (+ 27004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:37.187122 (+ 12494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.187172 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:37.187236 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.187297 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.349880 (+1162583us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b54001c3c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:38.349936 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:38.350087 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28097408}
I20260709 13:04:38.349364  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:38.352157  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.352836  1587 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:38.354140  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.354770  9416 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.355371  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.356146  1741 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:38.357023  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.200052  9090 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.081s
I20260709 13:04:38.355337  1662 raft_consensus.cc:2410] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 13:04:38.202534  9459 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:38.190778  9269 consensus_queue.cc:237] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.170362  9439 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:38.297540  9437 consensus_meta.cc:287] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.712s	user 0.013s	sys 0.000s
I20260709 13:04:38.366822  9409 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:38.354822  9097 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.672s	user 0.005s	sys 0.010s
I20260709 13:04:38.297717  9250 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:38.201073  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:38.298245  9323 consensus_queue.cc:1059] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:38.299424  9405 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:38.300199  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129679us
I20260709 13:04:38.176452  9411 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.300925  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.301461  9408 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.297595  9466 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.302726  9318 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:38.303237  9465 log.cc:927] Time spent T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.101s	user 0.009s	sys 0.000s
I20260709 13:04:38.303572  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:38.195330  9418 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:38.305429  9227 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.664s	user 0.014s	sys 0.003s
I20260709 13:04:38.315018  1593 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:38.358240  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.422323  1738 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:38.187537  9469 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:38.364470  1664 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:38.348204  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.423686  1735 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:38.370146  1734 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:38.357877  9090 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 13:04:38.377871  1584 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:38.427305  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.428030  1584 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.428097  1660 raft_consensus.cc:2468] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:38.388845  9167 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:38.383950  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.395342  1736 raft_consensus.cc:2468] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:38.393968  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:38.402204  1663 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:38.409164  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:38.402428  1740 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
W20260709 13:04:38.410225  1557 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.009s	sys 0.009s
I20260709 13:04:38.418562  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:38.419922  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 218277us
W20260709 13:04:38.385377  9389 log.cc:927] Time spent T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.910s	user 0.006s	sys 0.011s
I20260709 13:04:38.382658  1741 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
W20260709 13:04:38.424785  1558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.010s	sys 0.011s
I20260709 13:04:38.439792  1632 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.429944  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:38.419699  9190 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.010s	sys 0.011s
I20260709 13:04:38.302561  9128 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.420452  9323 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:38.437880  1668 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 13:04:38.438066  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:38.438546  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:38.440582  1558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.010s	sys 0.012s
I20260709 13:04:38.424983  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.456344  1593 raft_consensus.cc:2393] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.456979  1734 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:04:38.441181  1738 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:38.383741  9412 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.459066  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:38.459424  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.442137  1660 raft_consensus.cc:1240] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:04:38.460389  1584 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.443562  1741 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 13:04:38.460992  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.451893  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:38.453233  1557 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.009s	sys 0.010s
I20260709 13:04:38.457152  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.443320  1740 raft_consensus.cc:1217] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Deduplicated request from leader. Original: 13.8->[15.9-20.11]   Dedup: 15.9->[18.10-20.11]
I20260709 13:04:38.464783  1740 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [15.9-20.11]
I20260709 13:04:38.466467  1736 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:38.454608  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.386062  9437 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.434119  9446 raft_consensus.cc:3055] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:38.477977  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.480070  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:38.439659  9469 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:38.465390  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 740858us
I20260709 13:04:38.473393  1660 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:38.474139  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: true dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:38.484218  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.484512  1587 raft_consensus.cc:2393] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.411505  9402 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.389631  9459 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.497799  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.499437  9459 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.499732  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.502475  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.503907  9430 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:38.504463  9430 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:38.488026  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 16 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.469650  9090 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.429193  9418 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.472210  9303 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:38.432405  9167 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:38.386936  9262 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.483295  1734 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:38.449002  9250 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:38.486092  2317 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:38.486613  1586 raft_consensus.cc:2393] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 18: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.490175  1662 pending_rounds.cc:85] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 13
I20260709 13:04:38.482442  9248 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:38.535038  9248 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.493672  9466 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.430562  9227 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:38.444693  9190 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.010s	sys 0.011s
I20260709 13:04:38.539757  9227 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.540884  1666 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.11-21.12]
I20260709 13:04:38.540273  9430 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.544272  9250 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:38.544572  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56492 (request call id 1277) took 1387 ms (client timeout 1817 ms). Trace:
I20260709 13:04:38.544854  9250 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:38.545131  1661 rpcz_store.cc:269] 0709 13:04:37.157651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.157822 (+   171us) service_pool.cc:224] Handling call
0709 13:04:38.544353 (+1386531us) spinlock_profiling.cc:243] Waited 76.6 ms on lock 0x7b54001a81d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:38.544480 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":176135424}
I20260709 13:04:38.547103  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.501365  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.548508  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.469054  9305 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.481493  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.559082  1593 raft_consensus.cc:2368] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:38.512085  2318 raft_consensus.cc:2368] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 24. Current term is 25.
I20260709 13:04:38.513401  1632 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.517853  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.518674  9283 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:38.519060  9469 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.522245  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.525411  9457 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.525826  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.527972  9418 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.511807  9475 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:38.584692  9475 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:38.585150  9475 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:38.585424  9477 log.cc:927] Time spent T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.074s	user 0.001s	sys 0.000s
I20260709 13:04:38.585724  9335 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.586192  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.587081  1593 raft_consensus.cc:2393] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:38.588058  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.588672  9335 leader_election.cc:290] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:38.528652  9259 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:38.589829  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.590143  9259 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:38.590603  1734 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:38.591046  9259 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:38.591498  9259 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:38.591601  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.592659  9259 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.549872  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.480679  9446 raft_consensus.cc:740] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:38.569904  1738 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
W20260709 13:04:38.583249  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69933.5us
I20260709 13:04:38.533638  1662 pending_rounds.cc:107] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 13:04:38.604365  1662 raft_consensus.cc:2889] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:38.593019  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.605350  1734 raft_consensus.cc:2468] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:38.595410  2314 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:38.607447  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 150419us
I20260709 13:04:38.607851  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:38.609647  1587 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:38.610471  1738 raft_consensus.cc:2393] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:38.613919  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:38.612078  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.621096  1665 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 27: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:38.628844  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:38.629822  1734 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:38.619062  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.630954  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.631716  9324 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:38.631814  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:38.632294  9324 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:38.633394  9324 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.634029  9323 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:38.601181  9303 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:38.654675  2314 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
W20260709 13:04:38.657908  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 167258us
I20260709 13:04:38.677292  1587 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:38.632743  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:38.685817  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.694267  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.652441  1734 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:38.731510  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.733099  2317 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:38.573628  9305 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.744722  1586 raft_consensus.cc:1275] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 10. (term mismatch)
I20260709 13:04:38.627810  9106 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 13:04:38.645881  9478 log.cc:927] Time spent T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 13:04:38.747742  2312 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:38.750459  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.751479  2311 raft_consensus.cc:2393] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:38.646064  9461 raft_consensus.cc:3055] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:38.752437  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 119746us
I20260709 13:04:38.758096  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:38.745311  1586 pending_rounds.cc:85] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 9
I20260709 13:04:38.759375  1586 pending_rounds.cc:107] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20260709 13:04:38.759793  1586 raft_consensus.cc:2889] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:38.820371  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:38.821449  2317 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:38.549286  9167 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.732268  9179 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:38.739571  9259 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:38.742370  9303 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:38.861086  8792 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:38.870393  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:38.870654  9483 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:38.871894  2313 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:38.895862  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:38.757351  9455 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:38.872917  9305 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:38.748682  9402 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.747366  9363 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:38.880488  2316 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:38.747954  9106 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:38.751880  9407 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.145s
I20260709 13:04:38.757459  9461 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:38.851152  9318 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:38.889401  9167 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:38.844148  9474 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:39.002610  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:39.008232  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.027799  2317 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 27.
I20260709 13:04:39.022142  2313 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:04:38.915934  9248 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:38.915270  9363 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:38.949967  9106 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:38.951016  9455 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:39.013279  9318 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:38.908077  9483 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.050760  9305 raft_consensus.cc:2804] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:39.047641  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:39.056458  9424 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:39.071264  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:39.033726  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 54742.4us
I20260709 13:04:39.002153  9097 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:39.074657  9305 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:39.061992  9274 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:39.093823  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.091327  9305 raft_consensus.cc:3060] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.102355  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1250) took 1219 ms. Trace:
I20260709 13:04:39.096629  9262 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.114181  2313 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 13:04:39.099203  9457 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:39.123234  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.097445  9303 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.108472  1584 rpcz_store.cc:276] 0709 13:04:37.882703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.609251 (+726548us) service_pool.cc:224] Handling call
0709 13:04:38.643161 (+ 33910us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.643534 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 13:04:38.744524 (+100990us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.076360 (+331836us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.076441 (+    81us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:39.076525 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.076623 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.102155 (+ 25532us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4514,"prepare.queue_time_us":108107,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"thread_start_us":326858,"threads_started":1}
I20260709 13:04:39.047199  9474 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:38.924502  9446 consensus_queue.cc:260] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:39.151265  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.117461  9363 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.165623  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:39.166525  2317 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 27: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:39.129803  9262 consensus_queue.cc:1059] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 13:04:39.116082  9461 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.117254  9455 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:39.183423  1660 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.038s	sys 0.078s
W20260709 13:04:39.158592  9484 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 13:04:39.153635  9446 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:39.157392  9475 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.572s	user 0.009s	sys 0.000s
I20260709 13:04:39.187261  2313 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
W20260709 13:04:39.186023  1660 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.039s	sys 0.078s
I20260709 13:04:39.220952  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3404) took 1064 ms. Trace:
I20260709 13:04:39.214159  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:39.206919  9490 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 13:04:39.233024  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 474989us
W20260709 13:04:39.207388  9488 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 13:04:39.223600  1663 rpcz_store.cc:276] 0709 13:04:38.156149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.579282 (+423133us) service_pool.cc:224] Handling call
0709 13:04:38.579798 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.617124 (+ 37326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.617175 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.654916 (+ 37741us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.080921 (+426005us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:39.081006 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.081175 (+   169us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.220688 (+139513us) raft_consensus.cc:1685] finished
0709 13:04:39.220717 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.220879 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21682,"raft.queue_time_us":21821,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":12057,"thread_start_us":329860,"threads_started":1,"wal-append.queue_time_us":397041}
I20260709 13:04:39.199189  9461 raft_consensus.cc:3060] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:39.206267  9492 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 13:04:39.198247  9424 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.192289  9475 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.221727  9318 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:39.241887  9250 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.696s	user 0.015s	sys 0.000s
I20260709 13:04:39.231563  9302 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:39.207916  9499 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.251332  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2789) took 1198 ms. Trace:
I20260709 13:04:39.238545  8792 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:39.253726  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.199780  9291 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 13:04:39.265427  1733 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:39.225283  9179 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:39.262410  1741 rpcz_store.cc:276] 0709 13:04:38.053352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.478074 (+424722us) service_pool.cc:224] Handling call
0709 13:04:38.495035 (+ 16961us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.495638 (+   603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.495685 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.496413 (+   728us) log.cc:844] Serialized 26 byte log entry
0709 13:04:38.595666 (+ 99253us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:38.595746 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.595845 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.251070 (+655225us) raft_consensus.cc:1685] finished
0709 13:04:39.251107 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.251258 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1643,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":148,"thread_start_us":98717,"threads_started":1,"wal-append.queue_time_us":254727}
I20260709 13:04:39.249002  9363 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:39.218231  9407 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
W20260709 13:04:39.277253  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.277439  1587 raft_consensus.cc:1240] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:39.259735  9448 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:39.293637  9494 log.cc:927] Time spent T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 13:04:39.293852  9448 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.296139  9448 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.258612  9424 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.297964  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3541) took 3596 ms. Trace:
I20260709 13:04:39.298362  1739 rpcz_store.cc:276] 0709 13:04:35.701229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.962010 (+260781us) service_pool.cc:224] Handling call
0709 13:04:35.975464 (+ 13454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:37.127082 (+1151618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.127134 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:37.127912 (+   778us) log.cc:844] Serialized 26 byte log entry
0709 13:04:37.132169 (+  4257us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:37.132219 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:37.132314 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.296932 (+2164618us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b24000c3818. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdead769c4 00007fcdead74acb 00007fcdeaf2fdc9 00007fcdeaf2fa55 00007fcdeaf2efb0 00007fcdedba5488 00007fcdedba5be8 00007fcdedb7d398 00007fcdedb7876a 00007fcdedb77bb6
0709 13:04:39.297606 (+   674us) raft_consensus.cc:1685] finished
0709 13:04:39.297629 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.297874 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":294911,"raft.queue_time_us":407485,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":139,"spinlock_wait_cycles":31031680,"thread_start_us":859168,"threads_started":2,"wal-append.queue_time_us":169930}
I20260709 13:04:39.299599  9483 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.300437  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.301337  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.302066  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.304203  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.306353  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3403) took 1153 ms. Trace:
I20260709 13:04:39.308743  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3402) took 1158 ms. Trace:
I20260709 13:04:39.307830  1661 rpcz_store.cc:276] 0709 13:04:38.152830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.546382 (+393552us) service_pool.cc:224] Handling call
0709 13:04:38.546850 (+   468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.547353 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.547416 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.547979 (+   563us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.028804 (+480825us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:39.028882 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.028983 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.306133 (+277150us) raft_consensus.cc:1685] finished
0709 13:04:39.306158 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.306295 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12824,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":2700,"thread_start_us":480357,"threads_started":1,"wal-append.queue_time_us":558209}
I20260709 13:04:39.284782  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.305490  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.310241  1741 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:39.264176  9302 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:39.272578  9499 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:39.290165  9493 log.cc:927] Time spent T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 13:04:39.290999  9179 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.291764  9457 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:39.307760  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1286) took 1463 ms. Trace:
I20260709 13:04:39.309237  1666 rpcz_store.cc:276] 0709 13:04:38.150013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.544004 (+393991us) service_pool.cc:224] Handling call
0709 13:04:38.559272 (+ 15268us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.572529 (+ 13257us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 13:04:38.681427 (+108898us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.078835 (+397408us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:39.094759 (+ 15924us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.098992 (+  4233us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:39.099615 (+   623us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.100109 (+   494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.100234 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.308460 (+208226us) raft_consensus.cc:1685] finished
0709 13:04:39.308488 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.308669 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13594,"prepare.queue_time_us":239013,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":2768,"raft.queue_time_us":15739,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":33933,"thread_start_us":426734,"threads_started":2,"wal-append.queue_time_us":445744}
I20260709 13:04:39.313612  1733 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:39.321676  1662 rpcz_store.cc:276] 0709 13:04:37.843986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.475289 (+631303us) service_pool.cc:224] Handling call
0709 13:04:38.489986 (+ 14697us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.618135 (+128149us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.618242 (+   107us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.618815 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 13:04:38.999027 (+380212us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:38.999064 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.999143 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.307489 (+308346us) raft_consensus.cc:1685] finished
0709 13:04:39.307522 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.307683 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":379853,"threads_started":1,"wal-append.queue_time_us":497836}
I20260709 13:04:39.315981  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:39.307027  9496 log.cc:927] Time spent T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.073s	user 0.004s	sys 0.000s
I20260709 13:04:39.317876  9302 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:39.325616  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2780) took 1397 ms (client timeout 1750 ms). Trace:
W20260709 13:04:39.346726  1733 rpcz_store.cc:269] 0709 13:04:37.925212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.064017 (+138805us) service_pool.cc:224] Handling call
0709 13:04:39.253558 (+1189541us) spinlock_profiling.cc:243] Waited 6.62 ms on lock 0x7b5800202898. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fa30a0 00007fcdf2fa2984 00007fcdec51d6d4 00007fcdec51d662
0709 13:04:39.322993 (+ 69435us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15223168}
I20260709 13:04:39.321523  9457 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:39.327060  1661 raft_consensus.cc:1240] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 13:04:39.306732  9502 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:39.334245  9475 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.293426  9474 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.264771  9250 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.335089  2316 raft_consensus.cc:2410] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 13:04:39.335923  9248 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.337141  2313 raft_consensus.cc:1240] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:39.299754  9424 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.340463  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.344858  1741 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:39.337792  8792 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.354818  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1290) took 1049 ms. Trace:
I20260709 13:04:39.383734  1661 rpcz_store.cc:276] 0709 13:04:38.305870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.318208 (+1012338us) service_pool.cc:224] Handling call
0709 13:04:39.326737 (+  8529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:39.354367 (+ 27630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.354724 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:39.381374  1739 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 13.
I20260709 13:04:39.377251  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2735) took 1313 ms. Trace:
I20260709 13:04:39.361224  9502 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:39.402357  1739 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 13:04:39.384011  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125587us
I20260709 13:04:39.380084  8855 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:39.405974  1738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.039s	sys 0.017s
W20260709 13:04:39.411098  1738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.039s	sys 0.017s
I20260709 13:04:39.416522  1739 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.11-21.12]
I20260709 13:04:39.416109  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1249) took 1594 ms. Trace:
I20260709 13:04:39.418867  1593 rpcz_store.cc:276] 0709 13:04:37.821411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.589066 (+767655us) service_pool.cc:224] Handling call
0709 13:04:38.589436 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.611820 (+ 22384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.611872 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.628341 (+ 16469us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.101620 (+473279us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:39.101748 (+   128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.101906 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.269431 (+167525us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:39.413389 (+143958us) raft_consensus.cc:1685] finished
0709 13:04:39.413418 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.416020 (+  2602us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2407,"raft.queue_time_us":5327,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":21294,"spinlock_wait_cycles":6140800,"thread_start_us":344480,"threads_started":1,"wal-append.queue_time_us":447008}
I20260709 13:04:39.387621  9248 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.404536  9501 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.414824  9500 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.437913  1739 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 13:04:39.443265  1585 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.444479  1585 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:39.432075  9393 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:39.448851  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1333) took 1147 ms. Trace:
I20260709 13:04:39.449265  1734 rpcz_store.cc:276] 0709 13:04:38.301896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.718500 (+416604us) service_pool.cc:224] Handling call
0709 13:04:38.719220 (+   720us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.719789 (+   569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.719848 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.746757 (+ 26909us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.385490 (+638733us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:39.385550 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.385624 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.448590 (+ 62966us) raft_consensus.cc:1685] finished
0709 13:04:39.448614 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.448776 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27514,"raft.queue_time_us":6004,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":16373,"thread_start_us":276361,"threads_started":1,"wal-append.queue_time_us":380554}
I20260709 13:04:39.453269  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.454161  1593 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 26.
I20260709 13:04:39.455586  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.456285  1593 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:39.457257  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.458971  1593 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:39.461417  1585 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 13:04:39.447484  9090 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.932s	user 0.015s	sys 0.000s
I20260709 13:04:39.414474  8855 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.466692  1585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3637) took 2315 ms. Trace:
I20260709 13:04:39.468459  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.467247  1585 rpcz_store.cc:276] 0709 13:04:37.146691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.154281 (+  7590us) service_pool.cc:224] Handling call
0709 13:04:37.163091 (+  8810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.442812 (+2279721us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b4c0027b5c0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:04:39.462076 (+ 19264us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.462301 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41242496}
I20260709 13:04:39.475041  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.475824  1585 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:39.475508  9227 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.476732  1593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.527804  1593 raft_consensus.cc:2393] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:39.466388  9327 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 13:04:39.480264  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.449966  9500 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.449793  9501 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.421005  9457 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:39.498396  9512 log.cc:927] Time spent T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 13:04:39.500607  9499 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.502070  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.543064  1734 raft_consensus.cc:2376] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:04:39.503446  8855 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.504601  9250 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.515018  9248 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.439267  9502 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.519407  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.535578  9455 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:39.524245  9259 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.634s	user 0.008s	sys 0.005s
I20260709 13:04:39.524905  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:39.526834  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.536844  1739 raft_consensus.cc:2376] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23. Re-sending same reply.
I20260709 13:04:39.560045  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.537709  9227 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.536473  9482 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.545746  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.466508  9090 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.552196  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.538152  9446 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.555897  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.577459  1734 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:39.537901  9305 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.564059  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1332) took 1368 ms. Trace:
I20260709 13:04:39.574636  1585 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:39.583556  1706 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.703s (SENT)
I20260709 13:04:39.576329  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.584811  1735 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 26.
I20260709 13:04:39.541198  9303 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 13:04:39.586423  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 732) took 3265 ms. Trace:
I20260709 13:04:39.587996  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.588635  1739 raft_consensus.cc:2376] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23. Re-sending same reply.
I20260709 13:04:39.586848  2315 rpcz_store.cc:276] 0709 13:04:36.314443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:36.314612 (+   169us) service_pool.cc:224] Handling call
0709 13:04:36.359994 (+ 45382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:37.046506 (+686512us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:37.046544 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:39.376568 (+2330024us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b2c00070ac8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:39.382133 (+  5565us) log.cc:844] Serialized 26 byte log entry
0709 13:04:39.425905 (+ 43772us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:39.425946 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.426083 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.571686 (+145603us) raft_consensus.cc:1685] finished
0709 13:04:39.571722 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.579866 (+  8144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":462607,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":18622,"spinlock_wait_cycles":75456384,"thread_start_us":435460,"threads_started":2,"wal-append.queue_time_us":131112}
I20260709 13:04:39.594660  1659 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:39.575557  9502 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:39.599996  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.601135  1584 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:39.594601  9167 consensus_meta.cc:287] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.581s	user 0.014s	sys 0.000s
I20260709 13:04:39.607808  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.608366  1735 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.561830  9455 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.612128  1585 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:39.612792  1585 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.619920  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.621335  1734 raft_consensus.cc:2410] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 12.
I20260709 13:04:39.605284  9407 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.589421  9302 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.578595  9482 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.589936  9446 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.638999  9305 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:39.639478  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.641266  9500 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.643683  9455 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:39.643891  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:39.644577  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.644485  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.646435  9501 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.656881  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2796) took 1169 ms. Trace:
W20260709 13:04:39.647235  1708 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.698s (SENT)
I20260709 13:04:39.653412  2313 raft_consensus.cc:2393] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:39.653455  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.657665  1733 rpcz_store.cc:276] 0709 13:04:38.487588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.362507 (+874919us) service_pool.cc:224] Handling call
0709 13:04:39.383479 (+ 20972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.383715 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:39.451126 (+ 67411us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.656139 (+205013us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.656214 (+    75us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:39.656277 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.656361 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.656797 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15103,"prepare.queue_time_us":176096,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"thread_start_us":242800,"threads_started":2}
I20260709 13:04:39.659760  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.655325  1592 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:39.660692  1734 raft_consensus.cc:2410] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
W20260709 13:04:39.627401  9106 consensus_meta.cc:287] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.545s	user 0.014s	sys 0.000s
W20260709 13:04:39.662809  1708 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.703s (SENT)
I20260709 13:04:39.665081  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.666622  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.667312  1733 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 17.
I20260709 13:04:39.663286  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:39.673593  1585 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:39.675124  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:39.678268  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.679536  1733 raft_consensus.cc:2410] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 13:04:39.681841  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:39.685315  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:39.685753  9521 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 13:04:39.686525  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.687829  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.687968  9446 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.688925  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.690061  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:39.690979  9411 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:39.691630  9411 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.692112  9411 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.678501  9318 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:39.627228  9407 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.696508  9407 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.691059  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
W20260709 13:04:39.677158  9522 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 13:04:39.704202  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.705085  1734 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:39.667166  9106 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.708349  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:39.767403  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:39.714787  1591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36806 (request call id 3644) took 1912 ms (client timeout 1762 ms). Trace:
I20260709 13:04:39.722062  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:39.784046  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 108951us
I20260709 13:04:39.709816  1733 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:39.785302  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:39.715806  9473 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.251s	user 0.004s	sys 0.006s
W20260709 13:04:39.783418  1591 rpcz_store.cc:269] 0709 13:04:37.802669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.802830 (+   161us) service_pool.cc:224] Handling call
0709 13:04:39.714505 (+1911675us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b54001fa254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.714671 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123156224}
I20260709 13:04:39.787446  1735 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:39.790294  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.734898  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:39.791313  1735 raft_consensus.cc:2376] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19. Re-sending same reply.
W20260709 13:04:39.714495  9480 log.cc:927] Time spent T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 1.094s	user 0.000s	sys 0.001s
I20260709 13:04:39.781028  9404 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:39.724102  9106 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.720167  9420 raft_consensus.cc:3055] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:39.719810  9468 raft_consensus.cc:3055] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:39.732569  9393 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:39.796147  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:39.796422  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61624.8us
I20260709 13:04:39.797009  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.797663  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.804946  1589 raft_consensus.cc:2410] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 13:04:39.806439  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.807119  1735 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:04:39.808012  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.809999  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.809231  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.811393  1735 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.814163  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:39.804250  9393 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:39.817091  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.818243  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.819128  1589 raft_consensus.cc:2410] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:39.800611  9410 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:39.823892  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.824990  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.800886  9420 raft_consensus.cc:740] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:39.826399  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.828900  1589 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 23 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:04:39.803865  9468 raft_consensus.cc:740] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:39.834918  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.835214  9468 consensus_queue.cc:260] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.833492  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.836052  1734 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 27: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:39.836700  1591 raft_consensus.cc:2410] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for term 19 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 13:04:39.837549  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.838482  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3650) took 1379 ms. Trace:
I20260709 13:04:39.838860  1591 rpcz_store.cc:276] 0709 13:04:38.458928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.837703 (+1378775us) service_pool.cc:224] Handling call
0709 13:04:39.838002 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.838175 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.838200 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.838242 (+    42us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:39.838275 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.838326 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.838429 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:39.841807  9468 raft_consensus.cc:3060] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:39.842024  1734 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:39.844751  1734 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:39.840250  1591 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:39.800765  9404 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.846931  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1253) took 1304 ms. Trace:
W20260709 13:04:39.847028  1665 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3408) took 1689 ms (client timeout 2200 ms). Trace:
I20260709 13:04:39.847370  1591 rpcz_store.cc:276] 0709 13:04:38.542616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.839828 (+1297212us) service_pool.cc:224] Handling call
0709 13:04:39.840091 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.846605 (+  6514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.846800 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:39.847510  1665 rpcz_store.cc:269] 0709 13:04:38.157805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.335994 (+178189us) service_pool.cc:224] Handling call
0709 13:04:39.846797 (+1510803us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.846967 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":653218688}
I20260709 13:04:39.848569  9404 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.848977  1591 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:39.849139  9404 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.858944  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:39.878618  2318 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1299) took 1997 ms (client timeout 1703 ms). Trace:
I20260709 13:04:39.878412  9404 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:39.879077  2318 rpcz_store.cc:269] 0709 13:04:37.881582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.904484 (+ 22902us) service_pool.cc:224] Handling call
0709 13:04:39.878285 (+1973801us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b54005ae454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.878463 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238749952}
I20260709 13:04:39.859437  9401 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:39.881735  2315 raft_consensus.cc:1240] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:39.821587  9446 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 13:04:39.881176  9401 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:04:39.882925  9401 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:39.882848  9375 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:04:39.870064  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:39.884163  9401 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.849364  9450 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:39.877398  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3529) took 1808 ms. Trace:
I20260709 13:04:39.859079  9318 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:39.861562  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.869483  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.891567  2311 raft_consensus.cc:2393] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:39.890831  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:39.893191  2318 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:39.893683  1586 rpcz_store.cc:276] 0709 13:04:38.068818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.819539 (+750721us) service_pool.cc:224] Handling call
0709 13:04:39.715971 (+896432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.716232 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.716258 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.716316 (+    58us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:39.716358 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.716438 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.877177 (+160739us) spinlock_profiling.cc:243] Waited 872 ms on lock 0x7b54001fa250. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.877315 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2005011840}
I20260709 13:04:39.895049  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:39.895778  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1255) took 1352 ms. Trace:
I20260709 13:04:39.896169  1586 rpcz_store.cc:276] 0709 13:04:38.543232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.894919 (+1351687us) service_pool.cc:224] Handling call
0709 13:04:39.895266 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.895467 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:39.895492 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.895537 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:39.895570 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.895624 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.895721 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:39.899451  2309 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1288) took 4258 ms (client timeout 2017 ms). Trace:
W20260709 13:04:39.899904  2309 rpcz_store.cc:269] 0709 13:04:35.641414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.645060 (+  3646us) service_pool.cc:224] Handling call
0709 13:04:39.898891 (+4253831us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54006a0254. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.899305 (+   414us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3879344384}
W20260709 13:04:39.901194  1664 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2719) took 2241 ms (client timeout 2069 ms). Trace:
W20260709 13:04:39.901430  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:39.901664  1664 rpcz_store.cc:269] 0709 13:04:37.660000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.088606 (+428606us) service_pool.cc:224] Handling call
0709 13:04:39.901019 (+1812413us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b5400010c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:39.901123 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516158208}
I20260709 13:04:39.902298  9283 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.871526  8791 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:39.857653  9302 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.867043  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3534) took 1771 ms. Trace:
I20260709 13:04:39.905679  1584 rpcz_store.cc:276] 0709 13:04:38.095524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.184907 (+1089383us) service_pool.cc:224] Handling call
0709 13:04:39.193463 (+  8556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.603344 (+409881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.866751 (+263407us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54000fa354. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:39.866971 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":935170432}
I20260709 13:04:39.906932  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:39.910557  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.920620  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:39.921319  1584 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:39.581456  1738 rpcz_store.cc:276] 0709 13:04:38.195349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.621801 (+426452us) service_pool.cc:224] Handling call
0709 13:04:38.622183 (+   382us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:38.622584 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.622614 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:38.654935 (+ 32321us) log.cc:844] Serialized 52 byte log entry
0709 13:04:39.411810 (+756875us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:39.563337 (+151527us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.563656 (+   319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.563744 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:39.563791 (+    47us) raft_consensus.cc:1685] finished
0709 13:04:39.563803 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.563964 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5825,"prepare.queue_time_us":162898,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":358,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":51899,"spinlock_wait_cycles":10282752,"thread_start_us":667247,"threads_started":2,"wal-append.queue_time_us":619773}
I20260709 13:04:39.922684  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.923260  1738 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:39.923374  1664 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:39.924098  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:39.925163  1667 raft_consensus.cc:1240] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 13:04:39.870604  1589 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:39.821205  9455 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:39.926810  1665 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:39.927341  1589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3552) took 1744 ms. Trace:
I20260709 13:04:39.927505  9455 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:39.927721  1589 rpcz_store.cc:276] 0709 13:04:38.182436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.870048 (+1687612us) service_pool.cc:224] Handling call
0709 13:04:39.870374 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.927078 (+ 56704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.927283 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:39.928927  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.930831  9406 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:39.932020  1661 raft_consensus.cc:1217] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Deduplicated request from leader. Original: 23.13->[25.14-25.14]   Dedup: 25.14->[]
I20260709 13:04:39.932579  1661 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 13:04:39.932749  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:39.931150  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.918112  9452 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.935370  9482 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.935273  9452 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.902859  9283 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.939226  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:39.939621  9283 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.940543  9227 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:39.940972  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:39.942008  1589 raft_consensus.cc:2410] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:39.919445  9164 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:39.863795  9424 consensus_queue.cc:1059] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 13:04:39.928671  9455 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:39.913125  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:39.929122  1735 raft_consensus.cc:2468] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:39.929757  9438 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:39.931895  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:39.917528  1705 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:39.942277  9452 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:39.914584  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:39.933420  1665 raft_consensus.cc:2468] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:39.917982  9420 consensus_queue.cc:260] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.962406  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:39.963452  1661 raft_consensus.cc:2376] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22. Re-sending same reply.
I20260709 13:04:39.965155  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:39.948676  1634 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:39.849628  1591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1254) took 1306 ms. Trace:
I20260709 13:04:39.888424  9404 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:39.944931  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 159651us
I20260709 13:04:39.946611  9482 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:39.946817  8791 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:39.970592  1584 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:40.003280  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:39.971256  1734 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:04:39.974697  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:39.996604  2281 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:39.976037  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 58630.2us
I20260709 13:04:40.007791  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
W20260709 13:04:39.991170  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44653.9us
I20260709 13:04:39.921801  9450 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:39.997772  1591 rpcz_store.cc:276] 0709 13:04:38.542948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.848481 (+1305533us) service_pool.cc:224] Handling call
0709 13:04:39.848798 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.849356 (+   558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.849574 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:39.943778  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48787.4us
I20260709 13:04:39.999581  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:39.947250  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:39.998179  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49625.1us
I20260709 13:04:40.005816  1662 raft_consensus.cc:2376] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23. Re-sending same reply.
I20260709 13:04:40.015302  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:40.006990  1705 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:40.023806  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:40.011701  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:40.016180  1741 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:40.029356  1591 raft_consensus.cc:2410] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:04:40.016495  1634 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:40.031028  1591 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 13:04:39.941004  9411 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:39.999492  9529 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:39.970185  9438 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:40.034273  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:40.038128  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:46322 (request call id 1256) took 1487 ms. Trace:
I20260709 13:04:40.038643  1586 rpcz_store.cc:276] 0709 13:04:38.550571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.028444 (+1477873us) service_pool.cc:224] Handling call
0709 13:04:40.037568 (+  9124us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.037762 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:40.037799 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:40.037855 (+    56us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:40.037897 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:40.037954 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:40.038068 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:40.009227  8791 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.031497  1591 pending_rounds.cc:85] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 10
I20260709 13:04:40.041153  1591 pending_rounds.cc:107] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 20.11
I20260709 13:04:40.041399  1591 raft_consensus.cc:2889] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:40.042764  1708 leader_election.cc:400] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:40.043402  1708 leader_election.cc:403] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:40.034755  1584 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:40.025925  9420 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:40.050768  1586 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 13:04:40.013916  9452 raft_consensus.cc:2804] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:40.011680  9106 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:40.023665  9404 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:40.037479  9283 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:40.051187  1586 pending_rounds.cc:85] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting all ops after (but not including) 11
I20260709 13:04:40.064491  1586 pending_rounds.cc:107] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 13:04:40.064905  1586 raft_consensus.cc:2889] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:40.066051  9283 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:40.066545  9283 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:40.066643  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:40.068022  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:40.068595  1586 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:40.051599  9079 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:40.032745  9526 raft_consensus.cc:2804] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:40.069850  9079 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.070348  9079 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:40.070377  9526 raft_consensus.cc:697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:40.070737  9446 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:40.071194  9446 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:40.060309  9482 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 13:04:40.071552  9526 consensus_queue.cc:237] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:40.072093  9446 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.073311  9482 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:40.073700  9482 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:40.074568  9482 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:40.075919  9482 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:40.076306  9437 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:40.069461  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:40.057299  9529 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:40.077740  1586 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 13:04:40.058583  9438 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:40.029387  9363 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 13:04:40.035089  9500 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
W20260709 13:04:40.058936  9485 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.870s	user 0.002s	sys 0.002s
I20260709 13:04:40.040874  8791 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:40.037591  9501 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 13:04:40.012914  9469 raft_consensus.cc:2804] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:40.060009  1733 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:40.061333  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:40.077873  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:40.028630  9227 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 13:04:40.092649  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:40.100813  9446 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:40.099677  2313 raft_consensus.cc:2468] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:40.093571  9404 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267) with higher term. Message: Invalid argument: T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:40.065217  9452 raft_consensus.cc:697] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:40.107901  9452 consensus_queue.cc:237] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.115077  9411 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:40.065595  9106 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.091498  9500 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:40.093356  9450 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.123415  9450 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:40.130060  9501 raft_consensus.cc:2804] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:40.119269  9106 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:40.140300  9164 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:40.130658  9501 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:40.140924  9469 raft_consensus.cc:697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:40.155568  9318 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:40.160456  9393 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.200739  9501 raft_consensus.cc:3060] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:40.165911  9164 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:40.274219  9532 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 13:04:40.318071  9393 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:40.271111  9469 consensus_queue.cc:237] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.263393  8791 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:40.895385  9543 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 13:04:40.269537  9164 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.483425  9079 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:40.470849  9283 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:40.912531  9482 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.836s	user 0.006s	sys 0.011s
I20260709 13:04:41.054944  9164 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:41.126847  9542 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.160s	user 0.002s	sys 0.000s
I20260709 13:04:41.142019  9283 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:41.183974  9540 log.cc:927] Time spent T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.264s	user 0.000s	sys 0.001s
W20260709 13:04:41.188479  9535 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 13:04:41.181933  8791 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:41.191895  9501 consensus_meta.cc:287] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.942s	user 0.008s	sys 0.006s
I20260709 13:04:41.192437  9501 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.196622  9501 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.319753  9453 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:41.323395  9453 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.335553  9453 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:41.346537  9323 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.710s	user 0.015s	sys 0.005s
W20260709 13:04:41.347100  9323 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.711s	user 0.015s	sys 0.005s
I20260709 13:04:41.347364  9323 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:41.348336  9527 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 1.547s	user 0.003s	sys 0.005s
I20260709 13:04:41.348228  9323 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:41.357180  1735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.289s	user 0.001s	sys 0.006s
W20260709 13:04:41.357856  1735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.289s	user 0.004s	sys 0.006s
I20260709 13:04:41.358777  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3564) took 1558 ms. Trace:
I20260709 13:04:41.359139  1735 rpcz_store.cc:276] 0709 13:04:39.800929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.067499 (+266570us) service_pool.cc:224] Handling call
0709 13:04:40.067811 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.068051 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:40.068321 (+   270us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.358391 (+1290070us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.358454 (+    63us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:41.358513 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.358586 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.358726 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":584,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":148,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"thread_start_us":1289637,"threads_started":1,"wal-append.queue_time_us":562996}
I20260709 13:04:41.360965  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:41.361953  1735 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42028 (request call id 1342) took 1283 ms (client timeout 1691 ms). Trace:
W20260709 13:04:41.362370  1735 rpcz_store.cc:269] 0709 13:04:40.079095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.360656 (+1281561us) service_pool.cc:224] Handling call
0709 13:04:41.361893 (+  1237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.367408  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3566) took 1560 ms. Trace:
I20260709 13:04:41.367864  1735 rpcz_store.cc:276] 0709 13:04:39.807094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.363345 (+1556251us) service_pool.cc:224] Handling call
0709 13:04:41.363733 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.363975 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:41.365176 (+  1201us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.366905 (+  1729us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.366963 (+    58us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:41.367040 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.367143 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.367323 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1313,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":101,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":136,"thread_start_us":1682,"threads_started":2,"wal-append.queue_time_us":1381}
I20260709 13:04:41.371207  1735 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 13:04:41.371996  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3567) took 1564 ms. Trace:
I20260709 13:04:41.372299  1735 rpcz_store.cc:276] 0709 13:04:39.807377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.370483 (+1563106us) service_pool.cc:224] Handling call
0709 13:04:41.370924 (+   441us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:41.371707 (+   783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.371934 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.375483  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3568) took 1567 ms. Trace:
I20260709 13:04:41.375836  1735 rpcz_store.cc:276] 0709 13:04:39.807663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.372981 (+1565318us) service_pool.cc:224] Handling call
0709 13:04:41.373299 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.373507 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:41.373869 (+   362us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.375079 (+  1210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.375164 (+    85us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:41.375233 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.375305 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.375435 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":174,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":102,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"spinlock_wait_cycles":144384,"thread_start_us":791,"threads_started":1,"wal-append.queue_time_us":1300}
I20260709 13:04:41.378015  1735 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:41.378697  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3570) took 1570 ms. Trace:
I20260709 13:04:41.379066  1735 rpcz_store.cc:276] 0709 13:04:39.808227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.377366 (+1569139us) service_pool.cc:224] Handling call
0709 13:04:41.377798 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.378437 (+   639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.378643 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.380283  1735 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:41.380824  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3571) took 1572 ms. Trace:
I20260709 13:04:41.381034  1735 rpcz_store.cc:276] 0709 13:04:39.808494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.379838 (+1571344us) service_pool.cc:224] Handling call
0709 13:04:41.380096 (+   258us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.380645 (+   549us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.380786 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.384090  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3573) took 1575 ms. Trace:
I20260709 13:04:41.384466  1735 rpcz_store.cc:276] 0709 13:04:39.808988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.381635 (+1572647us) service_pool.cc:224] Handling call
0709 13:04:41.381981 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.382250 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:41.382583 (+   333us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.383719 (+  1136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.383773 (+    54us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:41.383834 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.383901 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.384037 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":171,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":96,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":157952,"thread_start_us":810,"threads_started":1,"wal-append.queue_time_us":1179}
I20260709 13:04:41.388990  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3574) took 1579 ms. Trace:
I20260709 13:04:41.389326  1735 rpcz_store.cc:276] 0709 13:04:39.809273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.385642 (+1576369us) service_pool.cc:224] Handling call
0709 13:04:41.385971 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.386152 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 13:04:41.386472 (+   320us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.388345 (+  1873us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.388523 (+   178us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.388574 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:41.388679 (+   105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.388758 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.388918 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":602,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"raft.queue_time_us":420,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"spinlock_wait_cycles":15104,"thread_start_us":1198,"threads_started":1,"wal-append.queue_time_us":1598}
I20260709 13:04:41.391271  1735 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:41.391852  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2832) took 1553 ms. Trace:
I20260709 13:04:41.392217  1735 rpcz_store.cc:276] 0709 13:04:39.838152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.390617 (+1552465us) service_pool.cc:224] Handling call
0709 13:04:41.390896 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.391595 (+   699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.391796 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.393798  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3577) took 1534 ms. Trace:
I20260709 13:04:41.394371  1735 rpcz_store.cc:276] 0709 13:04:39.859822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.393108 (+1533286us) service_pool.cc:224] Handling call
0709 13:04:41.393347 (+   239us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.393495 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.393535 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.393566 (+    31us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:41.393594 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.393634 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.393742 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.401149  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3578) took 1541 ms. Trace:
I20260709 13:04:41.401515  1735 rpcz_store.cc:276] 0709 13:04:39.860184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.395484 (+1535300us) service_pool.cc:224] Handling call
0709 13:04:41.395846 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:41.396616 (+   770us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.396659 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:41.397388 (+   729us) log.cc:844] Serialized 52 byte log entry
0709 13:04:41.398906 (+  1518us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:41.398949 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.399022 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:41.400917 (+  1895us) raft_consensus.cc:1685] finished
0709 13:04:41.400942 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.401088 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":319,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":110,"thread_start_us":996,"threads_started":1,"wal-append.queue_time_us":1406}
I20260709 13:04:41.405045  1735 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 13:04:41.405685  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3579) took 1545 ms. Trace:
I20260709 13:04:41.406016  1735 rpcz_store.cc:276] 0709 13:04:39.860503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.404020 (+1543517us) service_pool.cc:224] Handling call
0709 13:04:41.404639 (+   619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.405390 (+   751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.405624 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.409233  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1340) took 1521 ms. Trace:
I20260709 13:04:41.409868  1735 rpcz_store.cc:276] 0709 13:04:39.888245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.406764 (+1518519us) service_pool.cc:224] Handling call
0709 13:04:41.407115 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.407364 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 13:04:41.407718 (+   354us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.408857 (+  1139us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.408910 (+    53us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:41.408966 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.409029 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.409160 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":167,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":107,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":778,"threads_started":1,"wal-append.queue_time_us":1143}
I20260709 13:04:41.412050  1735 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:41.412425  1735 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:41.413357  1735 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.414433  1735 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:41.424052  1735 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:41.424957  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2835) took 1514 ms. Trace:
I20260709 13:04:41.425355  1735 rpcz_store.cc:276] 0709 13:04:39.910412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.411598 (+1501186us) service_pool.cc:224] Handling call
0709 13:04:41.411918 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.424535 (+ 12617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.424889 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.433131  9560 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:41.433995  9560 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.435745  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1341) took 1360 ms. Trace:
I20260709 13:04:41.436084  1735 rpcz_store.cc:276] 0709 13:04:40.075587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.426223 (+1350636us) service_pool.cc:224] Handling call
0709 13:04:41.426561 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.426840 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:41.427421 (+   581us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.435453 (+  8032us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.435500 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:41.435546 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.435595 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.435698 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":122,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":100,"thread_start_us":7669,"threads_started":1}
I20260709 13:04:41.438738  9560 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:41.439358  9560 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.440394  9560 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.442945  9560 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:41.444378  1593 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:41.445158  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3656) took 1546 ms. Trace:
I20260709 13:04:41.445492  1593 rpcz_store.cc:276] 0709 13:04:39.898958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.443756 (+1544798us) service_pool.cc:224] Handling call
0709 13:04:41.444163 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.444870 (+   707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.445084 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.447847  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3657) took 1548 ms. Trace:
I20260709 13:04:41.448269  1593 rpcz_store.cc:276] 0709 13:04:39.899194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.446610 (+1547416us) service_pool.cc:224] Handling call
0709 13:04:41.446978 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.447491 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.447526 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.447577 (+    51us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:41.447612 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.447674 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.447791 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.450168  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3660) took 1550 ms. Trace:
I20260709 13:04:41.450542  1593 rpcz_store.cc:276] 0709 13:04:39.899992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.449297 (+1549305us) service_pool.cc:224] Handling call
0709 13:04:41.449645 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.449823 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.449850 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.449895 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:41.449931 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.449987 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.450105 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.452205  1593 raft_consensus.cc:1240] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:41.453868  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3661) took 1553 ms. Trace:
I20260709 13:04:41.454258  1593 rpcz_store.cc:276] 0709 13:04:39.900243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.451693 (+1551450us) service_pool.cc:224] Handling call
0709 13:04:41.452009 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.453594 (+  1585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.453817 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.456004  1593 raft_consensus.cc:1240] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 13:04:41.456988  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3662) took 1556 ms. Trace:
I20260709 13:04:41.457386  1593 rpcz_store.cc:276] 0709 13:04:39.900490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.455047 (+1554557us) service_pool.cc:224] Handling call
0709 13:04:41.455722 (+   675us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:41.456477 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.456926 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.459733  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3665) took 1558 ms. Trace:
I20260709 13:04:41.460176  1593 rpcz_store.cc:276] 0709 13:04:39.901306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.458219 (+1556913us) service_pool.cc:224] Handling call
0709 13:04:41.458608 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.459331 (+   723us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.459366 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.459421 (+    55us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:41.459466 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.459538 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.459675 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.462805  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3668) took 1417 ms. Trace:
I20260709 13:04:41.463472  1593 rpcz_store.cc:276] 0709 13:04:40.045608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.461501 (+1415893us) service_pool.cc:224] Handling call
0709 13:04:41.461887 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.462093 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.462119 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.462165 (+    46us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:41.462534 (+   369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.462616 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.462744 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.472007  1593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3672) took 1369 ms. Trace:
I20260709 13:04:41.472419  1593 rpcz_store.cc:276] 0709 13:04:40.102540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.464888 (+1362348us) service_pool.cc:224] Handling call
0709 13:04:41.465374 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:41.465830 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:41.465866 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:41.467246 (+  1380us) log.cc:844] Serialized 26 byte log entry
0709 13:04:41.469061 (+  1815us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:41.469101 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.469214 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:41.471498 (+  2284us) raft_consensus.cc:1685] finished
0709 13:04:41.471528 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.471737 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1467,"threads_started":1,"wal-append.queue_time_us":1878}
W20260709 13:04:41.491174  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.493039  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:41.499307  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2755) took 1551 ms (client timeout 1656 ms). Trace:
W20260709 13:04:41.499727  1662 rpcz_store.cc:269] 0709 13:04:39.948114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.492648 (+1544534us) service_pool.cc:224] Handling call
0709 13:04:41.499211 (+  6563us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.508342  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.509279  1662 raft_consensus.cc:2393] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:41.510667  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2762) took 1559 ms (client timeout 1706 ms). Trace:
W20260709 13:04:41.511157  1662 rpcz_store.cc:269] 0709 13:04:39.951633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.507977 (+1556344us) service_pool.cc:224] Handling call
0709 13:04:41.510597 (+  2620us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.512413  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:41.513455  1662 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2763) took 1416 ms (client timeout 1729 ms). Trace:
W20260709 13:04:41.509994  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.515614  9097 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.516148  9097 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.516732  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.517728  9549 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.518322  9549 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.517449  1661 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:41.527311  9549 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.529843  1661 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:41.529608  9097 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.534272  9190 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.542163  9190 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.544646  9190 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.545256  9190 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.545938  9190 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.548278  9190 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:41.549240  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.549607  9190 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.550366  9457 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:41.550189  9190 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:41.551746  1632 connection.cc:256] RPC call timeout handler was delayed by -1.2941540820000341s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17490000000000003s for a response
W20260709 13:04:41.552352  1632 connection.cc:256] RPC call timeout handler was delayed by -1.2824540820000152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17620000000000002s for a response
I20260709 13:04:41.548326  9563 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.548985  9097 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:41.553121  9097 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:41.552524  1661 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2760) took 1602 ms (client timeout 2022 ms). Trace:
W20260709 13:04:41.554335  1661 rpcz_store.cc:269] 0709 13:04:39.950840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.516389 (+1565549us) service_pool.cc:224] Handling call
0709 13:04:41.552450 (+ 36061us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.555835  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.556680  1661 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:41.538734  9560 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.557325  9097 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.558189  9097 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:41.557878  9560 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.558696  9097 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.560640  9560 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:41.560767  9097 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.553684  9190 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:41.513906  1662 rpcz_store.cc:269] 0709 13:04:40.097071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.511983 (+1414912us) service_pool.cc:224] Handling call
0709 13:04:41.513396 (+  1413us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.571213  1662 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:41.572124  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2759) took 1622 ms. Trace:
I20260709 13:04:41.572468  1662 rpcz_store.cc:276] 0709 13:04:39.949292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.570451 (+1621159us) service_pool.cc:224] Handling call
0709 13:04:41.570816 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.571654 (+   838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.572056 (+   402us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.574106  1662 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.11-21.12]
I20260709 13:04:41.574992  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2764) took 1477 ms. Trace:
I20260709 13:04:41.575701  1662 rpcz_store.cc:276] 0709 13:04:40.097410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.573416 (+1476006us) service_pool.cc:224] Handling call
0709 13:04:41.573815 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:41.574704 (+   889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.574937 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:41.577188  1662 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:04:41.557307  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2757) took 1608 ms. Trace:
W20260709 13:04:41.561007  1659 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2722) took 3883 ms (client timeout 1595 ms). Trace:
W20260709 13:04:41.557415  1632 connection.cc:256] RPC call timeout handler was delayed by -1.242854082000008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1806s for a response
I20260709 13:04:41.561717  9563 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:41.577243  1592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36802 (request call id 3576) took 1915 ms (client timeout 1850 ms). Trace:
I20260709 13:04:41.586608  1661 rpcz_store.cc:276] 0709 13:04:39.948734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.555455 (+1606721us) service_pool.cc:224] Handling call
0709 13:04:41.557120 (+  1665us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:41.590261  1592 rpcz_store.cc:269] 0709 13:04:39.662104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.690842 (+ 28738us) service_pool.cc:224] Handling call
0709 13:04:41.577014 (+1886172us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54001c3754. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:41.577164 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4317607808}
I20260709 13:04:41.591305  1661 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 13:04:41.591908  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2766) took 1493 ms. Trace:
I20260709 13:04:41.592227  1661 rpcz_store.cc:276] 0709 13:04:40.098014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.590551 (+1492537us) service_pool.cc:224] Handling call
0709 13:04:41.590916 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.591670 (+   754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.591854 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:41.592665  1667 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2761) took 1641 ms (client timeout 1570 ms). Trace:
W20260709 13:04:41.586724  1659 rpcz_store.cc:269] 0709 13:04:37.678108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:37.685104 (+  6996us) service_pool.cc:224] Handling call
0709 13:04:41.560711 (+3875607us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b5400014ad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:41.560929 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4313893760}
I20260709 13:04:41.587527  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2765) took 1489 ms. Trace:
W20260709 13:04:41.586977  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0475540820000333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20230000000000004s for a response
I20260709 13:04:41.593537  9563 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:41.593670  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0403540820000217s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
I20260709 13:04:41.594086  1662 rpcz_store.cc:276] 0709 13:04:40.097720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:41.576521 (+1478801us) service_pool.cc:224] Handling call
0709 13:04:41.576867 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.586426 (+  9559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.587464 (+  1038us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:41.603420  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0034540820000188s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
I20260709 13:04:41.597951  1588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3633) took 5669 ms. Trace:
I20260709 13:04:41.604225  1632 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:41.593065  1667 rpcz_store.cc:269] 0709 13:04:39.951240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.077573 (+126333us) service_pool.cc:224] Handling call
0709 13:04:41.592458 (+1514885us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b5400075dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:41.592587 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3451986688}
I20260709 13:04:41.602488  9097 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:41.594660  1705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.555s	user 0.003s	sys 0.010s
I20260709 13:04:41.604439  1588 rpcz_store.cc:276] 0709 13:04:35.928163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:35.984594 (+ 56431us) service_pool.cc:224] Handling call
0709 13:04:35.984938 (+   344us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:36.306115 (+321177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:36.306169 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:36.318313 (+ 12144us) log.cc:844] Serialized 52 byte log entry
0709 13:04:38.311032 (+1992719us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b18003cb920. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:38.412416 (+101384us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:38.412461 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.597533 (+3185072us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54002184d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:41.597600 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:41.597665 (+    65us) raft_consensus.cc:1685] finished
0709 13:04:41.597678 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.597860 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84182,"raft.queue_time_us":5117,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":17870,"spinlock_wait_cycles":498187904,"thread_start_us":101069,"threads_started":1,"wal-append.queue_time_us":2142286,"wal-append.run_cpu_time_us":1871,"wal-append.run_wall_time_us":1002827}
I20260709 13:04:41.607733  9097 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.608644  9550 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.608675  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:41.609790  9457 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:41.609422  9550 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.610351  9457 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:41.610500  9097 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:41.610764  1739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.578s	user 0.043s	sys 0.049s
I20260709 13:04:41.607498  1586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3655) took 1708 ms. Trace:
I20260709 13:04:41.611581  9457 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.612368  9550 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.611577  1586 rpcz_store.cc:276] 0709 13:04:39.898713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.091152 (+192439us) service_pool.cc:224] Handling call
0709 13:04:41.568630 (+1477478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:41.568902 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:41.569218 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.607203 (+ 37985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.607249 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:41.607300 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.607355 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.607449 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":140,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"thread_start_us":37183,"threads_started":1}
I20260709 13:04:41.604089  9482 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:41.614146  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.613803  9482 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:41.614986  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:41.615658  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.616514  9482 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.617089  9190 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:41.617661  1705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.578s	user 0.003s	sys 0.012s
W20260709 13:04:41.618248  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 1.58602e+06us
I20260709 13:04:41.617705  9190 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:41.611132  1739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.578s	user 0.043s	sys 0.049s
I20260709 13:04:41.621217  9190 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.623076  9190 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:41.624214  9190 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:41.607425  9563 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:41.620398  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3562) took 1830 ms. Trace:
I20260709 13:04:41.625456  9079 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.626905  1739 rpcz_store.cc:276] 0709 13:04:39.790032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.995656 (+205624us) service_pool.cc:224] Handling call
0709 13:04:40.000369 (+  4713us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.000644 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:40.001137 (+   493us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.620019 (+1618882us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.620080 (+    61us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:41.620144 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.620217 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.620338 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":169,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":11542,"thread_start_us":1586883,"threads_started":1,"wal-append.queue_time_us":960102}
I20260709 13:04:41.622349  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:41.630806  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:39.388398  1660 rpcz_store.cc:276] 0709 13:04:38.063874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.517663 (+453789us) service_pool.cc:224] Handling call
0709 13:04:38.517975 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.518197 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:39.192589 (+674392us) log.cc:844] Serialized 14 byte log entry
0709 13:04:39.376819 (+184230us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:39.376877 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:39.376946 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:39.377019 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:39.377178 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":163175,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":3197,"raft.queue_time_us":1230,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":7561,"thread_start_us":779649,"threads_started":2}
I20260709 13:04:41.632577  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:41.633692  9563 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:41.634181  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.634398  9563 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:04:41.636677  9563 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:41.637393  9190 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:41.637683  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2729) took 3929 ms. Trace:
I20260709 13:04:41.638105  1668 rpcz_store.cc:276] 0709 13:04:37.708185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.455646 (+747461us) service_pool.cc:224] Handling call
0709 13:04:38.476221 (+ 20575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:38.480310 (+  4089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.480344 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:38.517886 (+ 37542us) log.cc:844] Serialized 26 byte log entry
0709 13:04:38.998500 (+480614us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:38.998558 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:38.998644 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:40.057939 (+1059295us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:41.637314 (+1579375us) raft_consensus.cc:1685] finished
0709 13:04:41.637341 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.637593 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3665,"raft.queue_time_us":27039,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":92,"spinlock_wait_cycles":8250752,"thread_start_us":473341,"threads_started":1,"wal-append.queue_time_us":620400,"wal-append.run_cpu_time_us":7387,"wal-append.run_wall_time_us":1972234}
I20260709 13:04:41.638911  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:41.641808  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:41.644933  2316 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:41.646855  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:41.647584  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.652043  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.652951  1668 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 21: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:41.653841  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.654680  1660 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:41.654569  9079 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:41.656512  1667 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:41.658244  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:41.667977  1586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:41.670684  9482 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:41.671273  9482 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.672691  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:41.673936  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:41.674789  1588 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:41.675662  1588 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:41.678826  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58316 (request call id 1310) took 1976 ms. Trace:
I20260709 13:04:41.679296  2315 rpcz_store.cc:276] 0709 13:04:39.702336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.702566 (+   230us) service_pool.cc:224] Handling call
0709 13:04:39.881473 (+178907us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.882241 (+   768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.678600 (+1796359us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54005ae450. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:41.678756 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":401188096}
I20260709 13:04:41.680132  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:41.681080  2316 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:41.683457  1589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:41.684120  1589 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:41.679163  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:41.685004  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:41.685755  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.687515  1667 raft_consensus.cc:1240] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:41.689421  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:41.690091  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a"
I20260709 13:04:41.691641  1660 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:41.692415  1667 raft_consensus.cc:1240] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:41.693696  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:41.694326  1592 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 18.
I20260709 13:04:41.695513  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.697441  1592 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:41.699133  1591 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "316f3bf18ed94e32b238130a6bfedf1a" is_pre_election: true
I20260709 13:04:41.699975  1591 raft_consensus.cc:2410] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 13:04:41.700078  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.701406  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:41.701614  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.702116  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.702211  1659 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:41.704715  9482 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
/apache/dev/git/kudu/src/kudu/master/auto_leader_rebalancer-test.cc:278: Failure
Failed
Bad status: Illegal state: leader_transfer_task size should be 0, but 5
/apache/dev/git/kudu/src/kudu/util/test_util.cc:402: Failure
Failed
Timed out waiting for assertion to pass.
I20260709 13:04:41.708854   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 13:04:41.709869  1592 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:41.712715  2283 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:41.713145  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.721377  1589 raft_consensus.cc:2468] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
W20260709 13:04:41.749168  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:41.757926  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:41.758926  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.759949  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.760334  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.761824  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.766273  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:41.769001  1659 raft_consensus.cc:2393] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:41.769284  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.770160  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:41.770991  2281 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.691s (SENT) [suppressed 45 similar messages]
I20260709 13:04:41.771042  1667 raft_consensus.cc:2393] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
I20260709 13:04:41.771128  1661 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:41.774907  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.776051  1661 raft_consensus.cc:2393] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 27: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:41.776939  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64376.3us
I20260709 13:04:41.777033  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.796663  1661 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:41.779695  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.798494  1660 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 17.
I20260709 13:04:41.779218  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.806653  1667 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.777201  9570 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:41.803205  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.811883  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.812747  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.813812  1662 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:41.814278  1661 raft_consensus.cc:1275] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:41.778117  9560 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:41.816149  1659 raft_consensus.cc:1275] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 13:04:41.817960  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.818724  1659 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:41.819049  1664 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.876s	user 0.004s	sys 0.017s
W20260709 13:04:41.819536  1664 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.877s	user 0.004s	sys 0.017s
I20260709 13:04:41.822795  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.825490  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.830040  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.811205  9570 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:41.858183  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:41.860819  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 89847.2us
I20260709 13:04:41.838775  9570 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:41.864609  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.865401  1660 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:41.866752  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.870545  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2750) took 1938 ms. Trace:
I20260709 13:04:41.862475  9572 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:41.872467  9572 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:41.872867  9572 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:41.865912  9097 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:41.875451  9549 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:41.876094  9549 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.870923  1664 rpcz_store.cc:276] 0709 13:04:39.931729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.941642 (+  9913us) service_pool.cc:224] Handling call
0709 13:04:39.942009 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.942274 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:41.819921 (+1877647us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.863317 (+ 43396us) log.cc:844] Serialized 14 byte log entry
0709 13:04:41.863947 (+   630us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:41.864000 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:41.864060 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.864190 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:41.870474 (+  6284us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":158949,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":98,"raft.queue_time_us":921,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":5915,"spinlock_wait_cycles":3381248,"thread_start_us":1919284,"threads_started":2}
I20260709 13:04:41.887753  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.897868  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:41.904789  1668 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:41.906876  1587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36802 (request call id 3527) took 3849 ms. Trace:
I20260709 13:04:41.907356  1587 rpcz_store.cc:276] 0709 13:04:38.057427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.747578 (+690151us) service_pool.cc:224] Handling call
0709 13:04:38.845595 (+ 98017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:39.283860 (+438265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:41.906625 (+2622765us) spinlock_profiling.cc:243] Waited 431 ms on lock 0x7b540025dd54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:41.906804 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":992432384}
I20260709 13:04:41.906915  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.909762  1705 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:41.914109  1667 raft_consensus.cc:2468] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
W20260709 13:04:41.918018  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:41.918999  2283 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:41.935665  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:41.937470  1664 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 13:04:41.923070  9573 raft_consensus.cc:2764] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 13:04:41.949829  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.950418  9573 raft_consensus.cc:476] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Not starting pre-election -- already a leader
I20260709 13:04:41.948638  9572 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.970021  9572 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:41.971760  1668 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 13:04:41.952680  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.979700  1667 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:41.983284  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:41.984369  1664 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:41.951211  9573 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:41.987228  9573 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:41.988137  9573 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:41.985987  9581 raft_consensus.cc:2804] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:41.990705  9581 raft_consensus.cc:697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:41.970147  9549 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:41.998597  9581 consensus_queue.cc:237] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:41.999435  9305 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:41.998953  9549 consensus_queue.cc:1059] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:41.999842  9305 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:42.000679  9549 raft_consensus.cc:3055] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:42.001314  9549 raft_consensus.cc:740] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:42.003106  9549 consensus_queue.cc:260] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.005062  9549 raft_consensus.cc:3060] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:42.011826  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92942.1us
W20260709 13:04:42.016436  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80885.2us
W20260709 13:04:42.019663  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.019598  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.024706  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:42.024745  9586 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:42.025484  9586 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:42.026050  9586 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.026539  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.028079  9590 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:42.028618  9590 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:42.029243  9590 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:42.008934  9570 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.000830  9305 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.041036  9579 consensus_queue.cc:1059] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:04:42.041745  9305 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:42.043129  9583 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.043908  9583 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.048352  2280 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.056834  2309 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:04:42.068372  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:42.099889  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.110013  1734 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:42.110060  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.159299  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.109112  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1331) took 4134 ms. Trace:
I20260709 13:04:42.159766  1740 rpcz_store.cc:276] 0709 13:04:37.975050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:38.471252 (+496202us) service_pool.cc:224] Handling call
0709 13:04:38.471466 (+   214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:38.505895 (+ 34429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:38.505930 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:38.505971 (+    41us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:38.506004 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:42.108808 (+3602804us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b5400404054. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:42.108862 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:42.109031 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78675328}
W20260709 13:04:42.081883  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.141739  2309 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 18. Current term is 19. Ops: []
W20260709 13:04:42.088264  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.063851  9590 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.168987  9590 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.082979  9583 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:42.071414  9570 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.170979  9590 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:42.171694  1557 connection.cc:256] RPC call timeout handler was delayed by -1.9845655419999844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1892s for a response
W20260709 13:04:42.171345  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.178459  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.181902  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 122717us
I20260709 13:04:42.184604  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:42.185613  2309 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:42.066361  9579 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:42.064881  9587 raft_consensus.cc:3055] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:42.109483  9586 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.070976  9585 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.096s
I20260709 13:04:42.068023  9190 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.098s
I20260709 13:04:42.219986  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:42.170048  9274 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:42.178509  9375 raft_consensus.cc:3055] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:42.155139  9573 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.187906  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.164983  9591 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:42.190364  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:42.227666  1660 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.224948  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.222944  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:42.224617  1662 raft_consensus.cc:1240] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 13:04:42.224864  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.064602  9400 consensus_queue.cc:1059] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.129s
I20260709 13:04:42.233603  2310 raft_consensus.cc:2393] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:42.217485  9283 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.214458  9587 raft_consensus.cc:740] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:42.242755  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 174465us
W20260709 13:04:42.245514  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.225436  9375 raft_consensus.cc:740] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:42.247013  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.248162  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:42.247502  2283 leader_election.cc:336] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.251025  1634 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:42.236783  9563 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:42.253901  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:42.254091  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81769.6us
I20260709 13:04:42.256598  1634 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:42.255656  1557 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.716s (SENT)
W20260709 13:04:42.252117  2283 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.252723  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:42.258502  1661 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:42.251721  9274 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.257843  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.267524  1660 raft_consensus.cc:2468] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:42.266377  9274 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.254181  9563 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:42.271552  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.290943  1557 connection.cc:256] RPC call timeout handler was delayed by -1.8045655420000344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 13:04:42.291371  1557 connection.cc:256] RPC call timeout handler was delayed by -1.7217655420000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21840000000000004s for a response
W20260709 13:04:42.292076  1557 connection.cc:256] RPC call timeout handler was delayed by -1.5920309819999829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 13:04:42.292435  1557 connection.cc:256] RPC call timeout handler was delayed by -1.5445981159999747s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1595s for a response
I20260709 13:04:42.291280  9563 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.271991  9593 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 13:04:42.272130  9591 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.244453  9283 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.299259  9593 raft_consensus.cc:2764] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 13:04:42.272435  9323 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.273830  9586 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:42.256289  9457 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:42.274483  9587 consensus_queue.cc:260] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.275728  8791 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.309957  9587 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:42.278607  9375 consensus_queue.cc:260] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.266470  9305 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 13:04:42.292742  1557 connection.cc:256] RPC call timeout handler was delayed by -1.4133563079999476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.165s for a response
I20260709 13:04:42.293136  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:42.315886  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.271899  9274 raft_consensus.cc:2804] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:42.297432  2309 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
W20260709 13:04:42.321171  1557 connection.cc:256] RPC call timeout handler was delayed by -1.4081060380000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 13:04:42.326031  1557 connection.cc:256] RPC call timeout handler was delayed by -1.2941696780000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
W20260709 13:04:42.326370  1557 connection.cc:256] RPC call timeout handler was delayed by -1.2198695719999932s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260709 13:04:42.326622  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1780696779999857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1879s for a response
W20260709 13:04:42.326882  1557 connection.cc:256] RPC call timeout handler was delayed by -1.17446967799998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18830000000000002s for a response
W20260709 13:04:42.327126  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1664875100000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 13:04:42.327373  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1664875100000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 13:04:42.327617  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1598620709999636s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20260709 13:04:42.327867  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1504099069999825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15180000000000002s for a response
W20260709 13:04:42.328104  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1423099069999694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
W20260709 13:04:42.328344  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1419443209999827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
W20260709 13:04:42.328565  1557 connection.cc:256] RPC call timeout handler was delayed by -1.1348696779999727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260709 13:04:42.328796  1557 connection.cc:256] RPC call timeout handler was delayed by -1.0070696779999935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
I20260709 13:04:42.332799  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:42.335127  9586 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:42.336493  9586 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:42.336565  9573 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.337772  9274 raft_consensus.cc:697] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:42.338186  9586 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.339305  9274 consensus_queue.cc:237] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:42.345032  1557 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Timed out: RequestConsensusVote RPC to 127.0.107.196:35615 timed out after 1.570s (SENT)
I20260709 13:04:42.314446  9570 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.346724  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.312582  9323 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:42.334457  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.361004  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.361701  9164 consensus_queue.cc:1059] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:42.351105  9305 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:42.363130  9305 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.363431  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.363965  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:42.349963  9570 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.366600  9570 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:42.367218  9590 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:42.343066  9563 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:42.368250  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:42.368865  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.339303  9283 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:42.351814  1708 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.300s	user 0.004s	sys 0.011s
I20260709 13:04:42.353143  9323 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:42.363584  9305 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:42.369197  1660 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 27.
W20260709 13:04:42.348557  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92025us
W20260709 13:04:42.371230  1708 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.319s	user 0.004s	sys 0.011s
W20260709 13:04:42.377959  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 2.33724e+06us
I20260709 13:04:42.378517  1634 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:42.378942  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.381170  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.382007  9549 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:42.382323  9164 raft_consensus.cc:3055] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:42.382452  9283 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:42.382777  9549 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:42.383037  9283 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:42.382776  9164 raft_consensus.cc:740] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:42.383714  9283 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.385939  9598 consensus_queue.cc:1059] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:42.387236  9457 raft_consensus.cc:3055] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:42.384765  9164 consensus_queue.cc:260] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.387598  9457 raft_consensus.cc:740] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:42.388676  9164 raft_consensus.cc:3060] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:42.389972  9457 consensus_queue.cc:260] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.391304  9457 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:42.394268  9283 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.396843  9590 consensus_queue.cc:1048] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 13:04:42.398774  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:42.399003  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.399643  9590 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:42.400064  9590 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.400476  9590 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:42.403013  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:42.371491  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118815us
I20260709 13:04:42.379406  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.407121  9549 consensus_queue.cc:1059] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 13:04:42.412132  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.421579  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.431002  9593 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:42.447219  9323 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:42.448973  1584 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.367s	user 0.009s	sys 0.003s
W20260709 13:04:42.449576  1584 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.367s	user 0.009s	sys 0.004s
I20260709 13:04:42.450284  9190 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:42.450678  9190 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:42.451550  9190 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.429292  9599 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:42.452854  9599 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.452975  9190 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:42.453322  9599 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:42.396842  9283 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:42.460045  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47989.7us
I20260709 13:04:42.461163  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.430176  1705 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:42.498863  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:42.460064  9549 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:42.510356  1738 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:42.512548  1634 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.518162  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.519183  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 110543us
I20260709 13:04:42.521090  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.513264  9549 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.500885  1584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36806 (request call id 3652) took 2603 ms. Trace:
W20260709 13:04:42.501758  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800289c00 after lost signal to thread 434
W20260709 13:04:42.505020  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2830) took 2783 ms (client timeout 1968 ms). Trace:
I20260709 13:04:42.461252  9593 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:42.541765  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 111645us
W20260709 13:04:42.540822  1733 rpcz_store.cc:269] 0709 13:04:39.721982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.785723 (+ 63741us) service_pool.cc:224] Handling call
0709 13:04:42.504772 (+2719049us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400403654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:42.504922 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":575463808}
W20260709 13:04:42.540632  1634 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.540400  1584 rpcz_store.cc:276] 0709 13:04:39.897928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.044759 (+146831us) service_pool.cc:224] Handling call
0709 13:04:40.045193 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:40.045457 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:42.449978 (+2404521us) log.cc:844] Serialized 14 byte log entry
0709 13:04:42.500530 (+ 50552us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:42.500587 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:42.500633 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:42.500705 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:42.500826 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1097312,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":2635,"raft.queue_time_us":2047,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":5347584,"thread_start_us":2417699,"threads_started":2,"wal-append.queue_time_us":48462}
I20260709 13:04:42.540768  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.724910  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.749693  1733 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:42.750064  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 209355us
I20260709 13:04:42.725244  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.750936  1742 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:42.725931  9593 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.754253  9593 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.755702  9305 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.758127  9305 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:42.726346  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.759539  1740 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:42.760270  9591 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:42.760656  9601 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:42.543573  9606 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 13:04:42.609668  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.761582  1741 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:42.727515  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.762472  1739 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:42.728758  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.763671  1737 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:42.542392  9549 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 13 FOLLOWER]: Advancing to term 14
W20260709 13:04:42.614323  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.619669  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.770419  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.770848  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.771432  1740 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
W20260709 13:04:42.586258  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 125163us
W20260709 13:04:42.738462  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.764842  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 217224us
I20260709 13:04:42.771504  1735 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:42.751459  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.776588  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:42.777232  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.778144  1735 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:42.779337  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:42.780073  1735 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.781533  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.756983  9590 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.788318  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:42.791004  2280 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.791785  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.791816  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.793465  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:42.795208  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:42.799149  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.800489  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:42.801118  1661 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:42.802549  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.802805  9606 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:42.802881  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:42.803573  9606 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:42.803936  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.804411  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:42.804764  1737 raft_consensus.cc:2393] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:42.805142  1664 raft_consensus.cc:2368] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24.
I20260709 13:04:42.774806  9593 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:42.805653  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:42.806257  1737 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.805955  9593 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.806766  1663 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:42.808501  9593 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.808997  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:42.809635  2313 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:42.809983  9610 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:42.810492  9610 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.810887  9610 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:42.800997  9323 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 13:04:42.789772  9591 raft_consensus.cc:2804] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:42.814487  9591 raft_consensus.cc:697] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:42.814788  9531 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 2.550s	user 0.003s	sys 0.023s
I20260709 13:04:42.815565  9591 consensus_queue.cc:237] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.816923  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3558) took 3085 ms. Trace:
I20260709 13:04:42.817312  1738 rpcz_store.cc:276] 0709 13:04:39.738239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.926514 (+188275us) service_pool.cc:224] Handling call
0709 13:04:42.510075 (+2583561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:42.510914 (+   839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:42.816699 (+305785us) spinlock_profiling.cc:243] Waited 2.58 s on lock 0x7b5400403650. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:42.816857 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5940073856}
I20260709 13:04:42.818699  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.819711  9501 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:42.820240  9501 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:42.820501  1738 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 13:04:42.820888  9501 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:42.802549  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:42.821523  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.803643  1662 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:42.823230  1733 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:42.825650  9590 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.829686  9560 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:42.829949  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800308d80 after lost signal to thread 1525
I20260709 13:04:42.830646  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.831266  1740 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:42.829056  1737 raft_consensus.cc:2468] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:42.820917  1738 pending_rounds.cc:85] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 14
I20260709 13:04:42.805207  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.832667  1738 pending_rounds.cc:107] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:04:42.834136  1739 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:42.834412  1738 raft_consensus.cc:2889] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:42.836097  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 60103.4us
I20260709 13:04:42.837141  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:42.845436  1705 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:42.845754  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 43598us
I20260709 13:04:42.846645  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.846980  2313 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
W20260709 13:04:42.847512  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45989.1us
I20260709 13:04:42.847934  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:42.848212  1634 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.848896  1738 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 13:04:42.847560  2313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1131) took 1211 ms. Trace:
I20260709 13:04:42.849857  2313 rpcz_store.cc:276] 0709 13:04:41.635868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.808727 (+1172859us) service_pool.cc:224] Handling call
0709 13:04:42.847496 (+ 38769us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:42.853736  9164 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:42.853780  1735 raft_consensus.cc:2468] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:42.858671  1739 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 13:04:42.861203  1634 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.868072  1662 raft_consensus.cc:2468] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:42.788056  9608 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:42.842289  9560 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:42.875284  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.876843  1739 raft_consensus.cc:1275] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 13:04:42.832650  9601 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:42.883876  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:42.797308  9305 consensus_queue.cc:1059] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.079s
I20260709 13:04:42.903756  1738 raft_consensus.cc:1275] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 13:04:42.904168  1738 pending_rounds.cc:85] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 14
I20260709 13:04:42.904410  1738 pending_rounds.cc:107] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 13:04:42.904654  1738 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:42.802672  9609 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.084s
I20260709 13:04:42.906404  1661 raft_consensus.cc:2468] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:42.881603  9612 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:42.908742  1634 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:42.917562  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:42.915145  2281 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:42.804785  9606 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:42.928318  9585 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:04:42.939188  9563 consensus_meta.cc:287] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.568s	user 0.010s	sys 0.008s
I20260709 13:04:42.886448  9608 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.902352  9572 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 13:04:42.962920  9164 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:42.927510  9612 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:42.912657  9599 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:42.975205  9501 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:43.048287  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800337f00 after lost signal to thread 1617
I20260709 13:04:42.982137  9594 raft_consensus.cc:2804] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:43.050282  9594 raft_consensus.cc:697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:43.051625  9560 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.051400  9594 consensus_queue.cc:237] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:43.052850  9607 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.554s	user 0.000s	sys 0.004s
W20260709 13:04:42.982299  9603 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
W20260709 13:04:43.049741  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08002c6fa0 after lost signal to thread 1692
I20260709 13:04:42.998009  1736 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:43.046520  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:43.055426  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08002c6f80 after lost signal to thread 1693
W20260709 13:04:43.061671  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08003f6f60 after lost signal to thread 1766
I20260709 13:04:43.020741  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3563) took 3227 ms. Trace:
I20260709 13:04:43.011137  1666 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:43.063985  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 226926us
W20260709 13:04:42.965798  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 146761us
I20260709 13:04:43.064059  1734 rpcz_store.cc:276] 0709 13:04:39.793613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:40.034048 (+240435us) service_pool.cc:224] Handling call
0709 13:04:40.049451 (+ 15403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:42.163285 (+2113834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:43.020484 (+857199us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b5400434c54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:43.020676 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4737565056}
W20260709 13:04:43.066315  2339 debug-util.cc:398] Leaking SignalData structure 0x7b0800242a40 after lost signal to thread 1767
W20260709 13:04:42.995283  1634 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.067202  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 71958.8us
W20260709 13:04:43.070050  2339 debug-util.cc:398] Leaking SignalData structure 0x7b08000c1080 after lost signal to thread 2342
I20260709 13:04:43.071120  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:43.069267  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 222685us
W20260709 13:04:43.071092  1634 leader_election.cc:336] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:42.982547  9613 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.014298  9375 raft_consensus.cc:3060] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.031139  8791 raft_consensus.cc:3055] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:42.998950  9599 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:42.995352  9612 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:43.025767  9323 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:43.075364  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:42.987650  9610 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.083621  9323 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:43.081544  9597 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:43.076700  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 151009us
I20260709 13:04:43.077689  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:42.990177  9608 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:43.073642  1634 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.015955  9549 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.079367  8791 raft_consensus.cc:740] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:43.083155  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.065659  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.100656  9616 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:43.073621  9457 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 13:04:43.085420  9259 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:43.083949  9612 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.130023  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.091542  9599 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.085206  9593 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:43.106424  1708 leader_election.cc:304] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.088008  9573 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.087275  9501 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.115557  1741 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:04:42.994683  9164 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.058007  9601 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:43.145569  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.150447  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:43.148718  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:43.129653  9457 consensus_queue.cc:1243] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 13:04:43.119437  1735 raft_consensus.cc:1275] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:04:43.120987  1737 raft_consensus.cc:1275] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 13:04:43.124487  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:43.175992  1667 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:43.125348  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:43.126462  1661 raft_consensus.cc:1240] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:43.127612  1662 raft_consensus.cc:1275] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
I20260709 13:04:43.121071  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.080794  9613 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.181108  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.183470  1708 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:43.132512  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49477.1us
W20260709 13:04:43.187316  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.188356  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:43.190217  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:43.103274  1634 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:43.191107  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87900.3us
W20260709 13:04:43.191916  1634 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.192962  1634 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.134399  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.099493  9597 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.138316  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.140132  1660 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:43.200946  1660 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:43.098479  9305 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.142786  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:43.152081  1664 raft_consensus.cc:2393] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:43.117175  9608 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.102783  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.148775  9469 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.010s
I20260709 13:04:43.159415  9590 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:43.160105  8791 consensus_queue.cc:260] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:43.167156  1666 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49484 (request call id 2754) took 3221 ms (client timeout 1671 ms). Trace:
I20260709 13:04:43.209914  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:43.210338  1666 rpcz_store.cc:269] 0709 13:04:39.946146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:39.946898 (+   752us) service_pool.cc:224] Handling call
0709 13:04:43.166921 (+3220023us) spinlock_profiling.cc:243] Waited 3 s on lock 0x7b540013d3d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:43.167068 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6891798784}
I20260709 13:04:43.170802  9501 raft_consensus.cc:3055] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:43.171416  9616 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:43.128966  9259 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.178687  1705 leader_election.cc:400] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24.
I20260709 13:04:43.219566  1705 leader_election.cc:403] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:43.137130  9274 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:43.104565  9582 consensus_queue.cc:1048] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 13:04:43.116801  9570 consensus_queue.cc:1048] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 13:04:43.151877  9424 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:43.232329  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:43.208889  9587 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:43.220999  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.222888  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:43.243068  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:43.244320  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:43.186403  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.230376  9616 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.246966  9549 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:43.230183  9400 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.247936  9616 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.248224  9606 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:43.248600  9615 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:43.249051  9615 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.249403  9190 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:43.249868  9190 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.250070  9563 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:43.250737  9563 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.251181  9563 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.249441  9615 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:43.224745  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46203.1us
I20260709 13:04:43.253736  9598 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:43.254454  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.255771  1705 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:43.256631  1705 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:43.227589  1739 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:43.211238  9097 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:43.212656  9597 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:43.143646  9593 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:43.240888  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 138182us
I20260709 13:04:43.210987  9609 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:43.240130  9583 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.250226  9190 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.193634  1634 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:43.285046  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 92147.6us
W20260709 13:04:43.285844  1634 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.241648  8791 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.229985  9626 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:43.216056  9501 raft_consensus.cc:740] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:43.216243  9586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.034s	sys 0.008s
I20260709 13:04:43.224433  9259 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.268738  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:43.299999  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.301894  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:43.270921  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 49981.8us
I20260709 13:04:43.303221  2280 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:43.304347  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.306116  9400 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:43.306826  9400 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:43.308266  9400 raft_consensus.cc:2804] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:43.308745  9400 raft_consensus.cc:697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:43.309851  9579 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:04:43.310297  9579 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.310647  9579 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:43.315286  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:43.199046  9596 consensus_queue.cc:1048] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.060s
W20260709 13:04:43.216729  1556 connection.cc:256] RPC call timeout handler was delayed by -2.1910352440000338s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 13:04:43.318568  1556 connection.cc:256] RPC call timeout handler was delayed by -1.9960772339999835s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20260709 13:04:43.318953  1556 connection.cc:256] RPC call timeout handler was delayed by -1.9836377999999968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
W20260709 13:04:43.319248  1556 connection.cc:256] RPC call timeout handler was delayed by -1.9825772340000185s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260709 13:04:43.319563  1556 connection.cc:256] RPC call timeout handler was delayed by -1.9721401550000337s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260709 13:04:43.145752  9599 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.273566  9444 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.261263  9619 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:43.282015  9583 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.267941  9590 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 13:04:43.321710  9590 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:43.322156  9590 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:43.323169  9590 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.325004  9590 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:43.325835  9599 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:43.329392  9583 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.334757  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.335476  1737 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:43.336839  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:43.337827  9583 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:43.338197  9583 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.338558  9583 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.281234  9615 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.261287  9097 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.268024  9549 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:43.341910  9549 raft_consensus.cc:697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:43.343168  9549 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.344139  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:43.344858  2313 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:43.143573  9573 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.347656  9573 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:43.218436  9305 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.350343  9305 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:43.286500  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:43.351151  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64689.1us
W20260709 13:04:43.352488  1634 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.353641  9097 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.218904  9453 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:43.353242  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:43.361466  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.257431  9624 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 23. Current term is 24.
W20260709 13:04:43.362496  1556 connection.cc:256] RPC call timeout handler was delayed by -1.6805042020000087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15700000000000003s for a response
I20260709 13:04:43.322728  9400 consensus_queue.cc:237] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.343191  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:43.362927  1556 connection.cc:256] RPC call timeout handler was delayed by -1.6243772340000078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
W20260709 13:04:43.367600  1556 connection.cc:256] RPC call timeout handler was delayed by -1.5581042020000382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1706s for a response
W20260709 13:04:43.367913  1556 connection.cc:256] RPC call timeout handler was delayed by -1.4680702040000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
W20260709 13:04:43.368182  1556 connection.cc:256] RPC call timeout handler was delayed by -1.3616850390000081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260709 13:04:43.368466  1556 connection.cc:256] RPC call timeout handler was delayed by -1.1799378000000047s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.242s for a response
W20260709 13:04:43.371433  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:43.360100  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 58245.8us
W20260709 13:04:43.362280  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:43.292936  9623 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:43.299190  9586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.034s	sys 0.008s
I20260709 13:04:43.290395  9424 raft_consensus.cc:697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:43.373397  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.291671  9274 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:04:43.294587  9560 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:43.408433  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:43.375643  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 189267us
I20260709 13:04:43.373191  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:43.297096  9591 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:43.430758  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57435.6us
I20260709 13:04:43.434139  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:43.432175  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 187899us
I20260709 13:04:43.437541  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:43.438995  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 182394us
W20260709 13:04:43.439179  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 66102.6us
W20260709 13:04:43.440987  1634 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.442337  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.378841  9305 raft_consensus.cc:2804] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:43.374987  9606 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:43.446683  1634 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:43.450889  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.454080  1632 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.380784  9625 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.299302  9563 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.287866  9601 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.459410  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.463320  9613 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:43.361148  9622 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:43.439190  9624 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.438726  9626 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:43.456468  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.445130  9610 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:43.520749  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64361.4us
I20260709 13:04:43.450457  9305 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.444216  9274 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 13:04:43.505967  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63025.7us
I20260709 13:04:43.529811  1667 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 13:04:43.529183  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.535570  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.358062  9617 raft_consensus.cc:2804] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:43.448777  9616 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election won for term 22
W20260709 13:04:43.538530  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 79200.3us
W20260709 13:04:43.543720  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 171790us
W20260709 13:04:43.545200  1556 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.570s (SENT)
I20260709 13:04:43.547055  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.432547  9623 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.553162  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.520107  9622 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:43.527341  9305 raft_consensus.cc:3060] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:43.501538  9621 consensus_queue.cc:1059] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 13:04:43.449054  9501 consensus_queue.cc:260] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.545176  9625 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:43.458946  9606 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:43.576341  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54877.8us
I20260709 13:04:43.553290  9274 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:43.525576  9618 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:43.527144  9626 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:43.602545  9629 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 13:04:43.603605  9626 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.604195  9626 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:43.604976  9623 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:43.605887  9646 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.606424  9646 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.589699  9643 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.609560  9643 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:43.564085  9637 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:43.559645  9622 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:43.581431  9625 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:43.610875  9630 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 13:04:43.598157  9613 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:43.611672  9501 raft_consensus.cc:3060] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:43.589426  9617 raft_consensus.cc:697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:43.558725  9631 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.090s
W20260709 13:04:43.650296  9633 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 13:04:43.586890  9610 consensus_queue.cc:1048] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 13:04:43.556141  9424 consensus_queue.cc:237] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.648313  9625 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:43.710693  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:43.626394  9616 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:43.622710  9619 consensus_queue.cc:1059] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: INVALID_TERM, Last received: 20.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.090s
I20260709 13:04:43.701004  9618 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.623344  9641 consensus_queue.cc:1048] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.124s
W20260709 13:04:43.692603  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 147413us
W20260709 13:04:43.734781  9375 consensus_meta.cc:287] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.655s	user 0.010s	sys 0.003s
I20260709 13:04:43.735713  2311 raft_consensus.cc:2468] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:43.737872  9563 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:43.738819  9323 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.003s
I20260709 13:04:43.739282  9323 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.741001  9647 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.116s
I20260709 13:04:43.743180  9652 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:43.743758  9652 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:43.744644  9652 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.743220  9619 raft_consensus.cc:3055] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:43.745744  9619 raft_consensus.cc:740] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:43.746698  9619 consensus_queue.cc:260] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.9, Last appended by leader: 9, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.748373  9619 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:43.750015  9616 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.643939  9650 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.084s
W20260709 13:04:43.759083  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.709033  9615 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.778640  9259 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.789727  9323 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:43.763078  9618 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:43.791596  9635 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
W20260709 13:04:43.796519  9608 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.581s	user 0.005s	sys 0.003s
W20260709 13:04:43.802244  9190 consensus_meta.cc:287] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.000s
I20260709 13:04:43.802764  9190 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.805161  9190 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:43.722270  9457 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:43.807092  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 264206us
W20260709 13:04:43.772603  9632 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 13:04:43.809392  1659 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:43.815570  1632 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:43.819478  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 91131.5us
I20260709 13:04:43.833434  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:43.810513  9583 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.835237  1632 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:43.793385  9618 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:43.806660  9608 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:43.839370  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:43.841568  1556 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.706s (SENT)
I20260709 13:04:43.840540  9097 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:43.851897  9097 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:43.853361  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:43.853952  9097 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:43.854429  9097 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:43.854840  9097 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:43.857333  9618 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:43.858666  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:43.862780  1556 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 1.750s (SENT)
W20260709 13:04:43.864192  1556 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Timed out: RequestConsensusVote RPC to 127.0.107.194:37489 timed out after 2.069s (SENT)
I20260709 13:04:43.874415  2313 raft_consensus.cc:1275] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 13:04:43.836009  9190 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.879607  1740 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27. Ops: []
I20260709 13:04:43.872865  9079 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:43.881155  2310 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:43.881412  1739 raft_consensus.cc:1240] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:43.883244  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:43.884048  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:43.885284  2309 raft_consensus.cc:1240] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 24. Current term is 25. Ops: []
I20260709 13:04:43.789829  9617 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:43.936120  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68997.1us
I20260709 13:04:43.937103  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:43.931851  2315 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:43.904330  9646 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:43.910827  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:43.947774  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 680682us
W20260709 13:04:43.916345  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 333895us
I20260709 13:04:43.840148  9615 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:43.842586  9563 raft_consensus.cc:2804] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:43.809690  9579 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.949971  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:43.956092  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:43.836620  9457 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:43.958022  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:43.962750  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:43.965036  1740 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:43.930138  9612 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.791s	user 0.007s	sys 0.007s
I20260709 13:04:43.966527  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:43.964011  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:43.970386  1739 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:43.982383  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.003459  9656 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.017s
W20260709 13:04:44.004379  9655 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 13:04:44.004714  9616 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:43.987105  9631 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:44.006903  9654 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:43.930733  9079 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.005333  9323 consensus_queue.cc:1048] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:44.039831  9626 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:44.054176  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.054792  9259 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.056737  9652 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:44.058039  1705 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:44.059226  9639 consensus_queue.cc:1048] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:44.061401  9606 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.063751  9591 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:44.064206  9591 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.064569  9591 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:44.066610  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.067284  1735 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:43.929370  9453 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.071640  9453 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:44.033818  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67365.1us
W20260709 13:04:44.037021  8791 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.741s	user 0.011s	sys 0.001s
I20260709 13:04:43.939649  9559 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.103s
W20260709 13:04:44.038232  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47586.6us
I20260709 13:04:44.078022  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:44.078022  9559 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:43.959293  9621 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
W20260709 13:04:44.032552  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58058.2us
I20260709 13:04:44.080386  2316 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 27. Current term is 28. Ops: []
I20260709 13:04:44.080649  1733 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:44.082682  1733 raft_consensus.cc:1240] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 13. Current term is 14. Ops: []
I20260709 13:04:44.086112  1733 raft_consensus.cc:1217] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Deduplicated request from leader. Original: 13.8->[15.9-21.11]   Dedup: 15.9->[18.10-21.11]
I20260709 13:04:44.086673  1733 pending_rounds.cc:85] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 9
I20260709 13:04:44.086958  1733 pending_rounds.cc:107] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 13:04:44.087255  1733 raft_consensus.cc:2889] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:44.090170  1659 raft_consensus.cc:1275] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 13:04:44.094301  9400 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.094841  9400 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.049032  9654 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.049943  9079 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:44.033567  9563 raft_consensus.cc:697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:44.103572  1739 raft_consensus.cc:2468] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:44.095781  9622 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:43.965046  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:44.146494  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.147377  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:44.149170  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:44.156450  1634 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.157207  1634 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.158267  1634 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.096980  9646 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:44.160346  9646 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.163333  9646 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:44.145720  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 197068us
I20260709 13:04:44.150880  1740 raft_consensus.cc:2468] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:44.175191  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.176249  1740 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:44.177482  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.147396  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:44.179047  9601 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:44.159660  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.183925  1742 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 27: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:44.166781  1634 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.145669  1735 raft_consensus.cc:2468] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:44.134847  9660 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.185245  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.192983  1742 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:44.032964  9612 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.188899  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.194442  1735 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:44.195588  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.198604  9453 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:44.152940  9259 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:44.201573  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.201654  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.207484  1739 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:44.192498  1740 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:44.150880  9621 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.150166  9283 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.126705  9608 consensus_queue.cc:1059] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.064s
I20260709 13:04:44.192478  9637 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.203004  9618 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:44.222280  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:44.217952  9501 consensus_meta.cc:287] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.000s
I20260709 13:04:44.160372  9591 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.234890  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.235554  1740 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:44.205776  1735 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
W20260709 13:04:44.190680  9645 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.442s	user 0.000s	sys 0.004s
I20260709 13:04:44.209425  1739 raft_consensus.cc:1240] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [19.11-21.12]
W20260709 13:04:44.203864  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 231647us
I20260709 13:04:44.240234  1735 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:44.214435  9583 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:44.245329  1557 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Timed out: RequestConsensusVote RPC to 127.0.107.195:33737 timed out after 1.706s (SENT)
I20260709 13:04:44.249225  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:44.209285  9259 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.259382  1739 raft_consensus.cc:1275] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:04:44.228690  9618 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:44.218036  9283 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.197731  9660 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.266064  1737 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 13:04:44.218869  9637 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.212030  9621 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.261188  9657 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:44.277643  9305 consensus_meta.cc:287] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.707s	user 0.014s	sys 0.000s
I20260709 13:04:44.278149  9305 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.281692  9646 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.282258  9646 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.283229  9643 raft_consensus.cc:3055] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
I20260709 13:04:44.283686  9643 raft_consensus.cc:740] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:44.284564  9643 consensus_queue.cc:260] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.286188  9274 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.272014  9579 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 28 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.219055  9625 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.286693  9274 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.290307  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3587) took 1561 ms. Trace:
I20260709 13:04:44.291651  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.215502  9563 consensus_queue.cc:237] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.277397  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.293615  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.290875  1736 rpcz_store.cc:276] 0709 13:04:42.728830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.728991 (+   161us) service_pool.cc:224] Handling call
0709 13:04:42.729317 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:43.056155 (+326838us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.289893 (+1233738us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540039e9d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:44.290204 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616971136}
W20260709 13:04:44.293682  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 146433us
I20260709 13:04:44.269335  1740 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:44.293857  1666 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:04:44.298334  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.300483  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.302338  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.303372  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:44.304996  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.324051  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.325941  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.305799  1659 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:04:44.327579  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.329210  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.330672  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.270102  9190 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:44.329957  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:44.334604  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.335690  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.340337  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.341753  1659 raft_consensus.cc:2393] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 25: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:44.306602  1661 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:44.275739  9457 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.275312  9590 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:44.339185  1668 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:44.344467  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.349134  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.350733  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.291494  9400 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:44.353015  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.353962  1665 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:44.355103  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.355768  1665 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:44.356403  1665 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:44.358621  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.362170  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:44.363000  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.363015  1659 raft_consensus.cc:2376] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20. Re-sending same reply.
I20260709 13:04:44.344697  9190 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.374708  9259 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.374821  9274 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.375016  1736 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:44.359790  9590 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.384408  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:44.343716  9663 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
I20260709 13:04:44.289633  9079 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.391188  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.337767  9624 raft_consensus.cc:3055] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 LEADER]: Stepping down as leader of term 27
W20260709 13:04:44.398942  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69718.3us
I20260709 13:04:44.401547  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.402187  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:44.397648  1666 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:44.402364  1737 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:44.305133  9579 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 13:04:44.343753  9559 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:44.406189  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.400691  1557 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:44.407130  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.408190  1735 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:44.409982  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:44.410594  9079 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.398568  9624 raft_consensus.cc:740] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:44.412060  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.412745  1664 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:44.412268  9624 consensus_queue.cc:260] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.420778  9624 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:44.421882  9579 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:44.422287  9579 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:44.332231  9615 consensus_queue.cc:1048] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.066s
I20260709 13:04:44.414664  1663 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 13:04:44.332104  9676 consensus_queue.cc:1048] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.113s
I20260709 13:04:44.425454  9612 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.427145  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:44.428128  9654 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.414258  9501 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:44.429028  9501 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.429401  9501 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:44.361644  9671 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.086s
I20260709 13:04:44.434893  9668 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:44.435099  1660 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.436388  1660 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:44.405380  9618 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:44.438485  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.434118  1486 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 20 to 21, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:44.436326  2311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 750) took 2073 ms (client timeout 1757 ms). Trace:
I20260709 13:04:44.443893  2316 raft_consensus.cc:1240] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 27. Current term is 28. Ops: [27.18-27.18]
W20260709 13:04:44.423102  9597 consensus_meta.cc:287] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.137s	user 0.010s	sys 0.003s
I20260709 13:04:44.414024  1664 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:44.408226  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:44.456743  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48606.6us
I20260709 13:04:44.458308  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:44.450026  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:44.442435  2311 rpcz_store.cc:269] 0709 13:04:42.362605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:42.363121 (+   516us) service_pool.cc:224] Handling call
0709 13:04:44.436133 (+2073012us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b54006004d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:44.436249 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3153793408}
I20260709 13:04:44.462572  1660 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3458) took 1339 ms. Trace:
I20260709 13:04:44.454010  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:44.463165  9657 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.465543  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:44.462524  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:44.485085  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.467068  1664 raft_consensus.cc:1275] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 13:04:44.486090  2311 raft_consensus.cc:1240] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:44.486085  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:44.523797  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61382.2us
I20260709 13:04:44.463657  1660 rpcz_store.cc:276] 0709 13:04:43.123322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:43.128041 (+  4719us) service_pool.cc:224] Handling call
0709 13:04:43.140009 (+ 11968us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:44.434687 (+1294678us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b4c00164c80. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec2a9a20 00007fcdec2baa29 00007fcdec280644 00007fcdec2a7f2d 00007fcdec2a7b93 00007fcdedfd818b 00007fcdedfd8cb5 00007fcdedfd3f30 00007fcdedfd57de 00007fcdedfdfb49 00007fcdedfe0416
0709 13:04:44.462157 (+ 27470us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:44.462208 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:44.462257 (+    49us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:44.462292 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.462363 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:44.462509 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2983,"spinlock_wait_cycles":35097856}
I20260709 13:04:44.527724  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:44.521847  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.545485  1667 raft_consensus.cc:2393] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:44.588438  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 103385us
I20260709 13:04:44.415831  9660 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.527335  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:44.588840  1664 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
W20260709 13:04:44.589552  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44331us
I20260709 13:04:44.464461  9612 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:44.451524  9622 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.588698  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:44.423226  9559 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.417157  9637 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.388829  9457 consensus_queue.cc:1059] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.095s
I20260709 13:04:44.473312  9606 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:44.544185  9259 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:44.601824  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:44.606899  2309 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 19: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
W20260709 13:04:44.587872  9665 log.cc:927] Time spent T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
I20260709 13:04:44.587487  9656 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:44.587746  9608 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:44.472577  9079 consensus_queue.cc:1059] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.139s
I20260709 13:04:44.666347  9283 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.614564  9559 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:44.472748  9597 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:44.620682   431 thread.cc:527] Waited for 1000ms trying to join with rpc worker (tid 1594)
I20260709 13:04:44.473694  9583 consensus_queue.cc:1059] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.152s
I20260709 13:04:44.483734  9615 consensus_queue.cc:1059] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.159s
I20260709 13:04:44.423130  9579 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.670336  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1360) took 1348 ms. Trace:
I20260709 13:04:44.474931  9654 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:44.604696  9676 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.491580  9657 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.595835  9609 consensus_queue.cc:1048] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.069s
I20260709 13:04:44.643590  9606 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:44.674206  1738 rpcz_store.cc:276] 0709 13:04:43.321903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:43.322046 (+   143us) service_pool.cc:224] Handling call
0709 13:04:43.322448 (+   402us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:43.432600 (+110152us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:04:43.543238 (+110638us) log.cc:844] Serialized 14 byte log entry
0709 13:04:43.689899 (+146661us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:43.690612 (+   713us) log.cc:844] Serialized 52 byte log entry
0709 13:04:43.690787 (+   175us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:43.690846 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.663860 (+973014us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400202bd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:44.663917 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:44.663968 (+    51us) raft_consensus.cc:1685] finished
0709 13:04:44.670071 (+  6103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:44.670267 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7501,"prepare.queue_time_us":129289,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":45667,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":17755,"spinlock_wait_cycles":236110080,"thread_start_us":175150,"threads_started":2,"wal-append.queue_time_us":232214}
I20260709 13:04:44.668668  9608 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.703862  1665 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 13:04:44.648386  9622 consensus_queue.cc:1059] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 13:04:44.678628  9654 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:44.664985  9656 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.618750  9682 consensus_queue.cc:1059] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.090s
I20260709 13:04:44.682024  9583 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:44.703786  9606 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:44.714854  9608 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.783064  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:44.763702  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:44.720324  9305 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.702865  9676 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.651645  9637 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:44.721422  9652 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.664s	user 0.003s	sys 0.010s
I20260709 13:04:44.769753  9572 raft_consensus.cc:3055] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:44.820554  9190 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.819013  9597 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.750063  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1359) took 1473 ms. Trace:
I20260709 13:04:44.765261  2313 raft_consensus.cc:1275] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 13:04:44.763082  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:44.711869  9689 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:44.848788  1661 raft_consensus.cc:1275] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:44.843359  1665 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
W20260709 13:04:44.832288  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.754868  1664 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
W20260709 13:04:44.850368  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 232611us
I20260709 13:04:44.846303  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:44.858136  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:44.858273  2316 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:44.856178  1734 rpcz_store.cc:276] 0709 13:04:43.276837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:43.277043 (+   206us) service_pool.cc:224] Handling call
0709 13:04:43.277485 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:43.296431 (+ 18946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:43.296464 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:43.296511 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:43.296557 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.749806 (+1453249us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:44.749862 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:44.749995 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":42970368}
W20260709 13:04:44.776386  9097 consensus_meta.cc:287] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.921s	user 0.008s	sys 0.004s
W20260709 13:04:44.859442  2283 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.866221  2283 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:44.870432  2283 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.820652  9631 consensus_meta.cc:287] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.775s	user 0.012s	sys 0.000s
W20260709 13:04:44.827654  9684 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.002s	sys 0.002s
W20260709 13:04:44.827867  9670 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.002s	sys 0.002s
W20260709 13:04:44.827714  9687 log.cc:927] Time spent T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.116s	user 0.004s	sys 0.000s
I20260709 13:04:44.750918  9657 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.762286  9621 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.724967  9660 raft_consensus.cc:3055] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:44.854039  9656 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:44.822585  9591 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:44.907488  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:44.912717  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.913578  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:44.915228  2283 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:44.876117  9623 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:04:44.827637  9572 raft_consensus.cc:740] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:44.918321  1664 pending_rounds.cc:85] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 14
I20260709 13:04:44.918716  1664 pending_rounds.cc:107] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 13:04:44.919082  1664 raft_consensus.cc:2889] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:44.738220  9654 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:44.742463  9625 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:44.916195  2283 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:44.908116  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 351191us
I20260709 13:04:44.900749  9631 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:44.850339  9637 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.772213  9501 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.916100  9657 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:44.915560  9621 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:44.911626  9660 raft_consensus.cc:740] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:44.927229  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 336997us
I20260709 13:04:44.918239  9583 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:44.915094  9593 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:44.860543  9689 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.931279  1632 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:44.957031  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:44.958397  1632 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:44.959826  1632 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:44.965612  9572 consensus_queue.cc:260] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.966132  9660 consensus_queue.cc:260] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.966920  9501 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.967777  9660 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:44.968348  9559 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.969699  9652 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:44.970148  9652 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:44.970636  9652 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:44.971163  9274 raft_consensus.cc:3055] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:44.971603  9274 raft_consensus.cc:740] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:44.972522  9274 consensus_queue.cc:260] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.957837  9631 consensus_queue.cc:1059] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 13:04:44.905738  9097 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:44.976073  9097 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.977185  2310 raft_consensus.cc:2368] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
I20260709 13:04:44.977888  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3609) took 1097 ms. Trace:
I20260709 13:04:44.978287  1733 rpcz_store.cc:276] 0709 13:04:43.880432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.085562 (+205130us) service_pool.cc:224] Handling call
0709 13:04:44.085894 (+   332us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 13:04:44.088045 (+  2151us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 13:04:44.088356 (+   311us) log.cc:844] Serialized 14 byte log entry
0709 13:04:44.223806 (+135450us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:44.228976 (+  5170us) log.cc:844] Serialized 52 byte log entry
0709 13:04:44.229163 (+   187us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:44.230688 (+  1525us) log.cc:844] Serialized 14 byte log entry
0709 13:04:44.236767 (+  6079us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.236858 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:44.285352 (+ 48494us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b1800482260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:44.759630 (+474278us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b1800482260. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:44.977603 (+217973us) raft_consensus.cc:1685] finished
0709 13:04:44.977630 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:44.977817 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64627,"prepare.queue_time_us":8618,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":97,"raft.queue_time_us":24164,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":39364,"spinlock_wait_cycles":10287104,"thread_start_us":134773,"threads_started":1,"wal-append.queue_time_us":162465}
I20260709 13:04:44.981988  1659 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 13:04:44.985033  9646 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.988350  9689 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.966861  9676 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:44.993870  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.999603  9563 raft_consensus.cc:3055] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 13:04:45.001587  9457 raft_consensus.cc:2804] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:45.006268  9652 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:44.955088  9621 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.012284  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:45.013211  9666 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:44.950229  9657 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.015954  9079 raft_consensus.cc:2804] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:45.018100  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:45.023874  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:45.039181  1736 raft_consensus.cc:2410] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 28 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 13:04:45.039446  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:45.024832  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:45.028221  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 64979.5us
I20260709 13:04:45.026665  9618 raft_consensus.cc:2804] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 13:04:45.033185  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:44.999475  9637 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:44.994089  9274 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:45.028795  9591 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:45.052924  2309 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:45.054908  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.055959  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:45.033696  9631 raft_consensus.cc:3055] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:45.032575  9563 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.061946  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:45.070060  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.067950  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:45.013402  9608 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.044589  9457 raft_consensus.cc:697] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.080106  1660 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:45.043979  9097 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:45.047204  9593 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:45.071771  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:45.085074  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.085729  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.089672  1736 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:45.036744  9559 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.090812  9650 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.091359  9650 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.084514  9079 raft_consensus.cc:697] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.094466  9593 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.094961  9593 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:45.094652  9079 consensus_queue.cc:237] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.097921  1734 raft_consensus.cc:2410] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:45.098012  1739 raft_consensus.cc:1240] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27. Ops: []
I20260709 13:04:45.073777  9614 consensus_queue.cc:1048] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.027s
I20260709 13:04:45.060240  9610 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:45.063259  9631 raft_consensus.cc:740] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:45.101783  9610 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:45.110484  9618 raft_consensus.cc:697] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.113793  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.117669  9652 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.118060  9696 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.118462  9598 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 13:04:45.065636  9609 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 13:04:45.126012  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.086544  9612 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:45.126482  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:45.143563  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:45.143842  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:45.120869  1734 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:45.088230  9097 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:45.071987  9640 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 13:04:45.072790  9676 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 13:04:45.065902  9617 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 13:04:45.066072  9591 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:45.151568  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 227932us
I20260709 13:04:45.120528  9610 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.146085  9692 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.111629  9650 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.184459  9608 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.185746  9622 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.186352  9622 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.187614  9590 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.137971  9598 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.188472  9622 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.146445  9097 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.200026  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:45.200927  2309 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:45.238551  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:45.273644  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.311053  2309 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:45.142226  9696 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.287485  9615 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.326658  9606 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.189304  9598 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.190896  9631 consensus_queue.cc:260] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.154219  9563 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:45.346194  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 282135us
I20260709 13:04:45.349037  2281 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:45.212911  9692 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.219190  9457 consensus_queue.cc:237] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.352681  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:45.334187  9704 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.356909  9704 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.357650  9657 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.357867  9596 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.358460  9596 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.359030  9657 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:45.359444  9704 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.362946  9696 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.324684  9618 consensus_queue.cc:237] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.363520  9692 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.345578  9615 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.364405  9596 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.365566  9596 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:45.366158  9596 raft_consensus.cc:697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:45.368938  9700 log.cc:927] Time spent T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:04:45.368757  9596 consensus_queue.cc:237] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.374359  9563 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:45.380716  9587 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.381453  9587 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.382438  9606 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.353538  9631 raft_consensus.cc:3060] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:45.384146  9587 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.384807  9621 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.387643  9579 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:45.391078  9305 raft_consensus.cc:3055] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:45.391507  9305 raft_consensus.cc:740] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:45.392385  9305 consensus_queue.cc:260] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.393205  9559 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:45.393726  9559 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:45.394064  9593 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.394601  9559 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.396003  1740 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 24: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:45.352783  9591 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.389873  9598 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:45.400538  9593 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.402236  9654 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.404316  9654 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.407122  9305 raft_consensus.cc:3060] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:45.408367  9660 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:45.408895  9660 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:45.399076  9559 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:45.442750  9656 consensus_meta.cc:287] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.001s
I20260709 13:04:45.442667  9660 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.467770  9660 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:45.514739  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 361556us
W20260709 13:04:45.516264  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:45.531316  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.546550  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1358) took 2289 ms. Trace:
W20260709 13:04:45.547083  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.542387  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56492 (request call id 1322) took 1095 ms. Trace:
I20260709 13:04:45.547312  1741 rpcz_store.cc:276] 0709 13:04:43.255841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:43.256040 (+   199us) service_pool.cc:224] Handling call
0709 13:04:43.256461 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:43.444944 (+188483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:43.444978 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:45.099806 (+1654828us) spinlock_profiling.cc:243] Waited 5.14 ms on lock 0x7b2c00077ef8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:45.100195 (+   389us) log.cc:844] Serialized 26 byte log entry
0709 13:04:45.344589 (+244394us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:45.344686 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.344814 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:45.544143 (+199329us) raft_consensus.cc:1685] finished
0709 13:04:45.544199 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:45.545527 (+  1328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186,"raft.queue_time_us":11857,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":15481,"spinlock_wait_cycles":11828608,"thread_start_us":431963,"threads_started":2,"wal-append.queue_time_us":340021}
I20260709 13:04:45.533492  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:45.341482  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:45.600660  9714 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.603343  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3472) took 1133 ms. Trace:
I20260709 13:04:45.603899  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:45.604758  1668 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:45.605970  9713 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.607082  9713 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.605347  9714 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.610492  9723 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.611718  9723 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.366523  9097 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.552094  1663 rpcz_store.cc:276] 0709 13:04:44.446204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.446381 (+   177us) service_pool.cc:224] Handling call
0709 13:04:44.483316 (+ 36935us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:44.543695 (+ 60379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:44.543738 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:44.544413 (+   675us) log.cc:844] Serialized 52 byte log entry
0709 13:04:44.713948 (+169535us) raft_consensus.cc:1645] Marking committed up to 17
0709 13:04:44.715170 (+  1222us) log.cc:844] Serialized 14 byte log entry
0709 13:04:44.715393 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:44.715474 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:44.990939 (+275465us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b180041d6a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:45.098441 (+107502us) spinlock_profiling.cc:243] Waited 39.1 ms on lock 0x7b180041d6a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:45.370107 (+271666us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x7b180041d6a0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:45.537801 (+167694us) raft_consensus.cc:1685] finished
0709 13:04:45.537948 (+   147us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:45.541892 (+  3944us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72678,"prepare.queue_time_us":1488,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":1536,"raft.run_cpu_time_us":498,"raft.run_wall_time_us":114140,"spinlock_wait_cycles":183921152,"thread_start_us":53008,"threads_started":1,"wal-append.queue_time_us":108722}
I20260709 13:04:45.616353  9097 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.618093  9591 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:45.619275  9591 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.622030  9613 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:45.617229  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:45.623521  9593 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:45.598073  1666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.009s	sys 0.018s
I20260709 13:04:45.656412  1663 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:45.659446  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 420814us
I20260709 13:04:45.660926  9424 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.662057  9424 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.670037  9715 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.670943  9714 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.671221  9715 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.604018  1662 rpcz_store.cc:276] 0709 13:04:44.469780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.648247 (+178467us) service_pool.cc:224] Handling call
0709 13:04:44.648620 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:44.649000 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:44.649034 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:44.660989 (+ 11955us) log.cc:844] Serialized 26 byte log entry
0709 13:04:44.763629 (+102640us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b180044e300. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:45.052565 (+288936us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:45.052618 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.052811 (+   193us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:45.590269 (+537458us) raft_consensus.cc:1685] finished
0709 13:04:45.592177 (+  1908us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:45.603099 (+ 10922us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12357,"raft.queue_time_us":3190,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":2256,"spinlock_wait_cycles":9558272,"thread_start_us":288721,"threads_started":1,"wal-append.queue_time_us":479968}
W20260709 13:04:45.677853  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.677233  9625 consensus_queue.cc:1059] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 13:04:45.678593  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.679008  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.679641  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:45.679296  1486 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 21, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:45.680570  1664 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:45.680876  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:45.686928  1666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.009s	sys 0.019s
I20260709 13:04:45.660969  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:45.708166  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 175088us
I20260709 13:04:45.633375  9593 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.720029  9713 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:45.720974  9259 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.000s	sys 0.015s
W20260709 13:04:45.721251  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.703996  9621 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.717806  1662 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:45.716176  9625 consensus_queue.cc:1243] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 13:04:45.639042  9613 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.740813  1659 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2785) took 1305 ms. Trace:
I20260709 13:04:45.755064  9613 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.756914  9619 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:45.757597  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.758049  9619 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.758736  9617 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.759194  9424 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.759478  9617 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.760742  9619 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.761008  9591 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:45.762261  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.761917  9621 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.741474  1659 rpcz_store.cc:276] 0709 13:04:44.403216 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.403415 (+   199us) service_pool.cc:224] Handling call
0709 13:04:44.403901 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:44.982519 (+578618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.677157 (+694638us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b54003c9e54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:45.708142 (+ 30985us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328553344}
I20260709 13:04:45.769853  9720 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:45.770960  9720 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:45.725835  9097 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.774012  9097 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.778281  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:45.778537  9719 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 13:04:45.768905  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:45.778352  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:45.769045  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.794209  1659 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:45.795696  9621 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:45.796031  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 135350us
W20260709 13:04:45.800559  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 59220.9us
I20260709 13:04:45.451506  9656 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.801920  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:45.805872  9723 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.729944  9715 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.797187  1556 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:45.795509  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:45.826967  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 104359us
I20260709 13:04:45.806447  9305 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:45.827279  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.795647  9713 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:45.838711  1663 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:04:45.845631  9612 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.846010  9617 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.846662  9612 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.849802  9720 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.849988  9593 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:45.845680  9656 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.841176  9715 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:45.879663  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:45.900089  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93793.3us
W20260709 13:04:45.898957  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 58562.1us
W20260709 13:04:45.907719  9598 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.007s
I20260709 13:04:45.918609  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.918988  1741 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:45.919713  1661 raft_consensus.cc:2468] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:45.921007  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.922667  9694 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:45.923470  9694 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.925446  9593 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:45.926397  9593 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:45.929009  9593 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:45.896112  9668 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:04:45.930955  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.932480  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3471) took 1463 ms. Trace:
I20260709 13:04:45.932902  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:45.930846  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.935374  1668 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
W20260709 13:04:45.945964  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.935286  1661 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 14.
I20260709 13:04:45.948438  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:45.955479  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:45.937045  9097 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:45.959079  9643 raft_consensus.cc:3060] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:45.960345  2310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 754) took 1433 ms. Trace:
I20260709 13:04:45.960925  2310 rpcz_store.cc:276] 0709 13:04:44.526884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.527047 (+   163us) service_pool.cc:224] Handling call
0709 13:04:45.959622 (+1432575us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54002e8454. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:45.960055 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":873661696}
W20260709 13:04:45.943473  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 106101us
I20260709 13:04:45.982932  1664 raft_consensus.cc:2468] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
W20260709 13:04:45.984748  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 216110us
I20260709 13:04:45.985958  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:45.987063  9097 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:45.988188  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:45.989172  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 192210us
I20260709 13:04:45.990239  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:45.993863  9730 log.cc:927] Time spent T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:04:45.988284  9097 consensus_queue.cc:1048] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260709 13:04:45.995357  9305 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:45.996408  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:45.945364  9612 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:45.998708  9579 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.610s	user 0.014s	sys 0.007s
W20260709 13:04:46.000029  9559 consensus_meta.cc:287] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.590s	user 0.013s	sys 0.004s
I20260709 13:04:46.001042  9694 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.004611  2315 raft_consensus.cc:2410] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 13:04:45.932986  1667 rpcz_store.cc:276] 0709 13:04:44.469406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.643222 (+173816us) service_pool.cc:224] Handling call
0709 13:04:44.666083 (+ 22861us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:44.954570 (+288487us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:45.342868 (+388298us) log.cc:844] Serialized 14 byte log entry
0709 13:04:45.656853 (+313985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:45.656978 (+   125us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:45.657178 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.931926 (+274748us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b54003c8a54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:45.931983 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:45.932241 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":144187,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":6972,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":326,"spinlock_wait_cycles":662571776,"thread_start_us":667217,"threads_started":2}
W20260709 13:04:46.014510  9631 consensus_meta.cc:287] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.631s	user 0.010s	sys 0.012s
W20260709 13:04:46.015368  9725 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.555s	user 0.005s	sys 0.000s
I20260709 13:04:46.015249  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:45.988608  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:46.016184  9563 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.638s	user 0.012s	sys 0.008s
W20260709 13:04:46.036136  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 197531us
I20260709 13:04:46.037038  1662 raft_consensus.cc:2468] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 26.
W20260709 13:04:46.038017  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 49620.7us
I20260709 13:04:46.041217  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:46.046171  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 66822.2us
I20260709 13:04:46.046020  1741 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:46.049728  1668 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:46.037869  9612 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.059247  1740 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23. Ops: []
W20260709 13:04:46.061291  9610 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.816s	user 0.014s	sys 0.015s
I20260709 13:04:46.062039  9610 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.063304  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.064071  1740 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:46.064651  9610 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:46.053994  9694 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.075502  9694 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.076025  9694 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.055351  9305 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:46.078264  9305 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:46.078852  9305 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
W20260709 13:04:46.080595  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65614.2us
I20260709 13:04:46.081835  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.082728  1742 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:46.083760  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:46.085566  1634 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.086328  1634 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:46.087965  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.080791  1734 raft_consensus.cc:1275] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 13:04:46.089190  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.089859  9689 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.090058  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.089727  9598 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 13:04:46.092168  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:46.093092  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45690.7us
W20260709 13:04:46.097856  1706 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.096920  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:46.104347  1634 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.105074  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.106048  1737 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
W20260709 13:04:46.106614  1634 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.094898  9612 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.111140  9714 raft_consensus.cc:3055] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:46.111632  9714 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:46.113958  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.114821  1736 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:46.116052  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.116706  1736 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:46.117012  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.117738  1739 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:46.112967  9714 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.120016  1741 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:46.123211  9728 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 13:04:46.131284  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.132395  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:46.133122  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.134155  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.136134  1741 raft_consensus.cc:2393] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:46.120450  9612 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:46.139655  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.140205  9689 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.141237  2316 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:04:46.142388  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.146724  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.148566  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.150317  1741 raft_consensus.cc:2393] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 18: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:46.159466  9598 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:46.160436  9598 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.163408  9598 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:46.180352  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.120641  9631 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:46.181023  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.186627  2310 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:46.189459  1741 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 23
I20260709 13:04:46.189309  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.195439  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.199211  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.189874  9714 raft_consensus.cc:3060] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:46.202636  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.203318  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.203931  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.204643  1737 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:46.203332  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.217386  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.201156  9715 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:46.218317  2313 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:46.221997  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.219161  1738 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:46.224872  1740 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:46.290474  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:46.291666  9719 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:46.290652  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62487.8us
I20260709 13:04:46.290714  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.291829  9563 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:46.291038  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.295236  1734 raft_consensus.cc:2410] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
W20260709 13:04:46.295945  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81519.9us
I20260709 13:04:46.305433  1737 raft_consensus.cc:2393] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:46.305846  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 118088us
I20260709 13:04:46.306869  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.308413  1737 raft_consensus.cc:1240] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 13:04:46.309731  1659 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:46.309942  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:46.309768  9259 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.311651  1737 raft_consensus.cc:1240] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: []
I20260709 13:04:46.318612  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.313486  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.319808  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.320441  1737 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:46.320255  9305 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.321090  1737 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 13:04:46.318092  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:46.325419  1741 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:46.329442  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.329732  1486 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:46.332333  1486 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 19, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:46.335961  1486 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 11 to 17, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:46.343370  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.344381  1668 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 24: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:46.346732  1557 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:46.347124  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:46.348145  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:46.305454  9719 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.349231  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.349574  9651 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.349777  9643 consensus_queue.cc:1059] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:46.350250  9651 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.350785  9643 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader election won for term 18
I20260709 13:04:46.350495  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.351383  9643 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:46.351481  9719 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:46.351675  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 228866us
I20260709 13:04:46.351999  1659 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 27.
I20260709 13:04:46.352295  9719 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:46.313175  9563 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.354153  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.352756  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.296916  9689 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:46.368721  9689 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
W20260709 13:04:46.354385  1634 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.369392  9689 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:46.369829  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.370630  9689 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:46.371232  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47425.3us
I20260709 13:04:46.375635  2316 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
W20260709 13:04:46.305434  9660 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.837s	user 0.016s	sys 0.003s
I20260709 13:04:46.351399  9666 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.380930  2310 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:46.349291  9688 consensus_queue.cc:1059] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 13:04:46.386885  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.365564  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.365922  9563 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.367794  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:46.355469  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.377244  1735 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:46.378087  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55503.8us
I20260709 13:04:46.339174  1486 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 16, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:46.352730  9643 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 17.16, Last appended by leader: 16, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:46.382035  2282 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.391485  1736 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:46.407368  1738 raft_consensus.cc:2393] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 24: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:46.404511  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:46.353317  9719 raft_consensus.cc:3055] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:46.410494  1742 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:46.393046  9598 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:46.425937  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.417258  1668 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:46.406862  9612 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.406580  9610 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:46.415071  9719 raft_consensus.cc:740] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:46.441674  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 73964.7us
I20260709 13:04:46.443030  1739 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:46.453547  9593 consensus_queue.cc:1048] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:46.428249  9660 raft_consensus.cc:2764] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 13:04:46.473766  1486 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 20 to 25, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 25 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:46.457975  9656 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:46.506927  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:46.514437  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 68420.5us
W20260709 13:04:46.515862  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55911.6us
I20260709 13:04:46.506840  1735 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:46.528945  9305 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:46.533108  1664 raft_consensus.cc:1240] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:46.533280  9719 consensus_queue.cc:260] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.535226  9719 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:46.545540  2316 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:46.548080  9610 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.556973  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:46.522795  9656 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:46.575659  1632 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.584671  9660 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:46.586453  1737 raft_consensus.cc:1240] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:46.590481  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 145236us
I20260709 13:04:46.563504  9579 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.599095  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2797) took 1391 ms. Trace:
I20260709 13:04:46.600065  1664 rpcz_store.cc:276] 0709 13:04:45.207237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:45.988686 (+781449us) service_pool.cc:224] Handling call
0709 13:04:46.532884 (+544198us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:46.533639 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:46.598736 (+ 65097us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54003c8f50. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:46.599001 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1234130688}
I20260709 13:04:46.562733  9612 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:46.626577  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61535.4us
I20260709 13:04:46.631047  1741 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:46.633548  9616 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:46.634673  9616 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:46.635079  9616 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:46.636611  9701 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 1.292s	user 0.004s	sys 0.015s
I20260709 13:04:46.636303  9616 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:46.637590  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 1.21525e+06us
I20260709 13:04:46.639463  9654 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:46.640286  9654 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.642970  9654 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:46.643097  9601 raft_consensus.cc:3055] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:46.643570  9601 raft_consensus.cc:740] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:46.646874  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3480) took 1865 ms. Trace:
I20260709 13:04:46.647439  1665 rpcz_store.cc:276] 0709 13:04:44.781796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.863794 (+ 81998us) service_pool.cc:224] Handling call
0709 13:04:44.864233 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:44.864647 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 13:04:44.864703 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:44.865220 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 13:04:45.124569 (+259349us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:45.124620 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.124710 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:46.635585 (+1510875us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:46.646180 (+ 10595us) raft_consensus.cc:1685] finished
0709 13:04:46.646214 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:46.646602 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10762,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5953,"spinlock_wait_cycles":5600512,"thread_start_us":258911,"threads_started":1,"wal-append.queue_time_us":450943}
I20260709 13:04:46.649895  9579 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.652704  9579 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:46.655817  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.690128  1557 leader_election.cc:400] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
I20260709 13:04:46.690747  1557 leader_election.cc:403] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:46.653385  9579 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:46.644722  9601 consensus_queue.cc:260] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:46.702939  9737 log.cc:927] Time spent T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
W20260709 13:04:46.703634  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:46.714231  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:46.714870  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:46.715255  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3483) took 1572 ms. Trace:
I20260709 13:04:46.716390  1556 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:46.699344  9616 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:46.696074  9610 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:46.715726  1663 rpcz_store.cc:276] 0709 13:04:45.142831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:45.957920 (+815089us) service_pool.cc:224] Handling call
0709 13:04:45.958598 (+   678us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:45.959295 (+   697us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:45.959324 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:45.960075 (+   751us) log.cc:844] Serialized 26 byte log entry
0709 13:04:46.221989 (+261914us) raft_consensus.cc:1645] Marking committed up to 10
0709 13:04:46.222036 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:46.222201 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:46.715008 (+492807us) raft_consensus.cc:1685] finished
0709 13:04:46.715037 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:46.715186 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1842,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":153,"thread_start_us":261309,"threads_started":1,"wal-append.queue_time_us":469636}
W20260709 13:04:46.708914  9694 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.632s	user 0.006s	sys 0.011s
I20260709 13:04:46.730186  9694 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.732581  9694 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:46.724457  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:46.736135  2310 raft_consensus.cc:2368] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:46.726055  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:46.726504  9650 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.726734  9563 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.737529  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:46.737847  9563 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.739068  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:46.740146  9715 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:46.737214  9650 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.741191  9259 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:46.741731  9259 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.742146  9259 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:46.727193  9666 consensus_queue.cc:1048] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:46.742761  9598 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.743233  9598 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:46.740736  9715 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.743968  9715 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:46.742766  9650 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.740201  9563 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:46.751792  9612 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:46.758016  2283 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.711195  9579 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:46.759052  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:46.760387  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.699561  9656 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:46.767827  9615 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.768946  9715 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:46.771654  9738 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.139s	user 0.000s	sys 0.000s
W20260709 13:04:46.773519  2315 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1152) took 2012 ms (client timeout 1911 ms). Trace:
W20260709 13:04:46.774093  2315 rpcz_store.cc:269] 0709 13:04:44.760769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.763432 (+  2663us) service_pool.cc:224] Handling call
0709 13:04:46.773246 (+2009814us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540052d0d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:46.773413 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2708565376}
W20260709 13:04:46.775897  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 151079us
I20260709 13:04:46.776964  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:46.781780  9650 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:46.713263  9654 raft_consensus.cc:3060] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 26
I20260709 13:04:46.743608  9598 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:46.757221  9714 consensus_meta.cc:287] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.541s	user 0.005s	sys 0.006s
W20260709 13:04:46.820791  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 44071.7us
I20260709 13:04:46.828136  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:46.829668  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:46.831125  9713 raft_consensus.cc:3060] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:46.832227  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:46.833981  9650 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:46.835116  9650 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:46.835536  9650 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:46.835886  9650 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:46.837214  9615 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:46.837922  9615 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:46.839401  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.840186  9615 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.840368  9688 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:46.840857  9688 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:46.841317  9688 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.843392  9572 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:46.846355  1666 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2786) took 2437 ms. Trace:
I20260709 13:04:46.847642  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:46.846792  1666 rpcz_store.cc:276] 0709 13:04:44.409124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:44.542970 (+133846us) service_pool.cc:224] Handling call
0709 13:04:44.849259 (+306289us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:44.849678 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:44.849708 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:44.850315 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 13:04:45.716901 (+866586us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:45.718677 (+  1776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:45.718857 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:45.718974 (+   117us) raft_consensus.cc:1685] finished
0709 13:04:45.719000 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:46.846032 (+1127032us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400118350. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:46.846250 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":479,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"spinlock_wait_cycles":703038848,"thread_start_us":836768,"threads_started":1,"wal-append.queue_time_us":601363,"wal-append.run_cpu_time_us":5312,"wal-append.run_wall_time_us":1055690}
W20260709 13:04:46.850736  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 116648us
W20260709 13:04:46.851910  1634 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.852509  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.853586  9615 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:46.856115  9651 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.857357  9715 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:46.857493  9579 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:46.856949  1486 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 20 to 21, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:04:46.833907  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 130021us
I20260709 13:04:46.861186  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:46.865396  9651 consensus_queue.cc:1059] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.003s
W20260709 13:04:46.858953  2283 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 136 similar messages]
W20260709 13:04:46.872838  2283 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.861166  9714 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:46.889045  1663 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:04:46.841320  9713 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:46.886651  9719 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:46.898387  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46522.9us
W20260709 13:04:46.901269  1634 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.904202  1634 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.904748  1634 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.905798  9660 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 13:04:46.907303  1634 leader_election.cc:336] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.907868  1634 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.908789  9660 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:46.910475  1634 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.911121  1634 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:46.909880  9719 consensus_queue.cc:1059] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:04:46.919273  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:46.986335  2283 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:46.941277  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:46.988605  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53620.1us
I20260709 13:04:46.929416  9660 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:46.995391  1634 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.021974  9079 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:47.022889  9079 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.024791  9650 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:47.029362  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 390584us
W20260709 13:04:47.007768  1634 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:46.942193  9259 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.006268  9715 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.041915  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 302845us
I20260709 13:04:47.031929  9654 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
I20260709 13:04:47.066567  1632 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:47.075601  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 340388us
I20260709 13:04:47.074494  1632 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:47.076439  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.042837  9616 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.096463  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.073165  9079 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.061025  9715 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:47.122326  9745 log.cc:927] Time spent T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 13:04:47.122583  9715 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:47.116353  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 513697us
W20260709 13:04:47.129735  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.117213  9618 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:47.118898  9259 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.120123  9650 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.090845  9097 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:47.094657  9689 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.156324  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 289941us
W20260709 13:04:47.157444  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 296318us
I20260709 13:04:47.091459  9616 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.169349  9651 consensus_queue.cc:1243] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 21.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 13:04:47.171056  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:47.146354  9618 raft_consensus.cc:697] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:47.178472  9618 consensus_queue.cc:237] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.181718  9563 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:47.156745  9079 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.131083  9598 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.183460  9651 raft_consensus.cc:2804] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:47.181222  1632 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:47.121346  9688 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.205165  2311 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:47.206060  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3636) took 1028 ms. Trace:
I20260709 13:04:47.206982  9688 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.206525  1733 rpcz_store.cc:276] 0709 13:04:46.068086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:46.073705 (+  5619us) service_pool.cc:224] Handling call
0709 13:04:46.074485 (+   780us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:46.075188 (+   703us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 13:04:46.410970 (+335782us) log.cc:844] Serialized 14 byte log entry
0709 13:04:47.091872 (+680902us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:47.091974 (+   102us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:47.092085 (+   111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:47.092231 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:47.095943 (+  3712us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":397992,"prepare.run_cpu_time_us":1370,"prepare.run_wall_time_us":33731,"raft.queue_time_us":52669,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":180,"thread_start_us":1005757,"threads_started":3,"wal-append.queue_time_us":584611}
I20260709 13:04:47.209353  9259 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:47.204571  9755 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.212165  9755 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.211290  1486 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 18, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 18 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.185937  9689 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.213868  1486 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:47.220482  2311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1327) took 1868 ms (client timeout 1925 ms). Trace:
I20260709 13:04:47.224712  9756 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.159863  9097 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:47.225701  9756 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.231570  9753 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.221572  2311 rpcz_store.cc:269] 0709 13:04:45.352204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:45.352387 (+   183us) service_pool.cc:224] Handling call
0709 13:04:47.220125 (+1867738us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b54007f0654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:47.220380 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4252192000}
W20260709 13:04:47.253779  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 83750us
I20260709 13:04:47.217473  9651 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:47.215788  9079 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.265484  9651 raft_consensus.cc:3060] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:47.266546  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 233479us
I20260709 13:04:47.277422  1486 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 16 to 22. New cstate: current_term: 22 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.272392  9714 raft_consensus.cc:3055] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:47.277057  1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 239) took 71198 ms (client timeout 15000 ms). Trace:
I20260709 13:04:47.286376  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2880) took 1134 ms. Trace:
W20260709 13:04:47.321374  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.290419  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.296564  9713 raft_consensus.cc:3055] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Stepping down as leader of term 18
W20260709 13:04:47.301590  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.341698  9713 raft_consensus.cc:740] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:47.342983  9713 consensus_queue.cc:260] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.330399  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.333892  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.334535  1733 raft_consensus.cc:1275] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:04:47.309971  9714 raft_consensus.cc:740] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:47.337141  1486 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 19, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.338790  9079 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.256769  9753 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.350049  9714 consensus_queue.cc:260] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.350121  1486 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.352339  1486 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 20 to 21, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:04:47.310727  1483 rpcz_store.cc:269] 0709 13:03:36.088112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:36.126575 (+ 38463us) service_pool.cc:224] Handling call
0709 13:03:36.986140 (+859565us) spinlock_profiling.cc:243] Waited 41 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb90b2 00007fcdf3e32a89 00007fcdf3dcada6 00007fcdf3d9c088 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc
0709 13:04:47.276644 (+70290504us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:34.750754 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:34.751624 (+   870us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:34.751678 (+    54us) write_op.cc:435] Acquired schema lock
0709 13:04:34.751723 (+    45us) tablet.cc:662] Decoding operations
0709 13:04:34.753658 (+  1935us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:34.753785 (+   127us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:34.753819 (+    34us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:04:34.755506 (+  1687us) tablet.cc:701] Row locks acquired
0709 13:04:34.755559 (+    53us) write_op.cc:260] PREPARE: finished
0709 13:04:34.755796 (+   237us) write_op.cc:270] Start()
0709 13:04:34.755996 (+   200us) write_op.cc:276] Timestamp: P: 1783602274755710 usec, L: 0
0709 13:04:34.756044 (+    48us) op_driver.cc:348] REPLICATION: starting
0709 13:04:34.757418 (+  1374us) log.cc:844] Serialized 6353 byte log entry
0709 13:04:35.041992 (+284574us) op_driver.cc:464] REPLICATION: finished
0709 13:04:35.313614 (+271622us) write_op.cc:301] APPLY: starting
0709 13:04:35.313785 (+   171us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:35.367599 (+ 53814us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:35.367845 (+   246us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:35.628702 (+260857us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:35.629190 (+   488us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=32,mrs_lookups=0
0709 13:04:35.629278 (+    88us) write_op.cc:312] APPLY: finished
0709 13:04:35.691795 (+ 62517us) log.cc:844] Serialized 148 byte log entry
0709 13:04:35.732402 (+ 40607us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:35.732658 (+   256us) write_op.cc:454] Released schema lock
0709 13:04:35.732766 (+   108us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:35.733449 (+   683us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:35.733490 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":214228,"reported_tablets":52,"spinlock_wait_cycles":94270208,"child_traces":[["op",{"apply.queue_time_us":256642,"apply.run_cpu_time_us":28518,"apply.run_wall_time_us":419782,"cfile_cache_hit":66,"cfile_cache_hit_bytes":71039,"num_ops":16,"prepare.queue_time_us":10056,"prepare.run_cpu_time_us":11306,"prepare.run_wall_time_us":31046,"raft.queue_time_us":192601,"raft.run_cpu_time_us":13521,"raft.run_wall_time_us":150059,"replication_time_us":231085,"thread_start_us":135385,"threads_started":4,"wal-append.queue_time_us":7836,"wal-append.run_cpu_time_us":60234,"wal-append.run_wall_time_us":2279576}]]}
I20260709 13:04:47.302603  1557 leader_election.cc:400] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:47.354586  1557 leader_election.cc:403] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 13:04:47.348806  9713 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:04:47.311587  1634 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:47.351699  9714 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:47.372066  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62752.6us
W20260709 13:04:47.370414  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 328600us
W20260709 13:04:47.382145  1634 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.383880  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.322487  1734 rpcz_store.cc:276] 0709 13:04:46.151556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:46.311021 (+159465us) service_pool.cc:224] Handling call
0709 13:04:46.311391 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:46.311769 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:46.606809 (+295040us) log.cc:844] Serialized 14 byte log entry
0709 13:04:47.277948 (+671139us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:47.278145 (+   197us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:47.280463 (+  2318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:47.280597 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:47.285097 (+  4500us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":213298,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":192437,"raft.run_cpu_time_us":69,"raft.run_wall_time_us":68,"thread_start_us":929324,"threads_started":3,"wal-append.queue_time_us":542821}
I20260709 13:04:47.374420  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2888) took 1150 ms. Trace:
W20260709 13:04:47.372110  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 69766.1us
W20260709 13:04:47.383912  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74695.2us
I20260709 13:04:47.382717  1634 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:47.318212  9579 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.390412  1733 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:47.350104  9616 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:47.393963  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:47.392500  1634 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.395730  1634 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:47.379345  9656 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.003s	sys 0.022s
I20260709 13:04:47.384472  1486 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.380832  9719 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:47.371618  9751 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:47.401810  9643 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.017s	sys 0.018s
W20260709 13:04:47.421895  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 140374us
I20260709 13:04:47.422822  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.400449  9259 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:47.400452  9770 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:47.404206  9656 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.003s	sys 0.022s
I20260709 13:04:47.385810  1738 rpcz_store.cc:276] 0709 13:04:46.223763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:46.410321 (+186558us) service_pool.cc:224] Handling call
0709 13:04:46.434918 (+ 24597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:46.549670 (+114752us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 13:04:46.572830 (+ 23160us) log.cc:844] Serialized 14 byte log entry
0709 13:04:46.937090 (+364260us) log.cc:844] Serialized 14 byte log entry
0709 13:04:47.240951 (+303861us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:47.241049 (+    98us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:47.241142 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:47.374069 (+132927us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b5400435b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:47.374144 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:47.374337 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2101,"prepare.queue_time_us":34018,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":130,"raft.queue_time_us":573733,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":221,"spinlock_wait_cycles":262957440,"thread_start_us":638149,"threads_started":2,"wal-append.queue_time_us":527246}
I20260709 13:04:47.436173  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.443892  1740 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 18.
I20260709 13:04:47.381244  9097 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.443205  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:47.419757  1486 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 4 to 10, term changed from 15 to 17, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to <none>, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted, no longer has a pending config: VOTER ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194), VOTER df6a18af06d543b2931837f5e6bab95f (127.0.107.195), VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 17 committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:04:47.430301  9643 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.017s	sys 0.019s
W20260709 13:04:47.429482  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 283134us
W20260709 13:04:47.454792  1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 258) took 15993 ms (client timeout 15000 ms). Trace:
I20260709 13:04:47.485881  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:47.487530  1483 rpcz_store.cc:269] 0709 13:04:31.461238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:47.454404 (+15993166us) service_pool.cc:211] Skipping call since client already timed out
0709 13:04:47.454645 (+   241us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:04:47.488518  1734 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:47.456138  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:47.491811  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54781.3us
W20260709 13:04:47.514882  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 249813us
W20260709 13:04:47.534639  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 94934us
I20260709 13:04:47.450196  9755 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:47.455065  9689 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.533612  9598 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.540328  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.541764  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:47.542378  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.543303  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.541901  1486 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:47.547833  2315 raft_consensus.cc:2410] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 13:04:47.552914  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.553187  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:47.554198  2309 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:47.431012  9579 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.562638  2310 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:47.562556  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:47.559296  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:47.559722  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68350.6us
I20260709 13:04:47.568118  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:47.569051  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.558887  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:47.571405  1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 241) took 15989 ms (client timeout 15000 ms). Trace:
W20260709 13:04:47.571862  1483 rpcz_store.cc:269] 0709 13:04:31.582159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:47.564430 (+15982271us) service_pool.cc:211] Skipping call since client already timed out
0709 13:04:47.571300 (+  6870us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 13:04:47.574363  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:47.575311  1733 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:47.548110  9601 raft_consensus.cc:3060] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:47.581693  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.582717  1663 raft_consensus.cc:2368] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24.
I20260709 13:04:47.586922  2317 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:47.586923  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.588276  1662 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
W20260709 13:04:47.590600  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 195691us
I20260709 13:04:47.593091  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.594923  9760 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:47.595757  9760 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:47.598157  1666 raft_consensus.cc:1240] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:47.601140  9579 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:47.493197  9770 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.568132  9689 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 13:04:47.608173  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.566874  9656 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:47.607334  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 50701.7us
I20260709 13:04:47.614784  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:47.616672  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:47.617647  2280 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:47.617808  9761 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.618436  9761 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.619601  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:47.619915  9766 raft_consensus.cc:2804] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:04:47.637028  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.596817  9760 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.638087  1663 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:47.638199  1740 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 27.
I20260709 13:04:47.647042  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:47.651381  9766 raft_consensus.cc:697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:47.652580  9766 consensus_queue.cc:237] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.654277  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.656919  1667 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:47.660372  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:47.661415  9770 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.662292  9765 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 13:04:47.663741  9765 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:47.664136  9765 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:47.671926  9719 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
W20260709 13:04:47.673532  9774 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 13:04:47.673866  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.674824  9768 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:47.674991  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:47.672734  9651 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.683300  2313 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:47.686689  1738 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:47.690167  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:47.690788  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 98142.9us
I20260709 13:04:47.691116  1735 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:47.697032  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.654178  9598 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.714637  1740 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:47.675315  9768 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:47.713606  1734 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:47.718490  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.718776  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:47.660267  9761 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.689841  9650 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:47.686818  9759 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.690303  9715 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:47.708599  9656 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.749276  1705 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:47.749665  9672 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:47.749771  9656 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.750730  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.750664  9672 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.752542  9656 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.753186  9762 raft_consensus.cc:2804] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:47.757493  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.758646  1663 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:47.743614  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.747519  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.747997  9747 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:47.748466  9769 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:47.724866  9616 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:47.728575  9593 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:04:47.748250  9650 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.755538  1705 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:47.726939  9768 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:47.777462  9672 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.781656  9754 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:47.781896  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.782655  2315 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:47.783102  9755 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 13:04:47.783566  9755 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:47.783947  9755 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Advancing to term 29
I20260709 13:04:47.788486  9756 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.665197  9765 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.760885  9759 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:47.771021  9616 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.773727  9757 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:47.728554  9660 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:47.764150  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 155472us
W20260709 13:04:47.774552  1634 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:47.807293  1634 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:47.796813  2312 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
I20260709 13:04:47.777781  2309 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:47.736653  9618 consensus_queue.cc:1048] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 13:04:47.746073  9715 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:47.792981  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:47.771194  9770 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:47.803494  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.810410  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.811487  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:47.728833  9598 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.804327  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:47.857649  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.817800  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.834022  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.873386  1664 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:47.873674  1662 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:47.874181  1664 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:47.871562  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 221562us
I20260709 13:04:47.813689  2314 raft_consensus.cc:1240] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 13:04:47.871238  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:47.873711  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.857026  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:47.876441  1660 raft_consensus.cc:2368] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:47.858203  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:47.832490  2280 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:47.878163  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45791.4us
I20260709 13:04:47.874404  2315 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 13.
I20260709 13:04:47.889046  1665 raft_consensus.cc:2410] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 9.
I20260709 13:04:47.882524  1733 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
W20260709 13:04:47.835016  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.890905  1667 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:47.894258  2317 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
W20260709 13:04:47.826660  9777 log.cc:927] Time spent T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 13:04:47.906904  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:47.894394  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:47.902654  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 183947us
I20260709 13:04:47.879917  9765 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:47.815181  9672 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:47.835834  9650 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.910247  1557 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:47.820453  9651 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.852358  9759 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:47.848163  9616 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:47.887765  9593 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:47.888427  9715 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:47.955925  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:47.956143  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45867.3us
I20260709 13:04:47.932340  1735 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:04:47.973937  1662 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 23: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:47.956703  2281 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:47.951472  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:47.888041  9787 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:47.961158  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:47.965891  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 172930us
I20260709 13:04:47.803253  9747 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.932083  1632 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:47.796844  9769 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:47.983915  1664 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:47.985641  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:47.914548  9610 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:47.913059  9760 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:47.993803  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.015725  1663 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
W20260709 13:04:47.999295  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.000109  1660 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:47.825706  9660 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.002149  1665 raft_consensus.cc:1240] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 13:04:48.002211  9097 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:48.003155  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 90544.4us
I20260709 13:04:48.005388  9767 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 13:04:48.009305  9791 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.010548  9747 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.011353  9593 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.020655  2315 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:47.965411  9656 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election won for term 25
W20260709 13:04:48.030145  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 274659us
W20260709 13:04:48.030576  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 98592.2us
I20260709 13:04:47.985234  9771 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:48.045747  9771 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.047924  9771 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.048594  9753 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.039382  9791 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.052917  9791 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.056463  9097 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:48.057027  9097 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.059185  9598 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.059514  9572 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.060020  9572 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.029311  9760 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:48.064432  9760 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.065712  9656 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:48.066952  9656 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:48.053191  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.070374  1735 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
I20260709 13:04:48.071009  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.071779  1741 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:48.072552  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.073305  1740 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 14: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:48.076568  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.037266  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.036947  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:48.015517  9610 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:48.041903  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.044220  9769 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.078886  9610 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:48.079494  9787 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:48.079926  9787 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.080581  9787 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:48.081074  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.073227  1734 raft_consensus.cc:1275] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:48.082165  1740 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:48.090197  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.083133  9757 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:04:48.091365  1632 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.091696  9757 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.092167  9757 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:48.092339  9615 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:48.092808  9615 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.093183  9615 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:48.094247  9672 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:48.092167  9259 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:04:48.094954  9672 raft_consensus.cc:697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:48.095132  9259 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.095597  9259 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:48.096036  9672 consensus_queue.cc:237] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:48.098119  1634 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.098647  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.089377  9610 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.098878  9079 consensus_queue.cc:1059] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:48.099498  9759 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.099773  1735 raft_consensus.cc:1240] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
W20260709 13:04:48.084496  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.592s (SENT)
I20260709 13:04:48.086310  9616 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:47.999428  9651 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:48.103322  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.098680  1634 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:48.107952  1667 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:48.110184  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.033656  9762 raft_consensus.cc:697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:48.093288  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:48.104461  9601 consensus_meta.cc:287] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.000s
I20260709 13:04:48.059218  9097 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.119767  9616 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:48.120086  9610 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.144543  9616 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:48.121887  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.125669  9759 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.146931  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.126840  9660 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.127280  9755 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.133966  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:48.134073  1476 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet e676e54141f7470db4551f4de439c017 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica e6adef9da7da44f6af0af5695cf9b704 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:48.134786  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:48.204428  1476 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 70564.4us
I20260709 13:04:48.138178  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.146169  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:48.162253  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52162.4us
W20260709 13:04:48.201556  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79694.8us
I20260709 13:04:48.211331  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:48.208652  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.140343  9572 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:48.279306  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 70771.2us
I20260709 13:04:48.225695  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:48.283545  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 199050us
W20260709 13:04:48.285332  1708 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.603s (SENT)
I20260709 13:04:48.203718  9650 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.205893  9713 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.207266  9660 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.206835  9610 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:48.140733  9651 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:48.317066  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 170964us
I20260709 13:04:48.319378  9650 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.285162  9791 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:48.223284  9766 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:48.145428  9768 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.323858  9770 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:48.325042  9650 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.330106  9762 consensus_queue.cc:237] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.330755  9615 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.299058  9572 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:48.333876  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48600.1us
W20260709 13:04:48.350945  1708 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.643s (SENT)
I20260709 13:04:48.336568  2281 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:48.295781  9794 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:48.213905  9751 consensus_queue.cc:1059] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.119s
I20260709 13:04:48.356827  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.360682  2312 raft_consensus.cc:2368] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
W20260709 13:04:48.359347  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 266085us
I20260709 13:04:48.357350  9794 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.363108  9794 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:48.363564  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.691s (SENT)
I20260709 13:04:48.363907  9755 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.364078  9615 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:48.364681  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.707s (SENT)
I20260709 13:04:48.213795  9616 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.374199  1632 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.361129  9079 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:48.362073  9579 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 13 FOLLOWER]: Advancing to term 14
I20260709 13:04:48.340866  9791 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.365142  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.400725  9615 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.384177  9805 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:48.406409  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:48.406988  9805 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.379990  9766 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 13:04:48.414714  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.417006  9809 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:48.406451  9615 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.417430  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.417762  9809 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:48.418257  1662 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:48.417062  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.419998  1660 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:04:48.412940  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.422199  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.422099  9809 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.419032  9756 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:48.423156  1665 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:48.417006  9805 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.424320  9615 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.364923  9598 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:48.422351  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:48.438347  1735 raft_consensus.cc:1275] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 13:04:48.440156  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.411720  9759 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.378620  9753 consensus_queue.cc:1048] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
W20260709 13:04:48.484925  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 125297us
I20260709 13:04:48.489076  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:48.489683  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 132770us
I20260709 13:04:48.489995  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.490219  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:48.490703  1736 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:48.493587  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 395442us
I20260709 13:04:48.314265  9713 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:48.495719  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:48.504410  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.411989  9766 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:48.513932  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.514971  1666 raft_consensus.cc:2410] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 25 because replica has last-logged OpId of term: 24 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 13:04:48.514814  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.515632  1740 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:48.516309  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.518612  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.520192  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.488677  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.533090  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.533319  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.533849  1734 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
I20260709 13:04:48.523412  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
W20260709 13:04:48.511668  1634 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.536059  1634 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:48.323345  9660 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.541040  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:48.542971  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 178321us
W20260709 13:04:48.549338  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.734s (SENT)
I20260709 13:04:48.544067  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:48.544934  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:48.547168  1556 leader_election.cc:400] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24.
I20260709 13:04:48.551622  1556 leader_election.cc:403] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:48.553581  1556 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:48.491608  9650 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:48.537674  1634 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.557152  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 144317us
I20260709 13:04:48.557659  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:48.558019  1634 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.563406  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:48.559259  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.466594  9598 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:48.479504  9769 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.602315  2314 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:48.532670  9766 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:48.445267  9656 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.114s
I20260709 13:04:48.555900  9760 raft_consensus.cc:3060] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 24
I20260709 13:04:48.534193  9759 raft_consensus.cc:3060] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:48.604266  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:48.651918  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.652459  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:48.663993  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59762.8us
I20260709 13:04:48.626178  1736 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
W20260709 13:04:48.627581  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 74155.4us
W20260709 13:04:48.583701  9787 consensus_meta.cc:287] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.503s	user 0.009s	sys 0.003s
I20260709 13:04:48.548467  9616 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:48.635383  9800 log.cc:927] Time spent T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.226s	user 0.000s	sys 0.000s
I20260709 13:04:48.639619  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:48.680667  1667 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 14.
W20260709 13:04:48.673612  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 171514us
W20260709 13:04:48.683158  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92918.6us
W20260709 13:04:48.615502  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.951s (SENT)
I20260709 13:04:48.686734  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.690119  1736 raft_consensus.cc:2376] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23. Re-sending same reply.
I20260709 13:04:48.708187  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.709331  1742 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 23: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:48.556996  9650 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.715529  1556 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.651144  9768 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:48.720160  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:48.722702  1668 raft_consensus.cc:2393] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 20: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:48.589778  9816 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.137s
I20260709 13:04:48.726881  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.727711  1736 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:48.722463  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:48.730860  1667 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:48.612282  9769 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:48.709707  9780 log.cc:927] Time spent T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 13:04:48.710886  9785 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 13:04:48.689702  9079 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:48.745194  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.692526  9601 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 13:04:48.772936  1736 raft_consensus.cc:2376] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25. Re-sending same reply.
W20260709 13:04:48.781625  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 103492us
I20260709 13:04:48.743743  9791 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:48.740851  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.787138  1742 raft_consensus.cc:2376] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23. Re-sending same reply.
I20260709 13:04:48.779536  1738 raft_consensus.cc:1275] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:04:48.688733  9787 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.756233  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:48.771261  9259 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.675s	user 0.010s	sys 0.006s
I20260709 13:04:48.788012  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.796679  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:48.774657  9601 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:48.804669  1742 raft_consensus.cc:2393] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 20: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:48.802201  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:48.808113  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 192905us
I20260709 13:04:48.810708  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:48.805172  1634 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:48.809646  1708 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 1.966s (SENT)
I20260709 13:04:48.815598  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.818241  2311 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21.
W20260709 13:04:48.797600  9804 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.423s	user 0.000s	sys 0.002s
W20260709 13:04:48.817466  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:48.827225  9814 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 13:04:48.827229  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.828918  9650 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:48.778797  9598 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:48.831589  9769 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:48.832791  9765 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.879s	user 0.009s	sys 0.002s
I20260709 13:04:48.834309  1667 raft_consensus.cc:2468] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 26.
I20260709 13:04:48.821555  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.774888  9079 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:48.859403  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.808117  9601 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:48.797176  9259 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:48.837756  9757 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.745s	user 0.013s	sys 0.000s
I20260709 13:04:48.920409  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.923866  2280 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:48.937857  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:48.936944  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.936084  2312 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:48.940366  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.855913  9770 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 13:04:48.839100  9827 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.859499  9651 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:48.944849  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 135217us
W20260709 13:04:48.949270  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.002s (SENT)
I20260709 13:04:48.949771  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:48.920342  9817 log.cc:927] Time spent T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 13:04:48.943787  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.948189  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:48.951479  1668 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Advancing to term 29
I20260709 13:04:48.955580  1739 raft_consensus.cc:2368] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25.
W20260709 13:04:48.956128  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:48.966567  1736 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 27.
I20260709 13:04:48.917593  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:48.952586  9715 consensus_meta.cc:287] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.959s	user 0.011s	sys 0.006s
I20260709 13:04:48.944712  9756 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:48.979070  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61578.9us
I20260709 13:04:48.980268  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:49.092905  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 377399us
W20260709 13:04:49.095999  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 115823us
I20260709 13:04:49.097306  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:48.947095  9757 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.106707  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129609us
I20260709 13:04:48.976862  9771 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 13:04:49.114962  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:49.110679  1557 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:49.117926  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 168687us
I20260709 13:04:49.112986  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:49.194696  1708 leader_election.cc:336] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.027s (SENT)
I20260709 13:04:49.195235  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:49.187122  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74698.9us
I20260709 13:04:48.954149  9827 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.196695  1668 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 29.
W20260709 13:04:49.190477  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:49.088572  9765 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:49.103521  9770 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 13:04:49.109418  9764 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:49.088100  9756 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.109050  9803 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:48.977953  9715 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.109251  9813 raft_consensus.cc:2804] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 13:04:49.207366  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 110281us
W20260709 13:04:49.201020  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 88142.3us
I20260709 13:04:49.202147  9769 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:49.250560  9831 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.310s	user 0.001s	sys 0.000s
I20260709 13:04:49.253662  9660 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.257375  9754 raft_consensus.cc:3055] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 13:04:49.205758  9765 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:49.206658  9770 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.294124  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:49.263692  9829 log.cc:927] Time spent T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.154s	user 0.000s	sys 0.000s
I20260709 13:04:49.240890  9808 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:49.271426  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 47754.2us
I20260709 13:04:49.262149  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.191200  9826 consensus_queue.cc:1059] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.080s
I20260709 13:04:49.351500  1741 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:04:49.304289  1661 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:49.211444  9838 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:49.185021  9828 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.414s
W20260709 13:04:49.373849  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 179170us
W20260709 13:04:49.374442  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 104182us
W20260709 13:04:49.374630  1708 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.037s (SENT)
I20260709 13:04:49.375080  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:49.366837  9794 consensus_meta.cc:287] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 1.003s	user 0.015s	sys 0.000s
I20260709 13:04:49.367218  9835 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:49.368111  9842 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.298528  9754 raft_consensus.cc:740] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:49.368681  9806 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:49.298671  9770 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:49.294344  9787 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.294534  9813 raft_consensus.cc:697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:49.394474  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:49.396397  1741 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:49.397980  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 287366us
W20260709 13:04:49.390864  9610 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s	user 0.014s	sys 0.028s
I20260709 13:04:49.399384  1557 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.403045  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:49.383499  9762 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.010s	sys 0.002s
I20260709 13:04:49.291373  9803 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.390117  9593 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:49.408634  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:49.458422  2317 raft_consensus.cc:3060] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Advancing to term 29
I20260709 13:04:49.460520  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:49.462710  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 88101.2us
W20260709 13:04:49.465983  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57436.6us
W20260709 13:04:49.466295  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 14 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Timed out: RequestConsensusVote RPC to 127.0.107.193:41267 timed out after 2.124s (SENT)
W20260709 13:04:49.409907  9610 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.014s	sys 0.028s
I20260709 13:04:49.463641  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:49.465265  1633 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.040s	sys 0.044s
W20260709 13:04:49.467487  1633 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.041s	sys 0.044s
I20260709 13:04:49.467170  2315 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:49.401078  9806 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:49.412276  9838 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:49.414753  9828 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:04:49.459347  9762 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.011s	sys 0.003s
I20260709 13:04:49.467911  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:49.479844  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:49.483230  9601 consensus_meta.cc:287] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.000s
I20260709 13:04:49.499058  9601 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.483692  9759 consensus_meta.cc:287] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.791s	user 0.006s	sys 0.007s
I20260709 13:04:49.500787  9759 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:49.437868  9259 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.484313  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:49.379347  9794 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.486608  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:49.520012  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:49.488176  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.414642  9809 consensus_queue.cc:1048] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.072s
I20260709 13:04:49.492147  1664 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:49.498127  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.524369  1740 raft_consensus.cc:2468] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:49.488936  9579 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 1.082s	user 0.012s	sys 0.014s
I20260709 13:04:49.496908  2317 raft_consensus.cc:2468] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 29.
W20260709 13:04:49.521670  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53839.6us
I20260709 13:04:49.473934  9826 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:49.413928  9791 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 13:04:49.488299  9850 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:49.412662  9835 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.520606  9754 consensus_queue.cc:260] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 17.16, Last appended by leader: 16, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.524420  9757 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.524210  9259 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:49.539739  9826 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.542936  9579 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 13:04:49.571758  2310 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 14.
W20260709 13:04:49.572168  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 1.36219e+06us
I20260709 13:04:49.413482  9651 consensus_queue.cc:1059] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
W20260709 13:04:49.545313   431 thread.cc:527] Waited for 1000ms trying to join with rpc worker (tid 1564)
W20260709 13:04:49.546845  9766 consensus_meta.cc:287] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.872s	user 0.013s	sys 0.000s
W20260709 13:04:49.574707  9834 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.376s	user 0.000s	sys 0.001s
I20260709 13:04:49.574761  9766 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.576797  9844 raft_consensus.cc:3055] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:49.576570  9806 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.577301  9844 raft_consensus.cc:740] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:49.548799  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 82519.4us
I20260709 13:04:49.578199  9844 consensus_queue.cc:260] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.549057  9813 consensus_queue.cc:237] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:49.580855  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 92875.8us
I20260709 13:04:49.552505  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.581967  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.582280  9827 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.582872  9827 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:49.583069  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.583300  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.466202  9765 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.393630  9808 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.584834  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.390514  9842 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.604769  9616 consensus_meta.cc:287] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.917s	user 0.008s	sys 0.004s
I20260709 13:04:49.605381  9651 consensus_queue.cc:1243] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 13:04:49.606513  9842 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.563766  9259 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:49.607530  9259 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:49.610715  9760 consensus_meta.cc:287] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.928s	user 0.011s	sys 0.001s
I20260709 13:04:49.611174  9760 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24.
I20260709 13:04:49.541787  1660 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 28: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:49.614674  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:49.615216  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:49.615945  1660 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:04:49.553305  9601 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.557204  9803 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.583240  9827 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.559016  9610 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:49.559180  9593 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:49.561175  1733 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:49.559311  9835 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.603363  9840 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.589196  9850 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:49.604130  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:49.604233  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:49.604383  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:49.628001  1740 raft_consensus.cc:2368] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 28. Current term is 29.
I20260709 13:04:49.540450  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:49.538378  1741 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:49.564913  9826 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:49.568845  9839 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.301s	user 0.001s	sys 0.000s
W20260709 13:04:49.629175  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 88849.5us
W20260709 13:04:49.629383  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 254177us
I20260709 13:04:49.560822  9794 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:49.633133  1556 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:49.621337  9807 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.635823  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:49.638639  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.624619  9827 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:49.647410  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:49.653119  1557 leader_election.cc:400] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25.
I20260709 13:04:49.654390  1557 leader_election.cc:403] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:49.624784  9765 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:49.570358  9715 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.670473  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2898) took 1086 ms. Trace:
I20260709 13:04:49.634362  9850 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:49.682929  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.682801  1740 rpcz_store.cc:276] 0709 13:04:48.583771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.649682 (+1065911us) service_pool.cc:224] Handling call
0709 13:04:49.669652 (+ 19970us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:49.670047 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:49.670089 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:49.670136 (+    47us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:49.670174 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:49.670239 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:49.670396 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:49.640348  9835 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:49.631706  9803 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:49.684794  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.646241  9838 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:49.626456  9610 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.687315  1660 raft_consensus.cc:2468] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:49.686034  9616 raft_consensus.cc:2804] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:49.652846  9601 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:49.628597  9593 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.689268  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110565us
I20260709 13:04:49.666900  9855 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:49.651961  9791 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:49.716655  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 78086.3us
I20260709 13:04:49.690011  9838 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:49.689625  9760 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 25
W20260709 13:04:49.696959  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 124014us
I20260709 13:04:49.730470  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:49.737172  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.735108  9756 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:49.742529  1742 raft_consensus.cc:1240] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 13:04:49.739120  1705 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:49.691301  9616 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:49.752092  2283 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:49.753472  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:49.752378  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:49.754976  1740 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:49.758432  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.758941  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:49.803517  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3669) took 1289 ms. Trace:
I20260709 13:04:49.736466  9803 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:49.806841  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 57468.1us
W20260709 13:04:49.808921  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.803984  1735 rpcz_store.cc:276] 0709 13:04:48.467042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.492690 (+ 25648us) service_pool.cc:224] Handling call
0709 13:04:48.493096 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:48.578466 (+ 85370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:48.578499 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:48.613753 (+ 35254us) log.cc:844] Serialized 26 byte log entry
0709 13:04:48.942667 (+328914us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:48.942707 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:48.944899 (+  2192us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:49.568062 (+623163us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:49.741823 (+173761us) raft_consensus.cc:1685] finished
0709 13:04:49.741856 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:49.756133 (+ 14277us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8199,"raft.queue_time_us":31825,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":31154,"spinlock_wait_cycles":6534400,"thread_start_us":249596,"threads_started":1,"wal-append.queue_time_us":448836}
I20260709 13:04:49.756474  9616 raft_consensus.cc:3060] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:49.811427  2318 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:49.812826  9610 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:49.800437  9857 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:04:49.813814  9857 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.811437  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.815809  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.816452  1735 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:49.809455  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.818493  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:49.730772  9828 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:49.822371  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.741009  9769 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.753795  9756 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:49.744927  9835 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.744688  9808 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.823230  1735 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 26.
I20260709 13:04:49.829825  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:49.830549  1735 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:49.743677  9838 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:49.826642  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:49.833153  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:49.649304  9764 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.176s
I20260709 13:04:49.731361  9855 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.828845  9615 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:49.631886  9840 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.841079  9868 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:49.803977  9842 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 13:04:49.814186  9857 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:49.823685  9824 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:49.757030  9791 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.735718  9850 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.831419  9610 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:49.650707  9807 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.842821  9756 raft_consensus.cc:697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
W20260709 13:04:49.889982  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81085.7us
I20260709 13:04:49.829308  9769 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:49.895637  1708 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.898305  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:49.899748  1740 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 28.
I20260709 13:04:49.835012  9835 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.881919  9824 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:49.902837  9593 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:49.853222  9615 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.886735  9601 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:49.905974  2318 raft_consensus.cc:2410] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 21 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 13:04:49.904950  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:49.908138  1708 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.908275  1735 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 19.
I20260709 13:04:49.909777  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:49.908241  1740 raft_consensus.cc:2410] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:49.901364  9764 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 13:04:49.914204  1632 leader_election.cc:400] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 28. Current term is 29.
I20260709 13:04:49.914734  1632 leader_election.cc:403] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
W20260709 13:04:49.912909  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 154595us
I20260709 13:04:49.918502  1556 leader_election.cc:400] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:49.903661  9835 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.921562  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.922744  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:49.924163  1740 raft_consensus.cc:2368] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:49.899549  9770 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:49.928910  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 55811.8us
I20260709 13:04:49.930469  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.919906  9806 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 28. Current term is 29.
W20260709 13:04:49.932739  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:49.933226  9835 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:49.934583  1557 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.935487  9835 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:49.935892  9835 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:49.936232  9835 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:49.937608  9756 consensus_queue.cc:237] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.9, Last appended by leader: 9, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.940973  9615 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:49.945919  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:49.979919  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.981331  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:49.982098  1664 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:49.983238  1557 leader_election.cc:400] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
I20260709 13:04:49.983637  1557 leader_election.cc:403] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:49.984105  1736 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:49.947615  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:49.901585  9803 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:49.986138  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:49.986304  1556 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.986869  1660 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:49.984768  1708 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:49.988008  1708 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:49.983230  9842 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:49.993594  9601 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:49.994635  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:50.015456  2313 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:49.997370  9840 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.997709  9770 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:49.999933  9807 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.000672  9824 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:50.000672  9869 consensus_queue.cc:1059] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:50.002267  9871 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:49.907083  9828 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.003011  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:50.010800  1660 raft_consensus.cc:2468] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
W20260709 13:04:50.232632  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 247862us
I20260709 13:04:50.233754  9713 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:50.233958  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.234535  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.013018  9842 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:50.204739  2280 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:50.260588  2313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:50.178969  9610 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:50.179513  9593 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.197747  9840 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:50.261037  9807 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:50.002933  9827 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.292755  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.294595  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:50.279896  9754 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Advancing to term 18
I20260709 13:04:50.262228  9842 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:04:50.299741  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 95140.5us
W20260709 13:04:50.288695  9865 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 13:04:50.198570  9874 raft_consensus.cc:3060] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 26
I20260709 13:04:50.238412  9770 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.259186  9869 consensus_queue.cc:1243] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:50.332284  1660 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 19. Current term is 20. Ops: []
I20260709 13:04:50.299396  9855 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:50.338665  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 355714us
I20260709 13:04:50.188303  9835 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:50.323802  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 89868.2us
I20260709 13:04:50.301000  9824 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.311362  9572 raft_consensus.cc:2804] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:50.346928  1663 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3498) took 1942 ms. Trace:
W20260709 13:04:50.340433  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:50.312597  9791 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.353307  1663 rpcz_store.cc:276] 0709 13:04:48.404223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.404430 (+   207us) service_pool.cc:224] Handling call
0709 13:04:48.408120 (+  3690us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:48.408538 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:48.408574 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:48.444421 (+ 35847us) log.cc:844] Serialized 26 byte log entry
0709 13:04:50.346243 (+1901822us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b180013cde0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdedb854bc 00007fcdeaed348b 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1
0709 13:04:50.346450 (+   207us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:50.346488 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.346568 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:50.346619 (+    51us) raft_consensus.cc:1685] finished
0709 13:04:50.346632 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.346860 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42517,"raft.queue_time_us":12024,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":21377,"spinlock_wait_cycles":12072832,"thread_start_us":43520,"threads_started":1,"wal-append.queue_time_us":183249,"wal-append.run_cpu_time_us":1769,"wal-append.run_wall_time_us":1004186}
I20260709 13:04:50.347110  9572 raft_consensus.cc:697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:50.326540  9762 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:50.295892  1664 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:04:50.368670  2280 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:50.353909  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:50.345901  9824 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:50.351788  2317 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1341) took 1859 ms (client timeout 1787 ms). Trace:
I20260709 13:04:50.301473  9593 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.373299  1660 raft_consensus.cc:1275] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 13:04:50.371345  2317 rpcz_store.cc:269] 0709 13:04:48.492229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.492395 (+   166us) service_pool.cc:224] Handling call
0709 13:04:50.351601 (+1859206us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54006a1b54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.351709 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2035968384}
W20260709 13:04:50.374437  2281 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:50.374660  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 388408us
I20260709 13:04:50.315718  9807 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.379556  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:50.373597  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:50.381819  9572 consensus_queue.cc:237] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.382725  2312 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1171) took 1931 ms. Trace:
I20260709 13:04:50.362068  9869 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 13:04:50.378737  9877 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.393045  2312 rpcz_store.cc:276] 0709 13:04:48.451434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.451673 (+   239us) service_pool.cc:224] Handling call
0709 13:04:50.382523 (+1930850us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400750dd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.382646 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023701248}
I20260709 13:04:50.366182  9837 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:50.391564  9765 consensus_meta.cc:287] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.707s	user 0.014s	sys 0.000s
I20260709 13:04:50.375887  9812 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.338413  9770 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:50.397154  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:50.410221  1708 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.392379  9813 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:04:50.414943  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.398401  9762 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.411628  9875 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:50.423309  9824 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:50.431335  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.432202  1737 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:04:50.412544  9601 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:50.451066  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.451508  9807 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.452196  1735 raft_consensus.cc:2410] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
W20260709 13:04:50.454632  1733 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2911) took 1645 ms (client timeout 2033 ms). Trace:
W20260709 13:04:50.455132  1733 rpcz_store.cc:269] 0709 13:04:48.809135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.855569 (+ 46434us) service_pool.cc:224] Handling call
0709 13:04:50.454384 (+1598815us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b54000e33d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.454540 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1432496384}
I20260709 13:04:50.458717  9650 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:50.459232  9650 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:50.460249  9650 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.462306  9835 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:50.433095  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.475917  1737 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 13:04:50.477349  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1385) took 1361 ms. Trace:
I20260709 13:04:50.491210  1735 rpcz_store.cc:276] 0709 13:04:49.115885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.452876 (+1336991us) service_pool.cc:224] Handling call
0709 13:04:50.453221 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.453426 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.453456 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.477040 (+ 23584us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:50.477089 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.477148 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.477285 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:50.462659  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:50.480409  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2912) took 1643 ms. Trace:
I20260709 13:04:50.497959  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:50.499845  1735 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:50.493561  1737 rpcz_store.cc:276] 0709 13:04:48.837011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.432896 (+1595885us) service_pool.cc:224] Handling call
0709 13:04:50.433226 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.476833 (+ 43607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.480325 (+  3492us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:50.482870  9616 consensus_meta.cc:287] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.670s	user 0.013s	sys 0.001s
I20260709 13:04:50.474118  9828 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:50.484745  9760 consensus_meta.cc:287] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.739s	user 0.013s	sys 0.000s
I20260709 13:04:50.485720  9650 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:50.408839  9837 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:50.516881  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:50.540614  1737 raft_consensus.cc:1240] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 17. Current term is 18. Ops: []
W20260709 13:04:50.538415  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 105352us
W20260709 13:04:50.542876  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.418892  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:50.543591  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:50.544118  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 125307us
I20260709 13:04:50.508034  9601 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.550149  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1390) took 1200 ms. Trace:
I20260709 13:04:50.550601  1737 rpcz_store.cc:276] 0709 13:04:49.349902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.513244 (+1163342us) service_pool.cc:224] Handling call
0709 13:04:50.540410 (+ 27166us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.549822 (+  9412us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.550082 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.507336  9835 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.475059  9762 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:50.559242  1668 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3510) took 1842 ms (client timeout 1787 ms). Trace:
I20260709 13:04:50.540027  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.418640  9812 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.526837  9760 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25.
W20260709 13:04:50.559911  1668 rpcz_store.cc:269] 0709 13:04:48.709140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.755851 (+ 46711us) service_pool.cc:224] Handling call
0709 13:04:50.550747 (+1794896us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540013ecd4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.550865 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362730240}
I20260709 13:04:50.548976  9869 raft_consensus.cc:2804] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:50.518675  9879 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:50.413640  9877 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.539830  9097 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:50.573962  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3678) took 1177 ms. Trace:
I20260709 13:04:50.568007  9869 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:50.525851  9616 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.557667  9618 raft_consensus.cc:3055] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:50.599334  1737 rpcz_store.cc:276] 0709 13:04:49.396932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.572063 (+1175131us) service_pool.cc:224] Handling call
0709 13:04:50.572615 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.572946 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.572973 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.573016 (+    43us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:50.573061 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.573144 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.573842 (+   698us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.562243  9827 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.590590  1735 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:04:50.601184  9869 raft_consensus.cc:3060] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:50.615700  9837 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:50.597488  9874 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26.
I20260709 13:04:50.588621  9875 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.598496  9868 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.637319  9762 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.587163  9840 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.582305  9660 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.586814  9828 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.634534  9593 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.679375  9813 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:50.563239  9835 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.580292  9601 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.639710  9618 raft_consensus.cc:740] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:50.698138  9882 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 13:04:50.698623  9079 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.690035  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3682) took 1210 ms. Trace:
I20260709 13:04:50.709580  1737 rpcz_store.cc:276] 0709 13:04:49.479328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.677020 (+1197692us) service_pool.cc:224] Handling call
0709 13:04:50.689298 (+ 12278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.689659 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.689698 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.689747 (+    49us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:50.689802 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.689859 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.689971 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.641333  9097 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.595790  9879 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:50.718883  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 163148us
I20260709 13:04:50.687570  9855 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:50.687260  9715 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:50.728632  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1387) took 1542 ms. Trace:
I20260709 13:04:50.705112  9762 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:50.730214  1733 rpcz_store.cc:276] 0709 13:04:49.186878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.456236 (+1269358us) service_pool.cc:224] Handling call
0709 13:04:50.456642 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:50.457160 (+   518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.457192 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:50.490959 (+ 33767us) log.cc:844] Serialized 26 byte log entry
0709 13:04:50.662064 (+171105us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:50.662131 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.662244 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:50.728388 (+ 66144us) raft_consensus.cc:1685] finished
0709 13:04:50.728416 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.728562 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":144856,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":12707,"thread_start_us":112817,"threads_started":1,"wal-append.queue_time_us":201592}
I20260709 13:04:50.689817  9883 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 13:04:50.732856  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 192935us
W20260709 13:04:50.734283  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 191422us
I20260709 13:04:50.735064  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:50.740031  1708 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.740454  1737 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 20
W20260709 13:04:50.729470  9857 consensus_meta.cc:287] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.839s	user 0.014s	sys 0.002s
I20260709 13:04:50.743002  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:50.689407  9840 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.729354  9760 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:50.748092  1733 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:50.747663  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:50.704916  9828 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.706907  9660 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:50.740131  9259 consensus_meta.cc:287] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.132s	user 0.016s	sys 0.000s
I20260709 13:04:50.756341  2312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:50.760803  2316 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:50.768862  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.769253  2316 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:50.748914  9857 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.764205  9835 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.765136  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.779471  1668 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 14.
I20260709 13:04:50.734769  9715 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.788558  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.774329  9760 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.790043  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:50.790721  1668 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:50.790772  9715 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.765892  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55270 (request call id 2914) took 1650 ms. Trace:
I20260709 13:04:50.791419  9660 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.697477  9875 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.768985  2312 raft_consensus.cc:2376] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 29. Re-sending same reply.
I20260709 13:04:50.791424  1736 rpcz_store.cc:276] 0709 13:04:49.115410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.583053 (+467643us) service_pool.cc:224] Handling call
0709 13:04:50.765646 (+1182593us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b54003516d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.765817 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":907768704}
I20260709 13:04:50.792812  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.793109  9760 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.793537  9875 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.795715  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.796545  1736 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:50.797309  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.798774  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.773483  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:50.778003  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:50.801970  1660 raft_consensus.cc:2368] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:50.802803  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:50.803332  1660 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:50.803077  9828 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.773483  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.712268  9868 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.805308  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:50.808295  9826 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 1.168s	user 0.010s	sys 0.003s
I20260709 13:04:50.808451  2317 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:50.808826  9826 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.800980  9601 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.784341  9803 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:50.807353  9868 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.790061  9879 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.811441  1556 leader_election.cc:400] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:50.829542  1556 leader_election.cc:403] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
W20260709 13:04:50.812119  9838 consensus_meta.cc:287] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.947s	user 0.009s	sys 0.002s
I20260709 13:04:50.812491  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.773061  9259 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.837221  9838 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.813912  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.814822  9616 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.757028  9618 consensus_queue.cc:260] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.734603  9855 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.816190  9840 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.818167  9857 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 28 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.819121  9812 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:50.820168  9885 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 13:04:50.820577  9813 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:50.858124  9618 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:50.799441  9097 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:50.821887  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45105.9us
I20260709 13:04:50.823110  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3683) took 1307 ms. Trace:
W20260709 13:04:50.830374  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 107399us
I20260709 13:04:50.864529  1736 rpcz_store.cc:276] 0709 13:04:49.521345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.817241 (+1295896us) service_pool.cc:224] Handling call
0709 13:04:50.817638 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.817864 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.817889 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.817934 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:50.818035 (+   101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.818096 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.823031 (+  4935us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.866425  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.830826  1660 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:50.778839  9842 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.826097  9803 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.832705  9879 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:50.833019  9877 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.835888  9850 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:04:50.842408  9838 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.841235  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.837900  1664 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:50.843430  1737 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:50.886206  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.886932  1737 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:50.839915  9259 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:50.839983  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:50.840049  9754 consensus_meta.cc:287] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Time spent flushing consensus metadata: real 0.520s	user 0.013s	sys 0.001s
W20260709 13:04:50.854564  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 114537us
W20260709 13:04:50.864986  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:50.869480  1556 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:50.873994  9803 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.837456  1668 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:50.881040  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:50.878044  1736 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:50.889556  1557 leader_election.cc:400] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 18 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:50.889396  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:50.893010  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:50.888880  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:50.900249  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:50.906719  1708 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.900871  2315 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 16. Preceding OpId from leader: term: 18 index: 17. (index mismatch)
W20260709 13:04:50.911392  1708 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.912395  1664 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 28: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:50.912436  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.912956  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.913192  1660 raft_consensus.cc:2368] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
W20260709 13:04:50.901633  1634 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.913800  1733 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:50.917256  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.918273  1735 raft_consensus.cc:2410] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 13:04:50.902014  9877 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:50.921224  9877 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:50.922062  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:50.922220  9877 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.923028  1740 raft_consensus.cc:2393] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 28: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:50.925702  9713 raft_consensus.cc:3055] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:50.926651  9713 raft_consensus.cc:740] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:50.927099  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:50.928005  9842 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.928333  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:50.927904  9713 consensus_queue.cc:260] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.926132  9656 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:50.930773  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:50.928990  1735 raft_consensus.cc:2368] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23.
W20260709 13:04:50.914150  1634 leader_election.cc:336] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:50.933233  1708 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.925706  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:50.940325  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:50.941823  9593 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:50.942395  2317 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:50.946317  2310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58302 (request call id 1172) took 2493 ms (client timeout 2116 ms). Trace:
I20260709 13:04:50.952806  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3670) took 2485 ms. Trace:
I20260709 13:04:50.941376  9713 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:50.953284  1738 rpcz_store.cc:276] 0709 13:04:48.467472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.467667 (+   195us) service_pool.cc:224] Handling call
0709 13:04:48.518171 (+ 50504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:48.788118 (+269947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.952505 (+2164387us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b540039dad4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:50.952734 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":600124160}
I20260709 13:04:50.954325  9869 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:50.955127  1738 raft_consensus.cc:1240] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:50.956305  1739 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:50.956704  9869 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:50.957785  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2895) took 2374 ms. Trace:
I20260709 13:04:50.958058  9770 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:50.958214  1739 rpcz_store.cc:276] 0709 13:04:48.583068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.956318 (+373250us) service_pool.cc:224] Handling call
0709 13:04:50.956092 (+1999774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.956697 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.957541 (+   844us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b540039dad0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.957724 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4560035584}
I20260709 13:04:50.960115  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3685) took 1397 ms. Trace:
I20260709 13:04:50.937038  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:50.955798  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3684) took 1393 ms. Trace:
I20260709 13:04:50.957772  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:50.962926  1708 leader_election.cc:336] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.947664  1706 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:50.930094  9794 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:50.947953  9879 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:50.948799  1556 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:50.904415  9844 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:50.949556  9838 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:50.946431  2314 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1175) took 1531 ms. Trace:
I20260709 13:04:50.938817  1634 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:50.962064  1738 rpcz_store.cc:276] 0709 13:04:49.562098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.954544 (+1392446us) service_pool.cc:224] Handling call
0709 13:04:50.954918 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.955536 (+   618us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.955739 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:50.950379  2310 rpcz_store.cc:269] 0709 13:04:48.452746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.452979 (+   233us) service_pool.cc:224] Handling call
0709 13:04:50.943805 (+2490826us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400746154. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:50.946207 (+  2402us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2487758720}
I20260709 13:04:50.960451  1739 rpcz_store.cc:276] 0709 13:04:49.562560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.959259 (+1396699us) service_pool.cc:224] Handling call
0709 13:04:50.959600 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.959796 (+   196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.959823 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.959866 (+    43us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:50.959906 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.959958 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.960062 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.986459  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3695) took 1157 ms. Trace:
I20260709 13:04:50.987057  1739 rpcz_store.cc:276] 0709 13:04:49.829406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.985128 (+1155722us) service_pool.cc:224] Handling call
0709 13:04:50.985675 (+   547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.985953 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.985993 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.986055 (+    62us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:50.986116 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.986193 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.986361 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:50.961450  1661 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:50.965047  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:50.992149  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:50.949872  9656 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:50.993150  9656 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:50.965092  1706 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:50.993855  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.994728  9883 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:50.995172  9883 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:50.995604  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:50.960937  9850 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 18. Current term is 19.
I20260709 13:04:50.996387  9616 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:50.995762  9883 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:50.994719  9651 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 14 FOLLOWER]: Advancing to term 15
I20260709 13:04:51.001168  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.002303  9850 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 13:04:51.002800  1557 leader_election.cc:400] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23.
I20260709 13:04:51.003265  1557 leader_election.cc:403] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 13:04:51.003945  9850 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:50.996454  9871 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:50.974380  1708 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:51.014590  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51705.6us
I20260709 13:04:51.015763  9593 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:51.017781  9850 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23.
I20260709 13:04:50.977926  2314 rpcz_store.cc:276] 0709 13:04:49.408347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.408524 (+   177us) service_pool.cc:224] Handling call
0709 13:04:49.459213 (+ 50689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.003108 (+543895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.003140 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:50.003621 (+   481us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:50.003700 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:50.939183 (+935483us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b54000c49d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:50.939420 (+   237us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:50.939645 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1250054912}
I20260709 13:04:51.024919  1737 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:04:51.025784  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3689) took 1422 ms. Trace:
W20260709 13:04:51.025350  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 111196us
I20260709 13:04:51.026224  1737 rpcz_store.cc:276] 0709 13:04:49.603327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.967229 (+1363902us) service_pool.cc:224] Handling call
0709 13:04:51.024441 (+ 57212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.025523 (+  1082us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.025729 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:51.026818  1634 leader_election.cc:336] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.027276  1634 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.029114  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.029052  1705 leader_election.cc:400] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:51.031136  1705 leader_election.cc:403] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:50.972707  9770 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:50.974004  9793 raft_consensus.cc:2804] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Leader election won for term 29
W20260709 13:04:51.032936  1634 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.972848  9794 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:50.973163  9879 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:50.977223  9869 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:50.973381  9855 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.038869  1741 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1388) took 1784 ms. Trace:
I20260709 13:04:51.029919  9615 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.029656  9598 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:51.046682  1634 leader_election.cc:336] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:50.972420  9768 consensus_queue.cc:1048] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 13:04:51.048338  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3698) took 1126 ms. Trace:
I20260709 13:04:51.044500  9879 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.053369  1634 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:51.053709  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 77659us
I20260709 13:04:51.072018  9794 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.072732  9766 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.073320  9794 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.053083  9793 raft_consensus.cc:697] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:51.065565  9762 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489) with higher term. Message: Invalid argument: T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20.
I20260709 13:04:51.052973  9869 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:51.069490  9813 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:51.077476  9793 consensus_queue.cc:237] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.079736  2318 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58332 (request call id 777) took 1265 ms. Trace:
I20260709 13:04:51.080343  2318 rpcz_store.cc:276] 0709 13:04:49.814509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.821950 (+  7441us) service_pool.cc:224] Handling call
0709 13:04:51.079300 (+1257350us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x7b54006004d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:51.079649 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162297216}
I20260709 13:04:51.047291  9850 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.081933  9850 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.076658  9656 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:51.078198  9897 log.cc:927] Time spent T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 13:04:51.090485  1733 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:51.091214  1733 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2922) took 1108 ms. Trace:
I20260709 13:04:51.078976  9813 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.065086  1737 rpcz_store.cc:276] 0709 13:04:49.922363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:51.034590 (+1112227us) service_pool.cc:224] Handling call
0709 13:04:51.047759 (+ 13169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.047987 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:51.048014 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:51.048065 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 13:04:51.048110 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.048168 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.048276 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 13:04:51.094096  1733 rpcz_store.cc:276] 0709 13:04:49.983166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:51.089883 (+1106717us) service_pool.cc:224] Handling call
0709 13:04:51.090260 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.090944 (+   684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.091143 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 13:04:51.094085  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47440.8us
W20260709 13:04:51.097182  1634 leader_election.cc:336] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.097960  1634 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:51.069022  9764 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.101307  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.096213  9872 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:51.114745  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.114902  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.115823  1733 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:04:51.116848  1634 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.119180  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.122399  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.098042  9850 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.118238  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:51.126371  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.100967  9826 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:51.128468  1708 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.129115  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.129412  1705 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.136251  1666 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:51.113694  9598 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:51.136068  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.142643  1661 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:51.144042  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.146680  1665 raft_consensus.cc:2376] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 29. Re-sending same reply.
W20260709 13:04:51.146929  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.109748  9764 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.147866  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.149086  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.149873  9764 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:51.151538  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.152051  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.154721  9813 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.155772  9762 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:51.159511  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:51.161013  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:51.144712  1634 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.078994  9869 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.165158  1733 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 27.
I20260709 13:04:51.136425  9615 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:51.167303  1708 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.168537  1739 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3697) took 1305 ms. Trace:
I20260709 13:04:51.168797  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:51.176079  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46759.2us
I20260709 13:04:51.174942  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.175627  1739 rpcz_store.cc:276] 0709 13:04:49.863622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.988372 (+1124750us) service_pool.cc:224] Handling call
0709 13:04:50.988780 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.989002 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 13:04:50.989372 (+   370us) log.cc:844] Serialized 14 byte log entry
0709 13:04:51.143330 (+153958us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:51.165562 (+ 22232us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:51.165615 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.165696 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.168456 (+  2760us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23368,"prepare.queue_time_us":197,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":23892,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":18221,"thread_start_us":94142,"threads_started":1}
I20260709 13:04:51.167723  1666 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:51.167763  1634 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:51.185367  9901 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 13:04:51.189276  1556 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:51.192247  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.193485  1661 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:51.195017  2311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58316 (request call id 1340) took 3063 ms (client timeout 1656 ms). Trace:
W20260709 13:04:51.195487  2311 rpcz_store.cc:269] 0709 13:04:48.131759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.133623 (+  1864us) service_pool.cc:224] Handling call
0709 13:04:51.194781 (+3061158us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54007f0654. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:51.194939 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1405075200}
W20260709 13:04:51.200160  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.174783  9764 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:51.217893  9769 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:51.196233  9610 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.228020  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.232728  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.233804  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.237767  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.238823  1665 raft_consensus.cc:2410] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 27 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
W20260709 13:04:51.239455  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72183.3us
I20260709 13:04:51.206774  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.206646  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:51.241037  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:51.241254  1666 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:51.242321  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.242908  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.248351  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.249549  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.250214  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.251971  1661 raft_consensus.cc:2393] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:51.252417  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.197309  9593 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:51.198021  9759 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:51.197117  9757 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:51.246932  9610 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.265914  9769 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:51.245553  1557 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:51.196691  9890 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:51.266541  1666 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 13:04:51.248019  9862 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:51.280679  1742 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2897) took 2697 ms. Trace:
I20260709 13:04:51.288087  1666 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:51.275277  9759 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:51.307654  9759 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.314837  9651 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 13:04:51.285199  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.315688  2315 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58332 (request call id 773) took 1558 ms. Trace:
I20260709 13:04:51.316107  2315 rpcz_store.cc:276] 0709 13:04:49.757674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.757867 (+   193us) service_pool.cc:224] Handling call
0709 13:04:49.758263 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.911227 (+1152964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.315406 (+404179us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54004e61d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:51.315623 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2626553216}
I20260709 13:04:51.317719  9890 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.320220  9890 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.288108  1742 rpcz_store.cc:276] 0709 13:04:48.583535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:48.969377 (+385842us) service_pool.cc:224] Handling call
0709 13:04:49.742148 (+772771us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:49.744743 (+  2595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.280447 (+1535704us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54003534d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:51.280611 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501782400}
I20260709 13:04:51.284348  9850 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:04:51.323370  1659 raft_consensus.cc:2410] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 24 because replica has last-logged OpId of term: 23 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 13:04:51.325955  1665 raft_consensus.cc:1217] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Deduplicated request from leader. Original: 16.10->[19.11-20.12]   Dedup: 19.11->[20.12-20.12]
I20260709 13:04:51.331234  1665 raft_consensus.cc:1240] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [19.11-20.12]
W20260709 13:04:51.283111  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.328244  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:51.337646  2318 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:51.332780  2315 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:51.333588  1666 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 13:04:51.260532  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.338498  1708 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.339123  2311 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:51.339295  9890 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:04:51.340631  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80166.1us
I20260709 13:04:51.341773  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:51.343123  1708 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.343696  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.343823  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.344602  9890 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:51.343822  1666 raft_consensus.cc:1275] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
W20260709 13:04:51.346063  1708 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.346763  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:51.349488  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.350169  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.351058  1659 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:51.353318  1660 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:51.368381  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:51.373970  1708 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.378602  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.379470  1742 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:51.381357  2314 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 13:04:51.357432  1668 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 20. Current term is 21. Ops: []
W20260709 13:04:51.386555  1708 leader_election.cc:336] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.387053  1708 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:51.375280  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46836.7us
W20260709 13:04:51.328447  1632 connection.cc:256] RPC call timeout handler was delayed by -1.0128023519999942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20260709 13:04:51.278949  9610 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.370054  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
W20260709 13:04:51.401289  1708 leader_election.cc:336] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.403039  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.366948  9770 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:51.404029  1634 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.409245  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.410878  1662 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3515) took 1799 ms. Trace:
I20260709 13:04:51.411406  1662 rpcz_store.cc:276] 0709 13:04:49.611273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.614015 (+  2742us) service_pool.cc:224] Handling call
0709 13:04:51.409359 (+1795344us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b5800135c98. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb8a22 00007fcdf3dd1c09 00007fcdf2f2bee0 00007fcdf2ff50ab 00007fcdf2ff5131 00007fcdf2fb6fc0 00007fcdf2fa1929 00007fcdf2fa13d2 00007fcdec518bd4
0709 13:04:51.410053 (+   694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.410453 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:51.410498 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:51.410556 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:51.410601 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.410665 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.410781 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7098752}
I20260709 13:04:51.414988  2309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58302 (request call id 1180) took 1497 ms. Trace:
I20260709 13:04:51.378636  9572 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.375342  9845 raft_consensus.cc:3055] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
W20260709 13:04:51.420315  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 52038.7us
I20260709 13:04:51.440124  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.436430  1632 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:51.415405  2309 rpcz_store.cc:276] 0709 13:04:49.917980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:49.918264 (+   284us) service_pool.cc:224] Handling call
0709 13:04:51.088900 (+1170636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.089122 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:51.089149 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:51.089198 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0709 13:04:51.089242 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.089307 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.414750 (+325443us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54006004d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:51.414906 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2679615744}
I20260709 13:04:51.445420  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.448740  1733 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:51.350087  9850 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:51.367758  9651 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.415973  9753 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:51.435530  9877 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:51.435391  9893 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:51.367170  9769 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 13:04:51.442574  9826 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:51.415475  9259 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.465257  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.458251  2318 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:04:51.442063  9845 raft_consensus.cc:740] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:51.475270  9877 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:51.490080  9877 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.490386  9845 consensus_queue.cc:260] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.491671  9845 raft_consensus.cc:3060] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:51.452533  9833 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:04:51.493559  9833 raft_consensus.cc:697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:51.494619  9833 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.497786  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.472549  9753 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:51.499019  9753 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.499099  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.500017  1742 raft_consensus.cc:2393] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 24: Already voted for candidate 316f3bf18ed94e32b238130a6bfedf1a in this term.
W20260709 13:04:51.500330  9904 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 13:04:51.500837  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.392743  9793 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.456723  9842 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:51.433549  1634 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.506534  1735 raft_consensus.cc:1240] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 13:04:51.506798  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 73325.4us
I20260709 13:04:51.447368  9572 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.467154  9616 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.487426  1737 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:51.404353  9838 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:51.459084  9869 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 13:04:51.407900  9593 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:51.503007  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:51.507519  1634 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.511519  1738 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3692) took 1868 ms. Trace:
W20260709 13:04:51.521775  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 156669us
I20260709 13:04:51.517961  1738 rpcz_store.cc:276] 0709 13:04:49.643088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.980139 (+1337051us) service_pool.cc:224] Handling call
0709 13:04:50.980879 (+   740us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:50.981905 (+  1026us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:50.981945 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:50.982707 (+   762us) log.cc:844] Serialized 26 byte log entry
0709 13:04:51.169462 (+186755us) raft_consensus.cc:1645] Marking committed up to 15
0709 13:04:51.169974 (+   512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.170118 (+   144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:51.499673 (+329555us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:51.505755 (+  6082us) raft_consensus.cc:1685] finished
0709 13:04:51.505794 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.511436 (+  5642us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12917,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":102,"spinlock_wait_cycles":140188800,"thread_start_us":138426,"threads_started":1,"wal-append.queue_time_us":158852}
W20260709 13:04:51.518430  1634 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.521775  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.530368  1742 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 21: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:51.535807  1634 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.539464  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:51.547590  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.488222  9651 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.540715  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.505116  9259 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:51.562250  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:51.553330  9844 consensus_meta.cc:287] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.571s	user 0.010s	sys 0.004s
I20260709 13:04:51.563427  1666 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 13:04:51.571559  1742 raft_consensus.cc:1275] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:04:51.514242  9616 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.578233  1667 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 13:04:51.578614  9908 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 13:04:51.518124  9838 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.579241  1666 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:51.580657  1742 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:51.563869  9842 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:51.585788  9903 log.cc:927] Time spent T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.309s	user 0.003s	sys 0.000s
I20260709 13:04:51.048586  1741 rpcz_store.cc:276] 0709 13:04:49.254237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.470394 (+1216157us) service_pool.cc:224] Handling call
0709 13:04:50.470763 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:50.471015 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 13:04:50.871289 (+400274us) log.cc:844] Serialized 14 byte log entry
0709 13:04:51.038341 (+167052us) log.cc:844] Serialized 14 byte log entry
0709 13:04:51.038509 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 13:04:51.038557 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:51.038612 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.038685 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:51.038808 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2335,"prepare.queue_time_us":429185,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":14352,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":18774,"spinlock_wait_cycles":8283904,"thread_start_us":565698,"threads_started":2}
I20260709 13:04:51.603773  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.605211  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:51.549582  1634 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:51.607829  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58310.7us
I20260709 13:04:51.610134  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
W20260709 13:04:51.609232  1634 leader_election.cc:336] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.612965  1737 raft_consensus.cc:1275] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
I20260709 13:04:51.614347  1739 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: []
W20260709 13:04:51.611871  1634 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.615547  2314 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:51.616429  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:51.560266  9879 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.621338  1741 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:04:51.623829  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.624728  2309 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:51.548986  9850 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.626322  1737 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27. Ops: []
I20260709 13:04:51.605751  9869 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.630127  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.630951  2314 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 13:04:51.631158  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.635813  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:51.636936  1740 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: []
I20260709 13:04:51.638495  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.618858  9872 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.618783  9610 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:51.606267  9842 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.645280  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:51.648383  9862 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:51.653375  9713 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:51.654080  9862 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:51.655050  9862 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:51.655427  9862 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:51.648628  9872 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.664204  9872 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.665025  9862 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.672919  9713 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:51.674031  9713 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.674997  9838 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.675686  9838 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.676177  9838 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.676625  9713 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.645943  9869 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.679006  9879 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.679484  9869 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.680392  9862 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.682786  9877 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:51.687433  9883 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.690s	user 0.015s	sys 0.000s
I20260709 13:04:51.688123  9883 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.687299  9616 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.689182  9259 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.689949  9616 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.690344  1738 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 22: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:51.690446  9616 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.692814  9616 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.693656  9862 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.694352  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.694418  9572 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.694504  9862 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.696609  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.697078  9862 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.698328  9862 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.702636  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:51.703666  2311 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:51.703948  9656 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.709149  9838 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.700620  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.714993  9826 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.715626  9826 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:51.719164  9909 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.230s	user 0.001s	sys 0.000s
I20260709 13:04:51.723654  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.697136  9842 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.725303  1737 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:51.722831  9259 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.726446  9259 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.728912  9259 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.729091  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.729760  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.757648  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.730525  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.759125  1741 raft_consensus.cc:1813] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Handling vote request from an unknown peer 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:51.759624  1741 raft_consensus.cc:2368] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21.
W20260709 13:04:51.731734  9915 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 13:04:51.732121  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.744176  9753 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.761122  2318 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:51.732704  9651 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.739878  9826 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.764797  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.747507  9890 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:51.747989  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.766664  1738 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:51.750976  9883 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:51.775839  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.777839  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.783687  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:51.785663  9877 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:51.787146  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.790251  9787 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 13:04:51.788455  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.787604  9845 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.792132  9845 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.796922  9845 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.799722  1741 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:04:51.803045  1667 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2832) took 1038 ms. Trace:
I20260709 13:04:51.803560  1667 rpcz_store.cc:276] 0709 13:04:50.764626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.767854 (+  3228us) service_pool.cc:224] Handling call
0709 13:04:50.784136 (+ 16282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.578647 (+794511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.802766 (+224119us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b5400260f54. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:51.802956 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1825155328}
I20260709 13:04:51.772122  2311 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
I20260709 13:04:51.807691  1557 leader_election.cc:400] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21.
I20260709 13:04:51.808224  9850 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:51.810580  1735 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2921) took 1827 ms. Trace:
I20260709 13:04:51.811064  1735 rpcz_store.cc:276] 0709 13:04:49.982833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:51.047538 (+1064705us) service_pool.cc:224] Handling call
0709 13:04:51.506137 (+458599us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:51.507092 (+   955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.810351 (+303259us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54003520d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:51.810521 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1010774912}
I20260709 13:04:51.811460  9869 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:51.811710  9850 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:04:51.787163  1707 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:51.818531  9766 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:51.818594  1739 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:04:51.819352  9766 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.820103  9764 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.810591  1557 leader_election.cc:403] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:51.820720  9764 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.821177  9616 raft_consensus.cc:3060] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.824204  9764 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:51.827293  1665 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2840) took 1041 ms. Trace:
I20260709 13:04:51.827900  9754 consensus_queue.cc:1048] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:51.828385  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:51.828692  9754 consensus_queue.cc:1243] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:51.829226  2315 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:51.827759  9572 consensus_queue.cc:1048] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.003s
I20260709 13:04:51.827750  1665 rpcz_store.cc:276] 0709 13:04:50.786178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:51.258587 (+472409us) service_pool.cc:224] Handling call
0709 13:04:51.259134 (+   547us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:51.337001 (+ 77867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.826968 (+489967us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b54002e0f20. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf54dc0 00007fcdedfe0a02 00007fcdedfdcdd6 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:51.827223 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25182080}
I20260709 13:04:51.835786  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 17 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.838294  9842 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.838950  9842 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:51.840487  9767 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 13:04:51.842739  9766 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.792980  9787 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 13:04:51.809713  9651 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:51.844203  9787 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:51.813100  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.844806  9754 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:51.847862  1665 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 13:04:51.852948  1665 pending_rounds.cc:85] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:04:51.853400  1665 pending_rounds.cc:107] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 13:04:51.853569  9842 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:51.853919  1665 raft_consensus.cc:2889] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:51.855012  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:51.855549  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 430773us
I20260709 13:04:51.844624  9787 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
W20260709 13:04:51.856158  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.856608  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:51.857877  1634 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:51.858508  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50480.2us
I20260709 13:04:51.859233  1632 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:51.859318  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.860502  9259 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 13:04:51.860617  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 416639us
I20260709 13:04:51.861755  9616 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737) with higher term. Message: Invalid argument: T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21.
I20260709 13:04:51.862244  9767 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:04:51.862751  9767 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting leader election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:51.863140  9767 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.866030  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3686) took 2303 ms. Trace:
I20260709 13:04:51.866528  1740 rpcz_store.cc:276] 0709 13:04:49.562895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:50.963338 (+1400443us) service_pool.cc:224] Handling call
0709 13:04:51.027333 (+ 63995us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:51.637360 (+610027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:51.865774 (+228414us) spinlock_profiling.cc:243] Waited 609 ms on lock 0x7b54000df2d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:51.865964 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1401361664}
I20260709 13:04:51.867311  2315 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 13:04:51.869596  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:51.870580  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:51.871347  1708 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:51.872340  1708 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 131 similar messages]
W20260709 13:04:51.872507  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 64948.7us
I20260709 13:04:51.873085  9890 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:51.873690  9766 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:51.892920  9766 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.894325  9877 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:51.896574  9766 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:51.857074  9787 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:51.931773  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.879225  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:51.880045  9764 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 13:04:51.884822  9259 raft_consensus.cc:697] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:51.885766  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.886950  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:51.890033  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.889861  9813 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:04:51.875073  2311 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:51.897310  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.898576  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.967677  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.968618  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.898936  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.969409  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.967707  1660 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:51.971033  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81065.2us
W20260709 13:04:51.859808  1634 leader_election.cc:336] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.854923  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:51.893393  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.972052  1556 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.972633  2309 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:51.972206  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 112432us
W20260709 13:04:51.974179  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.905185  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:51.866071  1705 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.975044  2318 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.972188  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:51.976624  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 83255.9us
W20260709 13:04:51.977427  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.977819  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.977952  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.981178  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:51.907282  2283 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.910157  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:51.984848  1737 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:51.913144  1667 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:51.919341  1738 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:52.012171  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:51.960744  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.956466  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:51.962267  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 83183.4us
I20260709 13:04:52.014364  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:04:52.014523  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 58188.3us
I20260709 13:04:52.015605  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:51.955583  1483 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 23, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 23 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.017186  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:52.017702  9872 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:52.018132  9872 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:52.018435  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:52.018703  9872 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:52.019201  1663 raft_consensus.cc:2393] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:52.020640  1663 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:04:52.022245  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:52.023178  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:51.965548  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:52.024394  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.024930  1659 raft_consensus.cc:1275] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 13:04:51.901525  9753 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:51.990468  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 124462us
W20260709 13:04:51.990751  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:52.033752  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:51.991938  1556 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:51.987515  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80252.3us
I20260709 13:04:52.011097  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:52.009841  1634 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.026503  2309 raft_consensus.cc:2468] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:52.030892  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:51.968739  9826 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:52.035713  1634 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.013615  2318 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:52.037364  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:52.013111  1660 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 20.
W20260709 13:04:52.038698  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46888.3us
I20260709 13:04:52.054224  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.988824  9877 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:52.051236  1634 leader_election.cc:336] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.034332  1708 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:52.060173  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:52.017366  1483 catalog_manager.cc:5697] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 22. New cstate: current_term: 22 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.048527  1668 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:51.976187  9845 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:51.997983  9756 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:52.052482  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.071060  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:52.073041  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:52.073763  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.074232  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:51.962786  9616 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 13:04:52.010462  9890 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:52.023486  9883 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:52.030267  9838 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.030006  9651 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:52.036368  9844 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:52.089958  1483 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 11 to 13, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 13 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:52.097615  1483 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:51.998656  9869 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 13:04:52.056174  9656 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:04:52.101832  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 133) took 29715 ms (client timeout 15000 ms). Trace:
I20260709 13:04:52.100567  1483 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 16 to 17, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.048898  9713 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:52.022651  9842 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 13:04:52.121706  1483 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 24, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 24 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.143409  1483 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 18, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 18 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:04:52.087570  9787 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:52.115052  1486 rpcz_store.cc:269] 0709 13:04:22.473231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:25.761742 (+3288511us) service_pool.cc:224] Handling call
0709 13:04:52.088206 (+26326464us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:48.120354 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:48.121310 (+   956us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:48.121384 (+    74us) write_op.cc:435] Acquired schema lock
0709 13:04:48.121413 (+    29us) tablet.cc:662] Decoding operations
0709 13:04:48.371330 (+249917us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:48.391835 (+ 20505us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:48.394235 (+  2400us) tablet.cc:685] Acquiring locks for 17 operations
0709 13:04:48.431096 (+ 36861us) tablet.cc:701] Row locks acquired
0709 13:04:48.431165 (+    69us) write_op.cc:260] PREPARE: finished
0709 13:04:48.431406 (+   241us) write_op.cc:270] Start()
0709 13:04:48.431621 (+   215us) write_op.cc:276] Timestamp: P: 1783602288431324 usec, L: 0
0709 13:04:48.431670 (+    49us) op_driver.cc:348] REPLICATION: starting
0709 13:04:48.432716 (+  1046us) log.cc:844] Serialized 6627 byte log entry
0709 13:04:49.382364 (+949648us) op_driver.cc:464] REPLICATION: finished
0709 13:04:49.806027 (+423663us) write_op.cc:301] APPLY: starting
0709 13:04:49.823600 (+ 17573us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:50.820172 (+996572us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:50.820297 (+   125us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:51.398085 (+577788us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:51.508054 (+109969us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0709 13:04:51.508136 (+    82us) write_op.cc:312] APPLY: finished
0709 13:04:51.606187 (+ 98051us) log.cc:844] Serialized 156 byte log entry
0709 13:04:51.782164 (+175977us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.782441 (+   277us) write_op.cc:454] Released schema lock
0709 13:04:51.782549 (+   108us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:51.922300 (+139751us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:51.931126 (+  8826us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4747374,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":340705,"apply.run_cpu_time_us":85949,"apply.run_wall_time_us":2031039,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75221,"num_ops":17,"prepare.queue_time_us":259622,"prepare.run_cpu_time_us":14837,"prepare.run_wall_time_us":413344,"raft.queue_time_us":466189,"raft.run_cpu_time_us":90348,"raft.run_wall_time_us":329944,"replication_time_us":922609,"thread_start_us":793154,"threads_started":5,"wal-append.queue_time_us":371596,"wal-append.run_cpu_time_us":28383,"wal-append.run_wall_time_us":1490657}]]}
W20260709 13:04:52.159351  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94431.1us
I20260709 13:04:52.161942  1483 catalog_manager.cc:5697] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 12 to 18, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 18 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:52.163991  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 121166us
I20260709 13:04:52.165817  1483 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.081063  1737 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
W20260709 13:04:52.176326  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 187207us
I20260709 13:04:52.187415  1483 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 19, leader changed from <none> to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 19 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:04:52.117954  1706 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:52.201972  1483 catalog_manager.cc:5697] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:04:52.211663  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 157509us
W20260709 13:04:52.212407  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 94556.9us
I20260709 13:04:52.120013  9879 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 13:04:52.214157  1705 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.221822  9769 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.222951  9890 consensus_queue.cc:1048] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:04:52.224403  9813 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:52.228075  1740 raft_consensus.cc:1240] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:52.229621  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:52.230337  1737 raft_consensus.cc:2468] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 18.
I20260709 13:04:52.224685  9890 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.231626  9890 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.114580  9877 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.101s
I20260709 13:04:52.197165  9842 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:52.221907  9934 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:52.204847  9845 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.244939  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:52.249897  9713 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:52.218845  9756 raft_consensus.cc:2804] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:52.244833  9933 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:52.253328  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 13:04:52.254927  1735 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:04:52.141098  9883 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.104s
I20260709 13:04:52.266060  9756 raft_consensus.cc:697] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:52.268263  1556 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.106405  9838 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.279503  1663 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 13:04:52.279330  1557 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:52.286525  9890 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:52.288165  9850 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.288978  9850 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:52.289714  9850 consensus_queue.cc:1243] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 13:04:52.293999  1556 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.297021  9842 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:52.301491  9756 consensus_queue.cc:237] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:52.283053  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:52.327181  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:52.327386  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44463.1us
W20260709 13:04:52.329118  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 295366us
W20260709 13:04:52.369931  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 344598us
I20260709 13:04:52.381026  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:52.382277  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 308492us
W20260709 13:04:52.383306  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.383709  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.253980  9877 raft_consensus.cc:3055] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:52.388528  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 61436.4us
W20260709 13:04:52.311312  9927 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 13:04:52.280690  9759 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.118774  9259 consensus_queue.cc:237] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.411479  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.412094  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.410637  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:52.437572  1662 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:52.327353  9936 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.448823  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.339948  9651 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:52.414070  9616 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:52.414695  9656 raft_consensus.cc:2804] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:52.389012  9877 raft_consensus.cc:740] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:52.476614  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 93330.4us
W20260709 13:04:52.487457  2283 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.307196  9767 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:52.380414  9713 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.085s
W20260709 13:04:52.473433  9928 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 13:04:52.388375  9610 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 13:04:52.409757  9845 consensus_queue.cc:1059] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 13:04:52.491925  9753 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:52.486711  9656 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:52.464063  9862 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 13:04:52.472453  9842 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.495210  2281 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:52.492352  9651 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:52.490154  9869 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:52.541697  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129721us
I20260709 13:04:52.552866  9943 consensus_queue.cc:1048] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
W20260709 13:04:52.563208  9872 consensus_meta.cc:287] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.544s	user 0.015s	sys 0.000s
W20260709 13:04:52.572396   431 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 1618)
I20260709 13:04:52.589967  9890 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Stepping down as leader of term 19
I20260709 13:04:52.600760  9890 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:52.601778  9890 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:52.603384  9877 consensus_queue.cc:260] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:52.543900  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 130411us
I20260709 13:04:52.605065  9767 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.566147  9842 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.606298  9838 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.465413  9936 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.594223  9872 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.443222  9759 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:52.610551  9759 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:52.566834  9879 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:52.606495  1632 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.607460  9618 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 13:04:52.612429  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.613003  9618 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:52.613763  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.613692  9618 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:52.614816  9618 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:52.611722  9872 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.615164  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.558122  9656 raft_consensus.cc:3060] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
W20260709 13:04:52.543116  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 94396.3us
I20260709 13:04:52.651911  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:52.571393  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 181957us
W20260709 13:04:52.610654  9926 log.cc:927] Time spent T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 13:04:52.608435  9842 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.605398  9877 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:52.684041  1556 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.682149  1632 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:52.693397  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 153491us
W20260709 13:04:52.693570  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 198490us
I20260709 13:04:52.710808  1632 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.714963  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.610646  9936 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.719363  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.675501  9826 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:52.724203  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:52.730144  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:52.725320  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:52.734038  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:52.708770  9946 log.cc:927] Time spent T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 13:04:52.727415  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:52.740654  1733 raft_consensus.cc:2468] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
W20260709 13:04:52.726450  1634 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.737263  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:52.728464  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:52.730877  1735 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:52.742041  1742 raft_consensus.cc:2468] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
W20260709 13:04:52.744112  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 255682us
I20260709 13:04:52.706947  9932 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:52.747494  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:52.696211  9872 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:52.716426  9833 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:52.730453  9618 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:52.724013  9842 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:52.742802  1634 leader_election.cc:336] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:52.777329  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80158.4us
I20260709 13:04:52.778105  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:52.779187  1660 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 21: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:52.739491  9969 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:52.737658  9936 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:52.706890  9713 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 13:04:52.736915  9764 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:52.752409  9933 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.737874  9938 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:52.776829  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:52.754729  9759 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election won for term 20
I20260709 13:04:52.791054  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:52.769017  9838 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:52.769177  9862 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.769429  9963 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:04:52.796729  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.809440  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:52.798224  1634 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:52.796458  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:52.808405  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:52.799599  9941 log.cc:927] Time spent T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
I20260709 13:04:52.769224  9651 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 13:04:52.784986  9932 raft_consensus.cc:697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:52.766521  9259 consensus_queue.cc:1048] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
W20260709 13:04:52.813866  9939 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.356s	user 0.002s	sys 0.000s
I20260709 13:04:52.808056  9850 raft_consensus.cc:2804] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:52.799912  9969 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:52.837476  9917 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.221s	user 0.002s	sys 0.000s
I20260709 13:04:52.817062  9963 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:52.868558  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 129668us
I20260709 13:04:52.814169  9759 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:52.871394  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74694.4us
I20260709 13:04:52.839339  9842 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:52.816241  9862 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.879035  1632 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:52.807847  9872 consensus_queue.cc:1059] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 13:04:52.858364  9838 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:52.795037  9958 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.091s
I20260709 13:04:52.798838  9764 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:52.867904  9850 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:52.886996  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:52.872692  9756 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:52.884003  9945 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
I20260709 13:04:52.895952  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:52.881258  9845 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:52.880690  9963 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:52.905676  2311 raft_consensus.cc:2410] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 13:04:52.808691  9933 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:52.904356  9949 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 13:04:52.890048  9970 raft_consensus.cc:2749] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:52.891520  9766 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:52.892428  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 96088.8us
I20260709 13:04:52.900817  9850 raft_consensus.cc:3060] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:52.976397  1735 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:04:52.974968  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
W20260709 13:04:52.978139  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62847.9us
W20260709 13:04:52.980460  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 65444.5us
I20260709 13:04:52.909353  9845 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:52.987696  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 111101us
I20260709 13:04:52.992245  9971 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:52.994056  9259 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:52.995101  9862 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.997388  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:52.999194  9259 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:52.999809  9259 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:53.001253  9259 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:52.987255  9616 raft_consensus.cc:2804] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:53.006000  9616 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:53.006402  9616 raft_consensus.cc:3060] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:53.007342  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.008086  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.010929  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.015161  9764 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:52.978734  9766 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.016600  2310 raft_consensus.cc:1240] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 13:04:53.017832  9766 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.020637  9933 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.021255  9756 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.021781  9756 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.024403  9756 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:53.026283  9862 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:53.026712  9975 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 13:04:53.026973  9862 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:53.029021  9976 log.cc:927] Time spent T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 13:04:53.032063  9862 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:52.913925  9957 raft_consensus.cc:3055] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 13:04:52.974052  9930 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 13:04:52.887344  9842 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.020573  2310 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 13:04:53.057896  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.059613  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "882fdd7ea10640a78914e0647598ad9b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.060544  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.042869  9845 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:53.070047  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 262793us
I20260709 13:04:52.983521  9980 raft_consensus.cc:2764] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 13:04:53.070554  1735 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 13:04:53.075155  9974 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 13:04:52.988843  9932 consensus_queue.cc:237] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.084641  9842 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:53.086743  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 306032us
I20260709 13:04:53.045490  9957 raft_consensus.cc:740] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:53.097849  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:53.074252  2281 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:53.101794  1741 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 13:04:53.048022  9838 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:53.126916  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52712us
I20260709 13:04:53.130882  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.146379  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:53.136343  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.144807  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.159335  1741 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:53.159826  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.156414  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:53.168939  1661 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 15: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:52.988592  9969 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.202293  2310 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:53.145802  9962 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:53.205283  1708 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:53.158730  9766 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:04:53.160593  9933 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.072973  9759 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.191629  9973 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:53.230055  9756 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:53.230719  9756 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.233716  9968 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:53.235426  9879 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:53.257915  1708 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.258702  1708 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:53.259795  9968 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:53.261763  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.262375  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:53.263509  9968 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:04:53.265365  1708 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:53.267465  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.213126  9977 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.235869  9842 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.233457  9756 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:53.241837  9766 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:53.235971  9879 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.222430  9933 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:53.282367  1708 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.212534  9962 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.304838  1740 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2963) took 1061 ms. Trace:
I20260709 13:04:53.315614  9879 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.315402  9963 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.312651  1740 rpcz_store.cc:276] 0709 13:04:52.243390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:52.243667 (+   277us) service_pool.cc:224] Handling call
0709 13:04:52.284926 (+ 41259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:52.829625 (+544699us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:52.829660 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:52.859677 (+ 30017us) log.cc:844] Serialized 26 byte log entry
0709 13:04:53.049137 (+189460us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:53.049180 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:53.050414 (+  1234us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:53.304494 (+254080us) raft_consensus.cc:1685] finished
0709 13:04:53.304525 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:53.304728 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":158842,"raft.queue_time_us":229092,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":10966,"thread_start_us":548952,"threads_started":2,"wal-append.queue_time_us":215590}
W20260709 13:04:53.311570  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.320490  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.321379  2310 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:53.213129  9957 consensus_queue.cc:260] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.327752  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:53.289570  9977 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.335472  9977 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:53.290184  9842 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.338446  9610 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:53.350829  9936 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.393242  9759 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.394172  9759 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.369771  9610 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:53.400489  9957 raft_consensus.cc:3060] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:04:53.432713  1708 leader_election.cc:336] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:53.435751  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:53.438854  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:53.440814  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
W20260709 13:04:53.442096  9992 log.cc:927] Time spent T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 13:04:53.442502  9877 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.443109  9877 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:53.445858  9656 consensus_meta.cc:287] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.761s	user 0.013s	sys 0.000s
I20260709 13:04:53.405534  9766 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.446308  9656 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.400132  9936 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.448377  9656 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.448676  9936 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.436417  9979 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:53.451305  9979 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.452842  9962 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:53.459079  9936 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.459681  9936 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.461213  9842 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.453338  9979 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.462553  9877 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.448939  9616 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.432170  9610 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:53.430320  9879 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:53.471670  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:53.489290  9890 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:53.494369  9610 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.497444  9610 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.503923  9943 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 13:04:53.504607  9943 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:53.516672  9842 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.517467  9842 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.520057  9842 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.474447  1480 maintenance_manager.cc:419] P e6b16382849d482ba544fc5ee58e0b0a: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033355
I20260709 13:04:53.475066  9963 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:53.474767  9850 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.475629  9936 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.477941  9616 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.479750  9879 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.552059  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.554049  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:53.577991  2309 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:53.552551  2283 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:53.560878  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.569885  1735 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 13:04:53.575268  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:53.578632  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:53.581961  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.555739  9933 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.586103  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.586130  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.591747  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:53.592934  2309 raft_consensus.cc:2468] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
W20260709 13:04:53.584029  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:53.583405  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:53.597769  2283 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.572309  9977 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:53.600242   431 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 1618)
I20260709 13:04:53.600207  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.601925  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.594244  1706 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:53.573565  9883 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:53.608657  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:53.607195  1634 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.611898  1741 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:04:53.578815  9869 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.582703  9972 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.583362  9934 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:53.590466  9616 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.642732  9891 raft_consensus.cc:2804] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:53.649736  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.650061  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.609573  9933 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:53.647842  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:53.642246  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:53.654044  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:53.654897  1738 raft_consensus.cc:2410] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
W20260709 13:04:53.621292  9618 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.838s	user 0.015s	sys 0.000s
I20260709 13:04:53.650768  1740 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
W20260709 13:04:53.655738  1708 leader_election.cc:336] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.560164  9943 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:53.672196  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 115540us
I20260709 13:04:53.666499  9891 raft_consensus.cc:697] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:53.618239  9977 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.626718  9883 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.686867  2280 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:53.666972  9618 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:04:53.678390  9986 log.cc:927] Time spent T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.477s	user 0.003s	sys 0.003s
I20260709 13:04:53.643442  9869 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.700552  2280 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:53.704025  9891 consensus_queue.cc:237] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.680044 10015 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:53.706470 10015 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.708911 10015 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.674062 10013 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.653070  9616 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.646903  9972 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.694247  9943 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:53.709486 10015 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
W20260709 13:04:53.716590  2283 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:53.688673  9933 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.713987  9766 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:53.725175  9934 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 14 FOLLOWER]: Advancing to term 15
W20260709 13:04:53.751474 10002 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 13:04:53.779096  9971 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.779886  9971 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.786342  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.788053  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.793010  1661 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Rejecting Update request from peer df6a18af06d543b2931837f5e6bab95f for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 13:04:53.797675  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.800351  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.801571  1665 raft_consensus.cc:1275] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 13:04:53.802114  1665 pending_rounds.cc:85] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 14
I20260709 13:04:53.802383  1665 pending_rounds.cc:107] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:04:53.802613  1665 raft_consensus.cc:2889] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:53.805843  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 80508.3us
I20260709 13:04:53.810071  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.811998  9850 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.812925  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.725908 10013 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.822373  1662 raft_consensus.cc:1275] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 20 index: 9. Preceding OpId from leader: term: 22 index: 10. (index mismatch)
I20260709 13:04:53.758044  9933 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.835546  1659 raft_consensus.cc:1217] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Deduplicated request from leader. Original: 19.12->[21.13-25.15]   Dedup: 21.13->[23.14-25.15]
I20260709 13:04:53.836640  1659 pending_rounds.cc:85] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 13
I20260709 13:04:53.837218  1659 pending_rounds.cc:107] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 13:04:53.836660  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:53.838155  1659 raft_consensus.cc:2889] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:53.838510  1667 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:53.849114  1665 raft_consensus.cc:1240] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 25. Current term is 26. Ops: [25.13-25.13]
W20260709 13:04:53.850558  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 301945us
I20260709 13:04:53.852269  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9e456da7a864a1b97fb4c468e75980d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:53.818753  9879 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:53.816486  9833 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:53.858604  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.862207  1665 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:53.847635  9977 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:53.872532  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 65052.3us
I20260709 13:04:53.873802  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.874859  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:53.864995 10015 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:53.884202 10016 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 13:04:53.885447  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.887941  9972 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.869316 10022 raft_consensus.cc:493] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:53.880390 10009 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.167s	user 0.000s	sys 0.002s
I20260709 13:04:53.903755  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.901824  1667 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:04:53.907528  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.908617  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.908485  1666 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [19.11-23.13]
I20260709 13:04:53.909334  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.878400  9971 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.866480  9833 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.923547  9259 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:53.930164  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.933583  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:53.903558 10015 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:53.943342  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:53.926558  9616 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.943624 10015 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.947834  9934 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 13:04:53.966528  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:53.965726 10016 log.cc:927] Time spent T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 13:04:54.013671  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 70414.5us
I20260709 13:04:54.016130  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:53.935592 10023 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:53.970948 10008 log.cc:927] Time spent T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 13:04:54.023094  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:54.033982  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.034796  9616 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:54.009778  9977 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:53.934620  9259 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:53.973340  9838 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:04:54.072336  9845 consensus_meta.cc:287] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.983s	user 0.016s	sys 0.000s
W20260709 13:04:54.072810 10017 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
W20260709 13:04:54.127249  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 104158us
W20260709 13:04:54.019691 10011 log.cc:927] Time spent T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 13:04:54.102610  9933 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:53.911489 10022 raft_consensus.cc:515] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.139780  9977 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.141005 10013 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.142939  9259 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.143375  9838 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:54.143854 10006 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.324s	user 0.001s	sys 0.001s
I20260709 13:04:54.152343  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.163347  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:54.172273  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.184173  1735 raft_consensus.cc:2468] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 19.
I20260709 13:04:54.185729  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.186815  1735 raft_consensus.cc:2468] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:54.175452  1739 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:54.173584  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.190075  1739 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 13:04:54.190466  1739 pending_rounds.cc:85] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting all ops after (but not including) 12
I20260709 13:04:54.190694  1739 pending_rounds.cc:107] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 13:04:54.190930  1739 raft_consensus.cc:2889] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:54.178073  9616 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:54.128077  9833 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.131410 10026 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:54.194047  9616 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.205833  9616 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.211683  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.194414  9838 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:54.214370  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:54.215778  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.127401 10023 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.219177  1735 raft_consensus.cc:2410] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:54.214784  1662 raft_consensus.cc:2410] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 13:04:54.162700  9845 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.225100  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.213681 10024 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.227768 10024 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.238610  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.239408  1733 raft_consensus.cc:2468] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:54.222553 10014 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.241802  9890 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.242372  9890 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.219930 10026 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.252568  9963 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:54.253340 10005 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.635s	user 0.000s	sys 0.005s
I20260709 13:04:54.253259  9963 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.255306  1737 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55270 (request call id 2970) took 1533 ms. Trace:
I20260709 13:04:54.255788  9963 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.256425  9963 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.260078  9759 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.261739 10022 leader_election.cc:290] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.263757  9845 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.226243  9960 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.265286  9960 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.256862  9963 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.266214  1737 rpcz_store.cc:276] 0709 13:04:52.721564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:52.726315 (+  4751us) service_pool.cc:224] Handling call
0709 13:04:52.749028 (+ 22713us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:52.752841 (+  3813us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:52.752881 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:53.336122 (+583241us) spinlock_profiling.cc:243] Waited 9.82 ms on lock 0x7b18004304e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaebe2a6 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a
0709 13:04:53.336997 (+   875us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b2c00077ef8. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdec8d6a11 00007fcdec8e6372 00007fcdee018a01 00007fcdedfdd43a 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2
0709 13:04:53.337569 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 13:04:53.432283 (+ 94714us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:53.432338 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:53.432415 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:54.252658 (+820243us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:54.255038 (+  2380us) raft_consensus.cc:1685] finished
0709 13:04:54.255064 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:54.255230 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37742,"raft.queue_time_us":3977,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":10548,"spinlock_wait_cycles":64547584,"thread_start_us":94365,"threads_started":1,"wal-append.queue_time_us":237663}
I20260709 13:04:54.268275  9960 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.240947 10014 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.248859  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.282629  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.285195  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:54.286095  1742 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.289681  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.291585  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.272761  9879 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.304702  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.307077 10014 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.297940 10029 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.313302  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.314090  1737 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:54.314447  1662 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:54.270483 10026 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.320865  1742 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:54.322539  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.343922  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.340672  9891 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:54.325074 10014 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.320530  9963 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.351615  9769 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:54.299727  9879 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.356258  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.358405  1662 raft_consensus.cc:2410] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 13:04:54.352433  9769 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.365437  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:54.365751 10028 log.cc:927] Time spent T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 13:04:54.367084  1733 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 28.
I20260709 13:04:54.352365 10023 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.321969 10029 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.397955  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:54.398978  1662 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.413111  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.355563  9891 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.360416  9616 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.405254 10023 raft_consensus.cc:493] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:54.412073 10030 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.188s	user 0.000s	sys 0.002s
I20260709 13:04:54.385628  9977 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.425787  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:54.418045  9943 consensus_meta.cc:287] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: Time spent flushing consensus metadata: real 0.666s	user 0.012s	sys 0.004s
I20260709 13:04:54.362538 10014 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.428826  1666 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:54.433260  1739 raft_consensus.cc:1275] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 13:04:54.435151  1666 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:54.435526  1666 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:54.436267  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.435460  1661 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 27.
I20260709 13:04:54.437350  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:54.436435  1666 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.443324  1666 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:54.444507  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.445604  1734 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55272 (request call id 3722) took 1341 ms. Trace:
I20260709 13:04:54.446131  1734 rpcz_store.cc:276] 0709 13:04:53.104404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:53.104541 (+   137us) service_pool.cc:224] Handling call
0709 13:04:53.188428 (+ 83887us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:53.282928 (+ 94500us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:53.282978 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:53.283623 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 13:04:53.368610 (+ 84987us) raft_consensus.cc:1645] Marking committed up to 11
0709 13:04:53.368657 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:54.434667 (+1066010us) spinlock_profiling.cc:243] Waited 84.7 ms on lock 0x7b54003511d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdde01 00007fcdedfdc83e 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6
0709 13:04:54.434728 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:54.434775 (+    47us) raft_consensus.cc:1685] finished
0709 13:04:54.434790 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:54.445539 (+ 10749us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":297,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5145,"spinlock_wait_cycles":194870016,"thread_start_us":55485,"threads_started":1,"wal-append.queue_time_us":137257}
I20260709 13:04:54.432672 10023 raft_consensus.cc:515] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.445237  1733 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:54.451773  2310 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 13:04:54.403851  9890 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.462178  1741 raft_consensus.cc:1275] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 13:04:54.465802  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.467324  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.469831  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.449941  9948 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:54.467917 10027 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.178s	user 0.000s	sys 0.002s
I20260709 13:04:54.486616  1733 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:54.487986  1556 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.476994  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.493403  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.494239 10023 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.428925  9616 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:54.494711 10032 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 13:04:54.494529 10021 consensus_queue.cc:1059] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:54.508952  1662 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:54.498373  9759 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.515252  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.515789  1666 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:54.504585  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.527371  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.510737  9936 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:54.549276  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.552778  1734 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:54.560735  9616 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.566617  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.569217  9616 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.521668 10026 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.518980 10034 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:04:54.582700  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 53513.2us
I20260709 13:04:54.542320  9963 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:54.586210  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.502621  9948 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.574018 10036 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.573043  9940 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:54.590209 10036 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.590550  9940 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:54.600982   431 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 1618)
I20260709 13:04:54.558555  9879 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.545397  9759 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.569907  9616 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.569473  9977 consensus_queue.cc:1048] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 13:04:54.596923 10034 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:04:54.638396 10033 log.cc:927] Time spent T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 13:04:54.601810  9963 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:54.640976  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 13:04:54.647681  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.655561  1632 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:54.646277  9971 raft_consensus.cc:493] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:54.675413  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.676868  9713 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:54.677924  1556 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.679503  9713 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:54.680754  1556 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.687165  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:54.658282  9990 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:54.691017  9656 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:54.670998 10014 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.694711  1737 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27. Ops: []
I20260709 13:04:54.694818  1556 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.699569  1556 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.695082 10039 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.011s
I20260709 13:04:54.703589 10043 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:54.656368  9838 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.650035 10034 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.704231 10043 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.691602  9833 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:04:54.692890 10029 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.709196  9656 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:54.687608  9259 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 13:04:54.685717  9971 raft_consensus.cc:515] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.706501 10014 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:54.735186  9971 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.735291 10043 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:54.732308  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 140957us
I20260709 13:04:54.713368  9713 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 13:04:54.720412  9891 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.741967  1556 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:54.744952  9656 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:54.745576  9656 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:54.743351  9713 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:54.743778  9759 leader_election.cc:290] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.744065  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:54.718149  9948 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:04:54.756584 10049 consensus_queue.cc:1048] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 13:04:54.758749  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:54.758944  1705 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:54.760259 10043 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:54.760829 10043 raft_consensus.cc:697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:54.761495 10023 raft_consensus.cc:3055] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:04:54.761977 10043 consensus_queue.cc:237] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.762552  9971 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:54.763402  9971 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.761977 10023 raft_consensus.cc:740] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:54.768759  9971 leader_election.cc:290] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.769611 10023 consensus_queue.cc:260] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.771275  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.771664 10023 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:54.777949 10024 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.788666  9769 leader_election.cc:290] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.752995 10026 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 13:04:54.882397 10026 consensus_queue.cc:1243] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:54.883419 10020 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.884042 10020 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.885739  9838 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.737953 10029 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 13:04:54.746893  9656 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:54.875095  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:54.878151  9963 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:54.753647 10034 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:54.777980 10036 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.892668  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49480 (request call id 3565) took 1099 ms. Trace:
I20260709 13:04:54.892976  1664 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 13:04:54.752086  9891 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:04:54.890152  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 133763us
W20260709 13:04:54.877615  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 118689us
I20260709 13:04:54.901540  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:54.893060  1661 rpcz_store.cc:276] 0709 13:04:53.797028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:53.797243 (+   215us) service_pool.cc:224] Handling call
0709 13:04:54.892462 (+1095219us) spinlock_profiling.cc:243] Waited 636 ms on lock 0x7b54001a72d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfe12fa 00007fcdf2fa2b17 00007fcdec51d6d4 00007fcdec51d662 00007fcdec51d591 00007fcdec51d50c 00007fcdec51c7c2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:54.892602 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1463903488}
I20260709 13:04:54.904981  9616 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.906472  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d66c6d05a0074fe2a7b7b2466f6182d7" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.886394 10020 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.915187  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.916939  1660 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:54.893810  9963 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.919512  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.920218  9963 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:54.922546  9883 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:54.922978  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.923940  1660 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
I20260709 13:04:54.929392  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.890965  9971 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:54.930333  9971 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:54.932761  9971 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:54.935881  1660 raft_consensus.cc:3055] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:54.937829  1660 raft_consensus.cc:740] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:54.948310  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:54.950270  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48152.6us
I20260709 13:04:54.949430  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.953953  1662 raft_consensus.cc:2410] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:54.950533  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.958643  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2874) took 1128 ms. Trace:
I20260709 13:04:54.959172  1664 rpcz_store.cc:276] 0709 13:04:53.830548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:53.830679 (+   131us) service_pool.cc:224] Handling call
0709 13:04:54.892612 (+1061933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 13:04:54.893490 (+   878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:54.958409 (+ 64919us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001a72d0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfdc8ad 00007fcdf2fa1419 00007fcdec518bd4 00007fcdec518b62 00007fcdec518a91 00007fcdec518a0c 00007fcdec517cc2 00007fcdec31848c 00007fcdec3178c6 00007fcdec31726f
0709 13:04:54.958571 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2394029824}
I20260709 13:04:54.967640  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.967625  1660 consensus_queue.cc:260] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.972765  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.974184  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.974769  2309 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 20.
I20260709 13:04:54.973860  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:54.976905  1660 raft_consensus.cc:3060] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:54.978860  9891 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:54.979641  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.979537  9891 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:54.981611  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.982440  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:54.981968  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 289467us
I20260709 13:04:54.983381  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.984725  1742 raft_consensus.cc:2468] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:54.985069  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.985989  1737 raft_consensus.cc:2410] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
W20260709 13:04:54.986877  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:54.987224  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.986929  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:54.989199  1742 raft_consensus.cc:2410] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 13:04:54.990338  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:54.983232  2310 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 15.
W20260709 13:04:55.005102  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:55.005607  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.006369  2309 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
W20260709 13:04:55.005872  1634 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:55.008205  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.012403  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:54.946274 10036 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:55.034049  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:55.014963  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.021663  1735 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:55.024366  9932 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:55.037882  9932 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.040058  9932 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:55.040352  9891 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:55.002843  9883 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:55.041184  9883 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.041625 10036 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.043630  9883 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.044209 10036 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:55.050897  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.052325  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.053885  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.055136  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.055583  9932 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.026386 10045 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:55.029229  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:55.025053  9934 consensus_queue.cc:1048] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:55.056263  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.070508  1736 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 20.
I20260709 13:04:55.061244  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.062057  1663 pending_rounds.cc:85] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 14
I20260709 13:04:55.049211  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.076915  1663 pending_rounds.cc:107] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 13:04:55.077368  1663 raft_consensus.cc:2889] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 13:04:55.063542  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 373423us
I20260709 13:04:55.061005  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:55.078230  1738 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:55.057897  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:55.077090  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:55.096473  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.097442  9940 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.098366  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.099689  1667 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:04:55.057175 10014 raft_consensus.cc:3055] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:55.069999 10045 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.083420  9877 raft_consensus.cc:2804] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 13:04:55.109383  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.082682  9616 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:04:55.103235  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 54118us
I20260709 13:04:55.083577  9977 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:04:55.105700 10014 raft_consensus.cc:740] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:55.095397  9869 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:55.126531  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.110620  1738 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:55.110754  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.115361  1557 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.115794  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.115149  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:55.121608  9977 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.107496  9759 raft_consensus.cc:2804] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:55.105679 10013 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:55.112274 10045 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:55.137471  1633 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:55.113008  9616 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.147204  1660 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:55.112710  9877 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:55.150803  1479 tablet_replica.cc:1406] Time spent applying in-flights took a long time: real 1.138s	user 0.000s	sys 0.000s
I20260709 13:04:55.146024  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.130918  9977 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:55.107339  9838 consensus_queue.cc:1059] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 13:04:55.132901  9963 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:55.134300 10029 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:55.181895  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.126796 10024 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 13:04:55.200618  1706 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.176282  9616 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:55.134675  9759 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.179786  9877 raft_consensus.cc:3060] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Advancing to term 20
I20260709 13:04:55.205343  1706 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:55.177827  1557 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.180284 10013 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:04:55.192775  9963 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
W20260709 13:04:55.247646  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 351044us
I20260709 13:04:55.240129  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.214092  9759 raft_consensus.cc:3060] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:55.188539  9883 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 13:04:55.168937 10037 consensus_queue.cc:1048] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
W20260709 13:04:55.254249  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.210232  9891 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:55.251582  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:55.243201  1706 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:55.257229  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.259529  1706 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.266800  1706 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:55.235267  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 57500.8us
I20260709 13:04:55.273094  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.246439  9937 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:55.274940  2318 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 13:04:55.276288  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:55.276777  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.249933 10043 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 13:04:55.275187  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 124483us
I20260709 13:04:55.248046  9963 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:55.278064  2318 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:55.284153  1666 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 13:04:55.284534  1557 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.294339 10040 raft_consensus.cc:2749] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:55.295377 10023 raft_consensus.cc:2804] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:04:55.295804 10023 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:55.296185 10023 raft_consensus.cc:3060] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:04:55.300011 10052 log.cc:927] Time spent T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 13:04:55.303750  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a5331c1cc5e4cbf8847232f0d5835c7" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.308094  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5a2ef920c894818b430473b4809d61d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:55.308851  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 123041us
I20260709 13:04:55.272768  9940 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:55.308460  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:55.314504  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.275413  9764 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:55.317765  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.312777  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.328006  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:55.328420  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:55.278898 10043 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.350324  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:55.353300  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "92fb459de10a4e038d6cbee7b88dc3ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.350891  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.350914  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.354839  2318 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 15.
I20260709 13:04:55.356354  1632 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:55.317021  9940 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.364176  1632 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:55.365466  2318 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:55.323761  9932 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:55.351325 10027 log.cc:927] Time spent T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 13:04:55.369881 10063 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:55.362094 10049 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:55.351859  9838 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:55.347922 10036 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 13:04:55.325032  9764 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.316617 10024 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:55.384352  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.359827  9656 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:04:55.402376  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:55.407207 10055 log.cc:927] Time spent T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260709 13:04:55.408313 10054 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
W20260709 13:04:55.409116 10053 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 13:04:55.355396 10043 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:55.354815  9259 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 13:04:55.417037  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 150275us
I20260709 13:04:55.375578 10046 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:04:55.429116  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 152774us
I20260709 13:04:55.384625  9940 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:04:55.437292  1479 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.262s	user 0.000s	sys 0.000s
I20260709 13:04:55.399636  9869 raft_consensus.cc:2804] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 13:04:55.423895  9764 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:55.449335  9756 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.424202  9656 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.438874 10041 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:55.446200  9259 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:55.482093 10050 log.cc:927] Time spent T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.223s	user 0.000s	sys 0.000s
W20260709 13:04:55.496384  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 108154us
I20260709 13:04:55.417004 10049 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.457541 10021 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:55.455080  9879 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.458943  9869 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.492911  9656 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:55.491614 10065 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 13:04:55.497813  9259 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:55.528891  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 144544us
I20260709 13:04:55.531898  2310 raft_consensus.cc:1275] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 13:04:55.495362  9756 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.523186  9869 raft_consensus.cc:3060] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Advancing to term 29
W20260709 13:04:55.554831  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:55.539439  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:55.522541  9879 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.550269  9883 consensus_queue.cc:1048] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 13:04:55.543781  9977 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.523669  9616 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.603904  9616 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:55.604593   431 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 1618)
I20260709 13:04:55.586248  1668 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2884) took 1701 ms. Trace:
I20260709 13:04:55.605100  1668 rpcz_store.cc:276] 0709 13:04:53.884493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:53.903035 (+ 18542us) service_pool.cc:224] Handling call
0709 13:04:53.903503 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:53.906450 (+  2947us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:53.906487 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:53.923717 (+ 17230us) log.cc:844] Serialized 52 byte log entry
0709 13:04:54.782269 (+858552us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18001a0020. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:55.149350 (+367081us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:55.325265 (+175915us) log.cc:844] Serialized 14 byte log entry
0709 13:04:55.349097 (+ 23832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:55.349190 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:55.581942 (+232752us) raft_consensus.cc:1685] finished
0709 13:04:55.585996 (+  4054us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:55.586177 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67911,"prepare.queue_time_us":256548,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":24697,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":13702,"spinlock_wait_cycles":5816704,"thread_start_us":402676,"threads_started":2,"wal-append.queue_time_us":1275611}
I20260709 13:04:55.608673 10045 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.528138  9833 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
W20260709 13:04:55.595876  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 92455.3us
I20260709 13:04:55.599206  9879 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.600332  9877 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.602006  9977 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.621651  1632 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:55.623734  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45934.1us
I20260709 13:04:55.599208  9756 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:55.642936  1631 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:55.645745 10036 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:55.648437  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:55.637714 10071 log.cc:927] Time spent T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 13:04:55.641211  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 111337us
I20260709 13:04:55.663461  1557 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:55.674881  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:55.676009  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58630.5us
I20260709 13:04:55.678371  1557 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.683115  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:55.617452  9764 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:55.693254  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.690848  2311 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 13:04:55.641877 10077 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:55.694674  1634 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:55.701247  1634 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:55.702447  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.670756 10029 raft_consensus.cc:2804] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:55.681006  9937 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 13:04:55.751449  1738 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:55.705224  1557 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:55.755491  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46417.4us
I20260709 13:04:55.757146  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:55.691563 10045 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:55.706085  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:55.693828 10026 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:55.779070  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 73911.1us
W20260709 13:04:55.751374  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.700778 10021 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:55.792616  1662 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: [19.11-23.13]
W20260709 13:04:55.785818  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53493.5us
W20260709 13:04:55.794059  1634 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:55.790033  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:55.800628  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:55.794502  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:55.801599  1740 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:55.682788  9259 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.806380  1737 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:04:55.726200  9990 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:55.719300  9764 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:55.824429  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:55.825738  2311 raft_consensus.cc:2393] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 23: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:04:55.733314 10029 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:55.727293  9877 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:55.815543  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 122823us
I20260709 13:04:55.726035 10037 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.832145  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "567113a84c1d465597e948caf9276c34" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:55.751770 10049 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.714607 10079 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:55.673425 10023 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.840708  1733 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 13:04:55.807432  9977 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.867916  9977 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.714283  9963 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.833894 10026 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:55.842794 10029 raft_consensus.cc:3060] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:55.856173 10080 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:55.861666  9990 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:55.896574 10023 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:55.898075  9259 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:55.720176 10014 consensus_queue.cc:260] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.902339 10014 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:55.832701  9957 consensus_queue.cc:1059] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
W20260709 13:04:55.882040  9759 consensus_meta.cc:287] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.618s	user 0.007s	sys 0.007s
I20260709 13:04:55.945711  9977 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.863085 10037 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.937721 10077 raft_consensus.cc:3055] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 13:04:55.960548 10090 log.cc:927] Time spent T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 13:04:55.937090 10080 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.955051 10077 raft_consensus.cc:740] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:55.969709  9940 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.971143 10037 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:55.972203  9963 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:55.973258  9869 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:55.975970  9869 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:55.978343 10080 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:55.978387 10077 consensus_queue.cc:260] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:55.986831 10043 consensus_meta.cc:287] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.529s	user 0.012s	sys 0.003s
I20260709 13:04:55.994791 10029 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:55.734630  1479 maintenance_manager.cc:643] P e6b16382849d482ba544fc5ee58e0b0a: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 2.117s	user 0.027s	sys 0.014s Metrics: {"bytes_written":3951,"delete_count":0,"lbm_write_time_us":17707,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":3,"reinsert_count":0,"update_count":139}
I20260709 13:04:55.786831  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:55.819991  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:56.068619  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:56.226970  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 406666us
I20260709 13:04:56.227239  1660 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Advancing to term 28
W20260709 13:04:56.228415  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:56.070305  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.079331  1664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49480 (request call id 3583) took 1025 ms. Trace:
W20260709 13:04:56.230031  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:56.230165  1662 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:04:56.232445  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:56.233686  9656 consensus_meta.cc:287] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: Time spent flushing consensus metadata: real 0.681s	user 0.012s	sys 0.000s
I20260709 13:04:56.234298  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:56.234849  9656 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.236529  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:56.237450  2311 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:56.237821  9948 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:56.140687  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:56.238804  9948 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.240058  1739 raft_consensus.cc:2410] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 13:04:56.002146 10043 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.079332  1659 raft_consensus.cc:1240] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 21. Current term is 22. Ops: [21.16-21.16]
W20260709 13:04:56.151890  9973 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.011s	sys 0.000s
I20260709 13:04:56.251401  1742 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:56.230062  1664 rpcz_store.cc:276] 0709 13:04:55.053490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:55.150218 (+ 96728us) service_pool.cc:224] Handling call
0709 13:04:55.177695 (+ 27477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:55.279689 (+101994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:55.279724 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:55.307447 (+ 27723us) log.cc:844] Serialized 26 byte log entry
0709 13:04:55.535088 (+227641us) raft_consensus.cc:1645] Marking committed up to 9
0709 13:04:55.535134 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:55.535215 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:56.078314 (+543099us) raft_consensus.cc:1685] finished
0709 13:04:56.078412 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:56.078982 (+   570us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7583,"raft.queue_time_us":32710,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":32112,"thread_start_us":268660,"threads_started":2,"wal-append.queue_time_us":486092}
I20260709 13:04:56.233211  9934 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.157212  9759 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.258615  9934 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.261281  9759 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.264081 10029 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.264120  1663 raft_consensus.cc:1275] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:04:56.267040 10057 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:56.268147 10057 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.270290  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:56.274417  1480 maintenance_manager.cc:419] P e6b16382849d482ba544fc5ee58e0b0a: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.202775
W20260709 13:04:56.164108  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:56.169351  9963 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:56.170369  9977 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:56.174386  2310 raft_consensus.cc:1240] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:56.176158  9883 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:04:56.183845 10078 log.cc:927] Time spent T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.466s	user 0.009s	sys 0.007s
I20260709 13:04:56.185125  9769 raft_consensus.cc:493] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:56.196502  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:56.206079  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 421150us
I20260709 13:04:56.147387 10048 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:56.216379 10037 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:04:56.229367  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:55.989727  9940 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:56.288484  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59276us
I20260709 13:04:56.305853  9963 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.306577  1661 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49484 (request call id 2909) took 1366 ms. Trace:
I20260709 13:04:56.307132  1661 rpcz_store.cc:276] 0709 13:04:54.939684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:54.939896 (+   212us) service_pool.cc:224] Handling call
0709 13:04:54.940351 (+   455us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 13:04:54.940899 (+   548us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:54.940944 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 13:04:54.946832 (+  5888us) log.cc:844] Serialized 52 byte log entry
0709 13:04:55.056160 (+109328us) spinlock_profiling.cc:243] Waited 4.28 ms on lock 0x7b4800210258. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf2f06aa1 00007fcdf2efdb37 00007fcdf2efd9ca 00007fcdf2ff526f 00007fcdf2ff4f37 00007fcdf300955c 00007fcdf3009508 00007fcdf3009488 00007fcdf3009443 00007fcdf30086ec 00007fcdf1f32a96
0709 13:04:56.066372 (+1010212us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b180020e4e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaed2e07 00007fcdf3d2dab9 00007fcdeaeef69b 00007fcdeaeed93f 00007fcdeaeefc02 00007fcdedb7720b 00007fcdedb7cd23 00007fcdedb7d0d1 00007fcdedf98a0f
0709 13:04:56.222776 (+156404us) raft_consensus.cc:1645] Marking committed up to 12
0709 13:04:56.224901 (+  2125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:56.281227 (+ 56326us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:56.281287 (+    60us) raft_consensus.cc:1685] finished
0709 13:04:56.281300 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:56.306338 (+ 25038us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":791,"raft.run_cpu_time_us":2201,"raft.run_wall_time_us":106670,"spinlock_wait_cycles":18727552,"thread_start_us":67687,"threads_started":1,"wal-append.queue_time_us":1174395}
I20260709 13:04:56.309103  9963 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.321985  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:56.309273  9769 raft_consensus.cc:515] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.322482 10048 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.322858  9883 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.319979  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.335582  9769 leader_election.cc:290] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.335587  9883 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:56.340957  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 553888us
I20260709 13:04:56.341878  9759 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:56.342712  9759 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:56.344295  9759 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.317423  9977 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.346638  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:56.348356  9977 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.349016 10037 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.356194  1664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.357227  1664 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
I20260709 13:04:56.362578  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:56.363395 10088 log.cc:927] Time spent T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.637s	user 0.015s	sys 0.009s
I20260709 13:04:56.391047  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.395412  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.405117  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.369185 10077 raft_consensus.cc:3060] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:56.411614  1634 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:56.412559  1634 leader_election.cc:336] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:56.413401  1634 leader_election.cc:336] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:56.414764  1733 raft_consensus.cc:1240] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Rejecting Update request from peer e6adef9da7da44f6af0af5695cf9b704 for earlier term 25. Current term is 26. Ops: []
I20260709 13:04:56.399680  9934 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:56.420460  1556 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.011s	sys 0.032s
W20260709 13:04:56.425946  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 646666us
I20260709 13:04:56.441689  9883 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.442649  9883 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.444954 10048 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.453900  9948 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.456377  1660 raft_consensus.cc:2468] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
I20260709 13:04:56.458207  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.459489  1668 raft_consensus.cc:2393] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 26: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:56.465698  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.478749  9957 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.479668  9957 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.484984  9957 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.485603  9656 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.488592  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:56.491204 10037 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:56.499203  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:56.503051  2309 raft_consensus.cc:1240] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:56.501120  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:56.505026  9616 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.506137  9616 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:56.507897  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:56.509099  9616 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.510290 10057 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.511068  9656 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.511927  9656 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:56.513374 10076 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.819s	user 0.007s	sys 0.017s
I20260709 13:04:56.515029  9656 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.515376  1736 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42028 (request call id 1440) took 1207 ms. Trace:
I20260709 13:04:56.515470  1557 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:56.518455  1664 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
I20260709 13:04:56.519017  9616 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:04:56.534091  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.543979  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:56.544334  1483 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 134) took 14573 ms (client timeout 15000 ms). Trace:
I20260709 13:04:56.548316  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.552260  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.553869  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.554627  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.555902  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.556180  9948 consensus_queue.cc:1059] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:56.557440  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:56.558183  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:56.560580  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce79fa9e7595481c8379fd2bdeca8a6e" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:56.561183  9883 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.562469  1660 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 13:04:56.567286  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:56.573063 10043 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.595700  1665 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:56.597986  9948 raft_consensus.cc:3055] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:56.598989  9948 raft_consensus.cc:740] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:56.600646  9948 consensus_queue.cc:260] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.568536  1482 catalog_manager.cc:5697] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 12 to 14, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 14 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:56.602756  2283 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:56.604064  1666 raft_consensus.cc:2410] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
W20260709 13:04:56.604964  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 169010us
I20260709 13:04:56.605171  9883 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 13:04:56.605873   431 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 1618)
W20260709 13:04:56.584074  1483 rpcz_store.cc:269] 0709 13:04:41.971288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:47.572716 (+5601428us) service_pool.cc:224] Handling call
0709 13:04:56.544006 (+8971290us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:52.588760 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:52.589589 (+   829us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:52.589650 (+    61us) write_op.cc:435] Acquired schema lock
0709 13:04:52.589678 (+    28us) tablet.cc:662] Decoding operations
0709 13:04:52.591178 (+  1500us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:52.591266 (+    88us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:52.591310 (+    44us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:04:52.592621 (+  1311us) tablet.cc:701] Row locks acquired
0709 13:04:52.592645 (+    24us) write_op.cc:260] PREPARE: finished
0709 13:04:52.592874 (+   229us) write_op.cc:270] Start()
0709 13:04:52.593052 (+   178us) write_op.cc:276] Timestamp: P: 1783602292592798 usec, L: 0
0709 13:04:52.593082 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 13:04:52.616324 (+ 23242us) log.cc:844] Serialized 4430 byte log entry
0709 13:04:53.276468 (+660144us) op_driver.cc:464] REPLICATION: finished
0709 13:04:53.592083 (+315615us) write_op.cc:301] APPLY: starting
0709 13:04:53.604046 (+ 11963us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:54.199059 (+595013us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:54.202811 (+  3752us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:54.780268 (+577457us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:54.916185 (+135917us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=44,mrs_lookups=0
0709 13:04:54.916248 (+    63us) write_op.cc:312] APPLY: finished
0709 13:04:54.917231 (+   983us) log.cc:844] Serialized 105 byte log entry
0709 13:04:55.150616 (+233385us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:55.179174 (+ 28558us) write_op.cc:454] Released schema lock
0709 13:04:55.192107 (+ 12933us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:55.212348 (+ 20241us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:55.212402 (+    54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17321,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":298155,"apply.run_cpu_time_us":59047,"apply.run_wall_time_us":1621456,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50129,"num_ops":11,"prepare.queue_time_us":296469,"prepare.run_cpu_time_us":5807,"prepare.run_wall_time_us":88600,"replication_time_us":630596,"thread_start_us":529924,"threads_started":3,"wal-append.queue_time_us":331403,"wal-append.run_cpu_time_us":2500,"wal-append.run_wall_time_us":1205845}]]}
I20260709 13:04:56.610041  2311 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 16.
W20260709 13:04:56.616695  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 202368us
I20260709 13:04:56.626986  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:56.630326  9656 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:04:56.634384  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 100662us
I20260709 13:04:56.615370  9883 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:56.627897  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.582414  1736 rpcz_store.cc:276] 0709 13:04:55.307386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:55.307525 (+   139us) service_pool.cc:224] Handling call
0709 13:04:55.307843 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 13:04:55.313951 (+  6108us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 13:04:55.313984 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 13:04:55.323645 (+  9661us) log.cc:844] Serialized 26 byte log entry
0709 13:04:55.527397 (+203752us) raft_consensus.cc:1645] Marking committed up to 14
0709 13:04:55.527455 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 13:04:55.527547 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 13:04:56.512427 (+984880us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b4400022b4c. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeae16c85 00007fcdeae176a2 00007fcdedb891db 00007fcdedb84aaf 00007fcdedb7886e 00007fcdedb77bb6 00007fcdedb777d8 00007fcdedb81221 00007fcdedb811d9 00007fcdedb81169
0709 13:04:56.514850 (+  2423us) raft_consensus.cc:1685] finished
0709 13:04:56.514872 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 13:04:56.515102 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8367,"spinlock_wait_cycles":53381376,"thread_start_us":157450,"threads_started":1,"wal-append.queue_time_us":270639}
I20260709 13:04:56.647306  2315 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:56.636576 10048 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:56.655474  9259 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:56.656175  9259 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.658921  9259 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:56.658686  1482 catalog_manager.cc:5697] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 7 to 18. New cstate: current_term: 18 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:04:56.660666  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 145369us
I20260709 13:04:56.662992 10043 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:56.663414  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:56.663668 10043 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.664209  1742 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:56.665757 10037 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:04:56.666210  9616 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:56.667128 10043 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.665221  1557 leader_election.cc:304] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:56.642201  9883 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:56.644151  9656 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.681691  1632 leader_election.cc:304] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:56.684592 10077 raft_consensus.cc:2749] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:56.687132  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:56.687687  9879 consensus_queue.cc:1048] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 13:04:56.686977  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:56.693899  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44500.8us
W20260709 13:04:56.713979  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 48940.6us
I20260709 13:04:56.718277  2315 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:04:56.717943  9948 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:56.719889  1557 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.728947 10041 consensus_queue.cc:1059] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:04:56.729907  1482 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 14. New cstate: current_term: 14 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:56.721668 10048 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:56.695906 10077 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 13:04:56.706396  9759 raft_consensus.cc:2749] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:56.704545  9879 consensus_queue.cc:1243] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [LEADER]: Peer ff44a4643e01482cb6b6a79b099fa766 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 13:04:56.707597  9656 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:56.737704  1740 consensus_meta.cc:287] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: Time spent flushing consensus metadata: real 0.892s	user 0.011s	sys 0.015s
I20260709 13:04:56.746361  9616 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:56.748461  1740 raft_consensus.cc:2468] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:56.761144  1735 raft_consensus.cc:1240] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:56.761730  1632 leader_election.cc:304] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:56.763533  9833 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:56.761941  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 159361us
W20260709 13:04:56.777227  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 13:04:56.780062  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 113225us
W20260709 13:04:56.784271  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 64667.7us
I20260709 13:04:56.785493  1557 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:56.791419  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.793725  9879 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:56.794281  9759 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:56.794829  9259 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:56.795253  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.799907  1482 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 9 to 11, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:56.805930  1705 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.814047 10080 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:04:56.813978  9833 consensus_queue.cc:1059] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:04:56.815305 10077 raft_consensus.cc:3055] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:56.815659 10077 raft_consensus.cc:740] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:56.816669 10077 consensus_queue.cc:260] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.817390  9934 raft_consensus.cc:3055] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 13:04:56.817840  9934 raft_consensus.cc:740] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:56.818958  9934 consensus_queue.cc:260] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.820940 10077 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:56.796659  9759 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.828961  9759 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.837728 10045 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:56.838567 10045 raft_consensus.cc:493] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:04:56.839279 10045 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:56.841099  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 44060.7us
I20260709 13:04:56.853642  2281 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.855502 10057 raft_consensus.cc:2804] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:56.856331 10057 raft_consensus.cc:697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:56.857556 10057 consensus_queue.cc:237] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.795284  9259 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:56.858964  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 52236us
W20260709 13:04:56.862843  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 77588.7us
W20260709 13:04:56.864295  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 89138.7us
W20260709 13:04:56.865720  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62200.4us
I20260709 13:04:56.812705  9656 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:56.874346  1634 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 115 similar messages]
W20260709 13:04:56.881170  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:56.881686  1634 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:56.883279  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:56.883782  2283 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.870599  9934 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:04:56.896375  9883 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.897809  1738 raft_consensus.cc:2393] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 24: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:56.882659  9957 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:56.825941  1482 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 13 to 18, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 18 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:56.911562  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 45412.3us
I20260709 13:04:56.892155 10048 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:56.886916  9656 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:56.914299  1482 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 7 to 15. New cstate: current_term: 15 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:56.862730  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
W20260709 13:04:56.927007  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45847.6us
W20260709 13:04:56.927187  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64466.9us
I20260709 13:04:56.930727  1557 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:56.933669 10047 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:56.919281  9759 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.953101  9883 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:56.959486  1482 catalog_manager.cc:5697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 1 to 17. New cstate: current_term: 17 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:56.938246 10048 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:56.949780  9656 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.987085  9656 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:56.924260  9259 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:56.988806  9259 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:56.947932  1705 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:56.943241  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 67267.9us
I20260709 13:04:57.015144  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:57.017611  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 100331us
I20260709 13:04:57.026471 10080 raft_consensus.cc:2804] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:04:57.026487  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:57.028059  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 108955us
I20260709 13:04:57.037113  1665 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 13:04:57.047703  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.024444  1482 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 6 to 16, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 16 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:57.052543  1663 raft_consensus.cc:2410] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a for term 23 because replica has last-logged OpId of term: 22 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 9.
I20260709 13:04:57.032487 10080 raft_consensus.cc:697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:57.037158 10048 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.056996  9656 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.059856 10080 consensus_queue.cc:237] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:57.061117  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 130545us
I20260709 13:04:57.062927  9838 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:57.063670  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 193972us
I20260709 13:04:57.063283  1482 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 12 to 15, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 15 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:57.042510 10082 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.069201  1705 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:57.081326  9759 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:57.081046  9656 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.082463  9616 raft_consensus.cc:493] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.069679 10021 raft_consensus.cc:493] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.080386  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.098181  9616 raft_consensus.cc:515] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.096894  1660 raft_consensus.cc:2468] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
W20260709 13:04:57.104944  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 49030.4us
I20260709 13:04:57.107082  1557 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:57.112521  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.113413  1634 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.129843 10021 raft_consensus.cc:515] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.131810  1738 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:04:57.124840  9883 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election won for term 16
W20260709 13:04:57.136175  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.130122  1634 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.105638 10082 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.161015  9838 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 13:04:57.157904  1634 leader_election.cc:336] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.164669  1735 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 13:04:57.172093  9883 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:04:57.189538  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:57.193804  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.196269  1738 raft_consensus.cc:1240] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
I20260709 13:04:57.206063  1632 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:57.204327  1737 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:57.247783 10117 log.cc:927] Time spent T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.165s	user 0.001s	sys 0.001s
I20260709 13:04:57.250016  1482 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 5 to 17. New cstate: current_term: 17 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:57.252178  1740 raft_consensus.cc:1240] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 23. Current term is 24. Ops: []
W20260709 13:04:57.256493 10125 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.197s	user 0.000s	sys 0.003s
W20260709 13:04:57.262205  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 218337us
W20260709 13:04:57.263679  1708 leader_election.cc:336] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.264122  1708 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:57.264616  9656 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.266404 10021 leader_election.cc:290] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:57.191376  9616 leader_election.cc:290] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:57.268708  1559 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:57.271266  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.272225  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.274032  1733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.279935 10045 raft_consensus.cc:515] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.280844  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:57.238924  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.281991  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45848.9us
I20260709 13:04:57.282335  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:04:57.282773  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:57.289588  1736 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 LEADER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 27. Current term is 28. Ops: []
I20260709 13:04:57.290019  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.304391  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:57.304700  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 197604us
I20260709 13:04:57.253261  1482 catalog_manager.cc:5697] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 8 to 18. New cstate: current_term: 18 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:04:57.302461  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.311237  1665 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:04:57.316576 10045 leader_election.cc:290] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.319101  1666 pending_rounds.cc:85] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 10
I20260709 13:04:57.319484  1666 pending_rounds.cc:107] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 13:04:57.319788  1666 raft_consensus.cc:2889] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:57.291051 10080 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:57.257349 10082 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.323714  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:57.258517  9883 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.326918  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "648ab3b5826e49fe81c3363d752d8af8" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:04:57.328593  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:57.342294  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45817.7us
W20260709 13:04:57.354841  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 149250us
I20260709 13:04:57.357352  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.357429  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.358318  9759 consensus_queue.cc:1059] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 13:04:57.359405 10082 raft_consensus.cc:3055] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Stepping down as leader of term 20
I20260709 13:04:57.359222  9759 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:04:57.359817 10082 raft_consensus.cc:740] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:57.360306  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:57.361884  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 98224.2us
I20260709 13:04:57.359223  1734 raft_consensus.cc:2410] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
W20260709 13:04:57.362694  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.363917  1708 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.303035 10077 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:57.368363  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 222030us
I20260709 13:04:57.372071  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:57.372815  2315 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:04:57.375097  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c6fca959014bb6a785fac033e5fe47" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.375926  1735 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:57.369643  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:57.409528  9891 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:57.414146  1708 leader_election.cc:336] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.414757  1708 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:57.416494  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.360968 10082 consensus_queue.cc:260] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.418648 10082 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:04:57.419231  1556 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:57.402313  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.419983  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.420585 10134 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:04:57.421437  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.421654 10021 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 13:04:57.421949  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.422309  9759 raft_consensus.cc:2804] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:57.422724  9759 raft_consensus.cc:493] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:57.423079  9759 raft_consensus.cc:3060] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:04:57.423274  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.423699  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.425736  9948 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:57.426525  9948 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:57.427206  9948 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:57.428188 10080 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:57.436944  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:57.437850  2315 raft_consensus.cc:2393] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 316f3bf18ed94e32b238130a6bfedf1a in current term 16: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:04:57.450393  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 90162.1us
I20260709 13:04:57.451278  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.448774  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:57.460105  1737 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:57.462944  2311 raft_consensus.cc:1240] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 13:04:57.463685  1738 raft_consensus.cc:1240] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 13:04:57.379168  9616 consensus_queue.cc:1059] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.063s
W20260709 13:04:57.471324  2283 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.473049  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:57.473966  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.478716  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:57.462356 10043 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:57.488974  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 68772.4us
I20260709 13:04:57.502396  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:57.507931  1632 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:57.490895 10122 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:57.508750  9656 raft_consensus.cc:3055] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:57.538915  9656 raft_consensus.cc:740] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:57.539963  9656 consensus_queue.cc:260] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.505641 10043 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:57.566496  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 95161.1us
I20260709 13:04:57.529665  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:57.577734  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.579198  1668 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 13:04:57.579387  1663 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.494189 10139 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:57.581631  1660 raft_consensus.cc:1240] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 27. Current term is 28. Ops: [27.16-27.16]
W20260709 13:04:57.585765  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 77948.8us
I20260709 13:04:57.579686  1668 pending_rounds.cc:85] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 14
I20260709 13:04:57.586560  1668 pending_rounds.cc:107] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 13:04:57.586907  1668 raft_consensus.cc:2889] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:57.593648  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.598739  1659 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 13:04:57.599164  1659 pending_rounds.cc:85] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 12
I20260709 13:04:57.599416  1659 pending_rounds.cc:107] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 13:04:57.599646  1659 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:57.552225 10132 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.602157 10132 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.552378 10134 raft_consensus.cc:493] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.604303 10134 raft_consensus.cc:515] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.606834 10134 leader_election.cc:290] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.551884  9616 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:57.584581  9879 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:57.595549  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.608376  9879 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:57.608893  9879 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:57.541484  9656 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:57.612255 10129 raft_consensus.cc:3055] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 LEADER]: Stepping down as leader of term 27
I20260709 13:04:57.612943 10129 raft_consensus.cc:740] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:57.588840  9833 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:57.613986 10129 consensus_queue.cc:260] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:57.617650   431 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 1618)
I20260709 13:04:57.621845 10077 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:57.621904 10143 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 13:04:57.631729  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.634683  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.635901  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4e74a6fd31c14c17a7248fe6eb410c77" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.559751 10043 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:57.638135  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 187776us
I20260709 13:04:57.605137 10132 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:57.641397  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45734.6us
W20260709 13:04:57.646193  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 53430.6us
W20260709 13:04:57.655498  1708 leader_election.cc:336] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.655786  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.656106  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:04:57.656201  1708 leader_election.cc:304] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:57.666337  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "422771fe34aa4b29ba2702785d38559a" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.671203  2309 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
W20260709 13:04:57.670928  1556 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:57.673770  1741 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:04:57.678104  1632 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:57.680131  1741 raft_consensus.cc:1275] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:04:57.680122  1632 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:57.685945  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.689888  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.693111  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "01669b686f2848abb4ec03f67aa748ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.703235  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ae371f2c53648c2992a835477d85184" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.690665  1660 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 26.
I20260709 13:04:57.707937  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.679702 10143 consensus_queue.cc:1243] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer df6a18af06d543b2931837f5e6bab95f log is divergent from this leader: its last log entry 28.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 13:04:57.707415 10120 raft_consensus.cc:493] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.710952  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce1eb834927c4d3caa029344d6f69de5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.674434  9259 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:57.640348  9759 raft_consensus.cc:515] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.691255  9957 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:57.679798 10043 raft_consensus.cc:2749] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:57.718968  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 63261.7us
I20260709 13:04:57.723582  1557 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.725973  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.726748  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.727484  1660 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:04:57.733909  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:57.734282  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 78819.1us
W20260709 13:04:57.735168  1708 leader_election.cc:336] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:57.716802 10120 raft_consensus.cc:515] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.751098  9759 leader_election.cc:290] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.754212 10080 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:57.754820 10080 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:57.755363 10080 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:57.713269  9259 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:57.735596  1708 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:57.758634  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfe67343bf1842888d6176ed2ffe1419" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.721622  9833 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:57.715914  9616 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:04:57.786911 10120 leader_election.cc:290] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:57.712993  9838 consensus_queue.cc:1059] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 13:04:57.750919 10144 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.121s
W20260709 13:04:57.794226  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 59063.7us
W20260709 13:04:57.772301  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 53960us
I20260709 13:04:57.778846  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:57.780293  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 56796.6us
I20260709 13:04:57.782811  9759 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 13:04:57.718334 10143 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 13:04:57.770318  9259 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:57.807322  1735 raft_consensus.cc:2468] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:57.819621  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:57.820647  9616 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:57.821292  9616 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:57.821825  9616 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:57.823858  9879 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.825285  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.826417  1742 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
I20260709 13:04:57.827872  1738 raft_consensus.cc:1240] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 13:04:57.827052  1736 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:57.826222  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf77b84dc20e4118ad47baa1808a85d9" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:57.838320  9838 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.838376  9879 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:57.839005  1557 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.839198 10139 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.839998 10139 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.840979  1557 leader_election.cc:304] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.842653 10139 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.844416 10139 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 13:04:57.846065  2281 leader_election.cc:304] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:57.846959  9838 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:57.848927 10121 raft_consensus.cc:2804] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:04:57.849525 10121 raft_consensus.cc:697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:57.850162  9838 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.850593 10121 consensus_queue.cc:237] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.852144  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.804221 10140 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:57.855376  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a860695ab4c493b83dfcd08d8bee8de" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:57.860500  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:57.826746 10057 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:57.862149 10057 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:57.864964 10057 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:57.866643  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:57.873108  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 43703.2us
I20260709 13:04:57.803175 10122 consensus_queue.cc:1048] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.115s
W20260709 13:04:58.041455  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 162329us
I20260709 13:04:58.042829  1557 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.049701 10157 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:58.050351 10157 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.052727 10157 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.053645 10129 raft_consensus.cc:3060] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:04:58.052649  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:57.941126 10154 consensus_queue.cc:1059] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
W20260709 13:04:58.057188  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 174845us
W20260709 13:04:58.044523  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 177674us
I20260709 13:04:58.066053  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:58.047184  9759 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.062981  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:58.044440  9259 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.070508  1659 raft_consensus.cc:1240] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 21. Current term is 22. Ops: []
I20260709 13:04:58.070820  1660 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 22.
I20260709 13:04:58.081477  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "110eba2eb7b34ea1b70ed63aca2feda8" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.085258  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:58.086056  2309 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
W20260709 13:04:58.101852  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 306646us
I20260709 13:04:58.070147 10143 raft_consensus.cc:2749] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:58.070721 10057 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.120656  1706 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:58.132021  9259 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.073148  9759 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.052654  9616 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:58.154886  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 112112us
I20260709 13:04:58.167673 10143 consensus_queue.cc:1059] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:04:58.169896 10143 raft_consensus.cc:3055] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Stepping down as leader of term 21
I20260709 13:04:58.170470 10143 raft_consensus.cc:740] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:58.171471 10148 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:58.172286 10148 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:58.173658 10148 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.175359 10118 raft_consensus.cc:3055] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 13:04:58.175880 10118 raft_consensus.cc:740] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:58.176981 10118 consensus_queue.cc:260] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.143965 10140 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:58.188800 10140 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting leader election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:58.189343 10140 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:58.189922 10080 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.192556 10080 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:04:58.194950 10118 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:58.198933  9838 raft_consensus.cc:493] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.199642  9838 raft_consensus.cc:515] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.133141 10057 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:58.204660  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48640.4us
I20260709 13:04:58.179340  9616 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:04:58.210932  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 130742us
I20260709 13:04:58.210189  1557 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:58.135254  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.235618  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.237376  1557 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:58.248344  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 51481.5us
I20260709 13:04:58.268512  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:58.270538  1659 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:58.231911 10155 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:58.272066  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 136831us
W20260709 13:04:58.273020  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.273443  1708 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.272197  9616 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.276822  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:58.278779  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:58.275249 10021 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:58.281001 10159 log.cc:927] Time spent T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: Append to log took a long time: real 0.139s	user 0.000s	sys 0.002s
I20260709 13:04:58.285791  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:58.286468  1661 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:58.287161 10057 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.292768  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.298494 10143 consensus_queue.cc:260] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:58.310289  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 55746us
I20260709 13:04:58.268157 10139 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:04:58.330276  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 122898us
I20260709 13:04:58.330619  1742 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 26.
I20260709 13:04:58.297753 10171 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:58.331048  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 210597us
I20260709 13:04:58.331825  1557 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.274325  9957 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader e6adef9da7da44f6af0af5695cf9b704)
I20260709 13:04:58.351713  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:58.363626  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:58.370918 10143 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:58.374261 10166 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:58.376662  2309 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:04:58.379627  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47865.6us
I20260709 13:04:58.380625 10140 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.381726 10021 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.272063 10155 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.384771 10139 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 13:04:58.380959  1557 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:58.385628 10169 log.cc:927] Time spent T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: Append to log took a long time: real 0.095s	user 0.004s	sys 0.000s
I20260709 13:04:58.386904  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:58.332456  9838 leader_election.cc:290] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.394575  1661 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 27.
I20260709 13:04:58.347013 10057 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:04:58.406360  1557 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.400453  1556 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:58.408795  1557 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:58.404568  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 62716us
W20260709 13:04:58.417074  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 144045us
I20260709 13:04:58.398381 10048 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:04:58.399606 10057 raft_consensus.cc:493] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:58.389855 10120 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
W20260709 13:04:58.430421  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.387315  9616 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.389303  9957 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.430935  1708 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.423883 10134 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:58.426716  9838 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.466259  9838 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:58.410099  9879 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 13:04:58.468442  1668 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:04:58.469813  9616 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.470067 10140 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.459811  1632 leader_election.cc:304] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:58.427973 10048 raft_consensus.cc:493] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:58.445596  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:58.429877 10021 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:04:58.431532 10057 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:58.480651  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
W20260709 13:04:58.494308  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 93894.8us
I20260709 13:04:58.496407  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:58.478456 10048 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:58.497300  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:58.501107 10172 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.519397  9759 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.502990  9971 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 13:04:58.506558  9957 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.481578 10156 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:58.478813 10037 raft_consensus.cc:2749] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:58.514817 10155 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:58.531757  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 123009us
I20260709 13:04:58.483175 10021 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 27 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
W20260709 13:04:58.528224  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67003.6us
I20260709 13:04:58.529738  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:58.530586  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:58.532673  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 110468us
I20260709 13:04:58.532894  1557 leader_election.cc:304] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:58.535506  2309 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:58.543792  1557 leader_election.cc:304] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.542500  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:58.545068  1708 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.548367  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.549329  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.550606  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.554772 10037 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:04:58.555269 10037 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:58.555733 10037 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:58.559410  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:58.560326  1667 raft_consensus.cc:2393] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
W20260709 13:04:58.560667 10175 log.cc:927] Time spent T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 13:04:58.567097 10021 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.551412  1742 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
I20260709 13:04:58.540217  9833 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.571229  9833 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.545241 10148 raft_consensus.cc:2749] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:58.528343 10172 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.580750 10172 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.582639  1736 raft_consensus.cc:2393] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 26: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:58.583451  1557 leader_election.cc:304] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.586555  2309 raft_consensus.cc:1240] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Rejecting Update request from peer ff44a4643e01482cb6b6a79b099fa766 for earlier term 22. Current term is 23. Ops: []
W20260709 13:04:58.594162  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47998.5us
I20260709 13:04:58.589653 10182 raft_consensus.cc:2749] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.617005  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.585057 10129 raft_consensus.cc:2749] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.616503  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.621093  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.621913  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:58.622998  1736 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 27
I20260709 13:04:58.625252  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.632325  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a44e16fd1e74bb585aa62b3d15e7384" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.595827 10166 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.636317  1735 raft_consensus.cc:1240] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Rejecting Update request from peer 316f3bf18ed94e32b238130a6bfedf1a for earlier term 24. Current term is 25. Ops: [24.13-24.13]
W20260709 13:04:58.650228  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:58.631932  9656 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:58.653908  9656 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.655951 10127 consensus_queue.cc:1059] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 13:04:58.656838 10127 consensus_queue.cc:1243] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Peer e6adef9da7da44f6af0af5695cf9b704 log is divergent from this leader: its last log entry 23.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 13:04:58.654246  9833 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.657485 10181 consensus_queue.cc:1059] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 13:04:58.658252  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:58.658670 10181 raft_consensus.cc:3055] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:58.658776  9656 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.659096 10181 raft_consensus.cc:740] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:58.585109 10048 raft_consensus.cc:515] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.660370  2311 raft_consensus.cc:2368] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27.
I20260709 13:04:58.660192 10181 consensus_queue.cc:260] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:58.680471  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 64223.2us
I20260709 13:04:58.682394  1665 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:58.683053  1665 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 27
W20260709 13:04:58.683996  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 147504us
I20260709 13:04:58.684470  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7f90c9079e24cf2a4c9ab3dd55172ea" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:04:58.684844  1708 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.685272  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.687402  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88e78eac7b5f4006a15379f62bc8c536" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:58.689217  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:58.686777  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:04:58.709605  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.711467  1557 leader_election.cc:304] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:58.709411  1708 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.727787  1557 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.690348 10181 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
W20260709 13:04:58.733629  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:58.729696 10184 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.115s	user 0.000s	sys 0.007s
I20260709 13:04:58.691253 10043 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.730363 10134 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:58.727422  9656 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:58.734103  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.731487 10139 raft_consensus.cc:2749] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:58.742075  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.748109  1736 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 27.
W20260709 13:04:58.748740  1632 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:58.750365  1634 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:58.757467  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:58.763257  1708 leader_election.cc:336] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.740777 10048 leader_election.cc:290] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.758670 10077 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 13:04:58.769119   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:04:58.759313 10057 raft_consensus.cc:515] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.774142  1557 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.785722 10077 raft_consensus.cc:493] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:04:58.793191 10077 raft_consensus.cc:515] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.795425 10077 leader_election.cc:290] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:58.797627  1661 raft_consensus.cc:3055] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 13:04:58.798206  1661 raft_consensus.cc:740] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:04:58.799202  1661 consensus_queue.cc:260] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:58.800424  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.800678  1661 raft_consensus.cc:3060] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:58.800932  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.801672 10174 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:04:58.802417 10174 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:58.804426  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "337669b46a6b4f75a35716542d1e20ae" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:58.805063  1740 raft_consensus.cc:3060] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:04:58.806079  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.816772  1557 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:58.816978  1740 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 27.
W20260709 13:04:58.818599  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50027.5us
I20260709 13:04:58.822474  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "278074adc39c4830a2791129a7815557" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.828001  1665 raft_consensus.cc:2468] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 27.
I20260709 13:04:58.830157  1735 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:04:58.833678  2315 raft_consensus.cc:2368] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23.
I20260709 13:04:58.828464  2281 leader_election.cc:304] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:58.837433  1662 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 13:04:58.837874  1662 pending_rounds.cc:85] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Aborting all ops after (but not including) 14
I20260709 13:04:58.838126  1662 pending_rounds.cc:107] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 13:04:58.838429  1662 raft_consensus.cc:2889] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:04:58.837100  1632 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:58.829583 10037 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.844944  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93491efa22a449f3aa27354cf02d7047" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:58.846279 10037 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:58.850080 10164 raft_consensus.cc:2804] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:04:58.850674 10164 raft_consensus.cc:697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:58.851569 10164 consensus_queue.cc:237] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:58.853648  9833 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:04:58.854472  1706 leader_election.cc:400] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23.
I20260709 13:04:58.854803  1706 leader_election.cc:403] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 13:04:58.855789 10157 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:58.836880 10139 consensus_queue.cc:1059] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.022s
W20260709 13:04:58.858614  1634 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.859411 10139 raft_consensus.cc:3055] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Stepping down as leader of term 24
I20260709 13:04:58.859771 10139 raft_consensus.cc:740] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:58.860967   431 tablet_replica.cc:333] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
W20260709 13:04:58.863942  1634 leader_election.cc:336] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:58.865854  1661 raft_consensus.cc:1275] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:04:58.827907 10021 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:58.864421  1634 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:04:58.874234 10128 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 13:04:58.874944   431 raft_consensus.cc:2243] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 LEADER]: Raft consensus shutting down.
W20260709 13:04:58.883436 10162 log.cc:927] Time spent T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 13:04:58.895366 10119 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
I20260709 13:04:58.837069 10134 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:58.884961 10139 consensus_queue.cc:260] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:58.986843 10139 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:58.901584 10181 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
W20260709 13:04:58.999966  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 135982us
W20260709 13:04:59.001758  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 131769us
I20260709 13:04:59.003175  1632 leader_election.cc:400] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27.
I20260709 13:04:59.003561  1632 leader_election.cc:403] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:04:58.997924  1479 maintenance_manager.cc:643] P e6b16382849d482ba544fc5ee58e0b0a: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 2.720s	user 0.108s	sys 0.048s Metrics: {"cfile_cache_miss":8,"cfile_cache_miss_bytes":25083,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":118948,"dirs.run_cpu_time_us":844,"dirs.run_wall_time_us":50747,"lbm_read_time_us":37511,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":22,"lbm_write_time_us":88031,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":8,"peak_mem_usage":252210,"reinsert_count":0,"thread_start_us":228489,"threads_started":2,"update_count":149}
I20260709 13:04:59.007411  1480 maintenance_manager.cc:419] P e6b16382849d482ba544fc5ee58e0b0a: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260709 13:04:59.008661  1479 maintenance_manager.cc:643] P e6b16382849d482ba544fc5ee58e0b0a: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260709 13:04:59.009697 10127 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:04:59.013979  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 70551.9us
I20260709 13:04:59.016419 10048 consensus_queue.cc:1048] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 13:04:59.018271 10048 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:59.018818 10048 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.021023 10048 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:04:59.023105  1558 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 13:04:59.023659 10134 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.024487 10134 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.025776 10157 raft_consensus.cc:2749] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate df6a18af06d543b2931837f5e6bab95f for earlier term 22. Current term is 23.
I20260709 13:04:59.027617  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.028558  1660 raft_consensus.cc:2393] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 22: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:04:59.030005 10134 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.033453 10057 leader_election.cc:290] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.038996  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:59.039861  1660 raft_consensus.cc:3060] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:59.040726  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:59.042770  1634 leader_election.cc:336] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.043354  1634 leader_election.cc:304] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:59.045033  9259 raft_consensus.cc:2749] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:04:58.999810  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 96918.8us
W20260709 13:04:59.053134  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 50116.1us
I20260709 13:04:59.053797  9259 raft_consensus.cc:2749] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615) with higher term. Message: Invalid argument: T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 for earlier term 26. Current term is 27.
I20260709 13:04:59.057544  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:59.058286  1737 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:04:59.059187  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:59.059824  2311 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
W20260709 13:04:59.054805  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 81797.1us
I20260709 13:04:59.062979  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d857fff084d3451d99a39db56ccd9a37" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:59.068931  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.069905  2315 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:59.074352  1557 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.078377  1737 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:59.093425  2311 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:04:59.095381  1736 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 13:04:59.095662  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.096355  1742 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 21.
I20260709 13:04:59.066126 10021 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:59.105815  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 108744us
W20260709 13:04:59.106818  1708 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.107331  1708 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.116693  1742 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 13:04:59.130905  1557 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.091102 10129 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:59.143471 10146 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:59.158995  9833 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:59.159825  9833 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:04:59.156352  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 49556.8us
W20260709 13:04:59.164388  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.164969  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.175112 10129 raft_consensus.cc:493] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.175827 10129 raft_consensus.cc:515] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:59.179654  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.180222  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.118935 10021 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.183154 10021 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.184362 10181 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 13:04:59.184859 10181 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:59.185076  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:59.185233 10181 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:59.185774  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.184362 10057 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:59.190359  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 46532.4us
W20260709 13:04:59.194381  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 157733us
I20260709 13:04:59.199059  1660 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:59.200489  2280 leader_election.cc:304] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:04:59.204092  1708 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.136291 10148 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 13:04:59.213778  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.215288  1736 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:59.203541 10043 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 13:04:59.209650 10167 raft_consensus.cc:2804] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:04:59.228219  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.230376  1632 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:59.235545  9833 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.238535 10181 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.244491  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.278088  1666 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:04:59.209306 10134 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.277132 10181 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.254103   431 raft_consensus.cc:2272] T e223adb74e71405ea633f364ffa8b6db P 316f3bf18ed94e32b238130a6bfedf1a [term 15 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.255250 10132 raft_consensus.cc:493] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:59.295094  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 45324us
I20260709 13:04:59.202605 10182 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 13:04:59.260998  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:59.299865  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:59.248566  1708 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.300577  1736 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Advancing to term 22
I20260709 13:04:59.251433  1738 raft_consensus.cc:2468] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 23.
I20260709 13:04:59.301222  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.301965  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c89f1823e773445c99646fed221320df" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.304018  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:59.304683  1661 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 22
I20260709 13:04:59.305804  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:04:59.306516  2315 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Advancing to term 22
W20260709 13:04:59.308809  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 80581.1us
W20260709 13:04:59.310086  1708 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.310580  1708 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:59.320710  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:59.324020  1708 leader_election.cc:336] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.324536  1708 leader_election.cc:304] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
W20260709 13:04:59.325664  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.326052  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.279897 10129 leader_election.cc:290] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.330099  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.305886 10167 raft_consensus.cc:697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:04:59.308001 10143 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:04:59.339496  1632 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:59.342470  1661 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:59.307605  9957 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:04:59.360432  1736 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:59.315557 10164 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:59.380517  1556 leader_election.cc:304] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.303268 10134 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:04:59.362879  1708 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.394586  1708 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.361770 10143 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:04:59.364641 10174 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:59.369654 10146 raft_consensus.cc:2749] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:59.377391 10157 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:59.400435  1556 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:04:59.340003 10132 raft_consensus.cc:515] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.370913 10171 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 13:04:59.377120 10120 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:04:59.388494  9957 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
W20260709 13:04:59.419023  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Remote error: Service unavailable: service kudu.tserver.TabletServerAdminService not registered on TabletServer
I20260709 13:04:59.390395 10164 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.398424  9879 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:59.415131 10143 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:59.481262 10134 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.423444 10199 raft_consensus.cc:2749] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:59.444708 10167 consensus_queue.cc:237] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.489145 10132 leader_election.cc:290] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.434270 10129 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.424561   431 tablet_replica.cc:333] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.491926  1660 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 13:04:59.494781  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78011us
I20260709 13:04:59.417646  2315 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:59.490067 10129 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.503947 10129 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.507599 10057 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:04:59.508544   431 raft_consensus.cc:2243] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.509397   431 pending_rounds.cc:70] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 2 pending ops.
I20260709 13:04:59.509783   431 pending_rounds.cc:77] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 21 index: 12 } timestamp: 7305634914959253504 op_type: NO_OP noop_request { }
I20260709 13:04:59.510249   431 raft_consensus.cc:2889] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.510612   431 pending_rounds.cc:77] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 19 index: 11 } timestamp: 7305634865934516224 op_type: NO_OP noop_request { }
I20260709 13:04:59.510995   431 raft_consensus.cc:2889] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.511408   431 raft_consensus.cc:2272] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:59.512564  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58407.1us
I20260709 13:04:59.515583  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.443483 10197 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader election won for term 22
I20260709 13:04:59.528710  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.535028  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.538163  1734 raft_consensus.cc:1275] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 13:04:59.514074 10132 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:59.541865  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:59.555955  1634 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.489919  9957 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.570297   431 tablet_replica.cc:333] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.571728   431 raft_consensus.cc:2243] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.572350   431 pending_rounds.cc:70] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:04:59.572630   431 pending_rounds.cc:77] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 16 } timestamp: 7305634951065583616 op_type: NO_OP noop_request { }
I20260709 13:04:59.572965   431 raft_consensus.cc:2889] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.573272   431 raft_consensus.cc:2272] T ce1eb834927c4d3caa029344d6f69de5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.579206 10021 raft_consensus.cc:493] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.580032 10021 raft_consensus.cc:515] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.582235   431 tablet_replica.cc:333] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.584143   431 raft_consensus.cc:2243] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.585170   431 pending_rounds.cc:70] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 2 pending ops.
I20260709 13:04:59.585561   431 pending_rounds.cc:77] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7305634954359541760 op_type: NO_OP noop_request { }
I20260709 13:04:59.586318   431 raft_consensus.cc:2889] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.587121   431 pending_rounds.cc:77] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 22 index: 14 } timestamp: 7305634897410347008 op_type: NO_OP noop_request { }
I20260709 13:04:59.587968   431 raft_consensus.cc:2889] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.588011 10164 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.589313   431 raft_consensus.cc:2272] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.592195 10143 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.592918 10120 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:04:59.593613 10120 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.596757 10120 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.601259   431 tablet_replica.cc:333] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.604252   431 raft_consensus.cc:2243] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.606786   431 raft_consensus.cc:2272] T 9a860695ab4c493b83dfcd08d8bee8de P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.547952 10197 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:59.611963 10197 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.544826 10057 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 13:04:59.614959   431 tablet_replica.cc:333] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.610407  1659 raft_consensus.cc:1275] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 13:04:59.560940  9656 raft_consensus.cc:493] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.630419  9656 raft_consensus.cc:515] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.631476 10174 consensus_queue.cc:1048] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 13:04:59.632714  9656 leader_election.cc:290] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.558234 10132 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.618101 10202 consensus_queue.cc:1048] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 13:04:59.622829  1666 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 13:04:59.634505  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75077.7us
I20260709 13:04:59.635174 10132 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.635715 10132 raft_consensus.cc:493] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.636152 10132 raft_consensus.cc:515] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.641112 10132 leader_election.cc:290] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.654834  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:59.634550  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 77219.2us
I20260709 13:04:59.659870   431 raft_consensus.cc:2243] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.663343  1738 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.636528 10140 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
W20260709 13:04:59.603116  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:59.670545  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 67436.3us
I20260709 13:04:59.678648  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "873c3a32540d4c2cb6d882298cb1eedc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.678808  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.679539  2311 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:59.678619 10021 leader_election.cc:290] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.665695  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.697644  1734 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 23: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
I20260709 13:04:59.683840 10143 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.685319  2311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.689868  1557 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704; no voters: 
I20260709 13:04:59.713594  1557 leader_election.cc:304] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:59.691421   431 pending_rounds.cc:70] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:04:59.714557   431 pending_rounds.cc:77] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 21 index: 15 } timestamp: 7305634831631601664 op_type: NO_OP noop_request { }
I20260709 13:04:59.715044   431 raft_consensus.cc:2889] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:04:59.715442   431 raft_consensus.cc:2272] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 13:04:59.695093  2283 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.718017  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:59.718482 10139 raft_consensus.cc:2749] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:04:59.718914 10202 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:04:59.722139  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:04:59.723924  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:59.725375  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.725982   431 tablet_replica.cc:333] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.726075  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:04:59.727957  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 182577us
W20260709 13:04:59.728206  1708 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.728832  1708 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.721175 10143 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 13:04:59.735955  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:04:59.743762  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:59.745020  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.745069  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.734679 10140 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:04:59.746680  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.748684  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7dc3659523cf47fa8918889f5129c8ba" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.757884  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "219129ac643e4650b611a85f565f5990" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.758194 10139 raft_consensus.cc:493] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.758762 10139 raft_consensus.cc:515] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.760833 10139 leader_election.cc:290] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.764353  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:04:59.766328  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a57c404b2c145bf95efa4a0649c0e33" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.742996 10174 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 13:04:59.745363 10143 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:04:59.768553  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:59.781849  1666 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 13:04:59.789422  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.789393  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.790086  1708 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.799786  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.806419  1662 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 22.
I20260709 13:04:59.808764  1736 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:59.809154  1557 leader_election.cc:304] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:04:59.810511  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0621716cd2fa4c0cb66038edb4352f37" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:04:59.817294  1556 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45706.9us
I20260709 13:04:59.806710 10132 raft_consensus.cc:493] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.820165 10132 raft_consensus.cc:515] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.817288 10182 raft_consensus.cc:2749] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:04:59.823195 10132 leader_election.cc:290] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:04:59.819643  1484 catalog_manager.cc:5697] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 7 to 10, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 10 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:04:59.826841  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:04:59.830332   431 raft_consensus.cc:2243] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.831092   431 raft_consensus.cc:2272] T 88e78eac7b5f4006a15379f62bc8c536 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.792253 10143 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:59.847568  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:04:59.848430  1662 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:04:59.789390  9656 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:04:59.843302  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.844363  1736 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
W20260709 13:04:59.846122  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 58503.4us
W20260709 13:04:59.861441  1708 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:04:59.862409  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.793442 10139 raft_consensus.cc:493] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:04:59.862815  1708 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.866458  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8e75da90507436ea52296522f44c15e" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.869740  1556 leader_election.cc:304] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.870635  1661 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "24ec87af7f274132adaaf5e59ef12b7b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.869058   431 tablet_replica.cc:333] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.873011   431 raft_consensus.cc:2243] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.873754   431 raft_consensus.cc:2272] T 5ae371f2c53648c2992a835477d85184 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.870548 10156 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 13:04:59.858409 10166 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 13:04:59.877133   431 tablet_replica.cc:333] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:04:59.868505  9656 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.884004  9656 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:04:59.889231  1662 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:04:59.885772  1557 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:04:59.892143  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.884645 10139 raft_consensus.cc:515] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:04:59.894649 10139 leader_election.cc:290] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:04:59.898057  1556 raft_consensus.cc:2686] T 7dc3659523cf47fa8918889f5129c8ba P 316f3bf18ed94e32b238130a6bfedf1a: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260709 13:04:59.891623  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:04:59.900625  1634 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:04:59.903577 10181 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader election won for term 25
W20260709 13:04:59.906847  1482 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 257) took 56120 ms (client timeout 15000 ms). Trace:
I20260709 13:04:59.906658 10181 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:04:59.908465 10181 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:04:59.906728  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.914726  1556 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.914568  1484 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 7 to 14, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 14 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:59.915781 10132 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:04:59.916656 10132 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:59.921458  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.923329  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:04:59.932945  1666 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:04:59.934417 10167 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 13:04:59.907437  1482 rpcz_store.cc:269] 0709 13:04:03.786433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.786799 (+   366us) service_pool.cc:224] Handling call
0709 13:04:59.906737 (+56119938us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:04:57.753645 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:04:57.754402 (+   757us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:04:57.754468 (+    66us) write_op.cc:435] Acquired schema lock
0709 13:04:57.754493 (+    25us) tablet.cc:662] Decoding operations
0709 13:04:57.755793 (+  1300us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:04:57.755908 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 13:04:57.755943 (+    35us) tablet.cc:685] Acquiring locks for 11 operations
0709 13:04:57.793172 (+ 37229us) tablet.cc:701] Row locks acquired
0709 13:04:57.793244 (+    72us) write_op.cc:260] PREPARE: finished
0709 13:04:57.793444 (+   200us) write_op.cc:270] Start()
0709 13:04:57.793640 (+   196us) write_op.cc:276] Timestamp: P: 1783602297793369 usec, L: 0
0709 13:04:57.793677 (+    37us) op_driver.cc:348] REPLICATION: starting
0709 13:04:57.794543 (+   866us) log.cc:844] Serialized 4418 byte log entry
0709 13:04:58.500984 (+706441us) op_driver.cc:464] REPLICATION: finished
0709 13:04:58.677600 (+176616us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b18003d2100. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef03f3 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:04:58.677805 (+   205us) write_op.cc:301] APPLY: starting
0709 13:04:58.677924 (+   119us) tablet.cc:1367] starting BulkCheckPresence
0709 13:04:58.792611 (+114687us) tablet.cc:1370] finished BulkCheckPresence
0709 13:04:58.792643 (+    32us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:04:58.960632 (+167989us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:04:58.972065 (+ 11433us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=0,mrs_lookups=0
0709 13:04:58.972138 (+    73us) write_op.cc:312] APPLY: finished
0709 13:04:58.972655 (+   517us) log.cc:844] Serialized 105 byte log entry
0709 13:04:58.972882 (+   227us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:58.973038 (+   156us) write_op.cc:454] Released schema lock
0709 13:04:58.973112 (+    74us) write_op.cc:341] FINISH: Updating metrics
0709 13:04:58.973670 (+   558us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:04:58.973701 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":28954,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":152263,"apply.run_cpu_time_us":19843,"apply.run_wall_time_us":295815,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50129,"cfile_init":1,"lbm_read_time_us":457,"lbm_reads_lt_1ms":4,"num_ops":11,"prepare.queue_time_us":79030,"prepare.run_cpu_time_us":16321,"prepare.run_wall_time_us":321864,"raft.queue_time_us":207893,"raft.run_cpu_time_us":62146,"raft.run_wall_time_us":156592,"replication_time_us":707121,"spinlock_wait_cycles":50886144,"thread_start_us":532378,"threads_started":4,"wal-append.queue_time_us":288162}]]}
I20260709 13:04:59.937894   431 raft_consensus.cc:2243] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:04:59.938868  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:04:59.941957  1556 leader_election.cc:304] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.942818 10127 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:04:59.947050  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2dffe36aa52848e8962262b0e01bc346" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.946499  1556 leader_election.cc:304] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.948796  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e165279fd0cb4e84a8c9e244d087f8db" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:04:59.949693  1556 leader_election.cc:304] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.949970 10132 raft_consensus.cc:2749] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:04:59.960692  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:04:59.962764  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:04:59.945791 10129 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 13:04:59.917110  1484 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 5 to 13, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 13 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:04:59.971580  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:04:59.973963   431 raft_consensus.cc:2272] T ce79fa9e7595481c8379fd2bdeca8a6e P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:04:59.976835  1556 leader_election.cc:304] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.982028  9656 raft_consensus.cc:2749] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:04:59.979624  1556 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:04:59.983202 10132 raft_consensus.cc:2749] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:04:59.953811 10148 raft_consensus.cc:2749] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:04:59.973074  1484 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 8 to 11, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 11 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:04:59.998723 10127 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:00.012023 10148 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:00.014012  1736 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:05:00.027019  9957 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 13:05:00.041342 10129 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.042029 10129 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:00.052316  1736 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:00.054764  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:00.057284  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.072131  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.058954 10181 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.074021   431 tablet_replica.cc:333] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:00.073875 10181 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.075549   431 raft_consensus.cc:2243] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:00.078226  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.084694  1734 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:05:00.082962   431 pending_rounds.cc:70] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:00.076604 10181 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.085997   431 pending_rounds.cc:77] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 23 index: 16 } timestamp: 7305634953444605952 op_type: NO_OP noop_request { }
I20260709 13:05:00.085894  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:05:00.086741   431 raft_consensus.cc:2889] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:00.095407   431 raft_consensus.cc:2272] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:00.097551 10199 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:05:00.097951 10199 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:00.098268 10199 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:00.113583  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.114491  1668 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:05:00.051661 10143 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:00.110999 10148 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:05:00.111438  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:00.122526  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:00.123994  1668 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:05:00.069566  1484 catalog_manager.cc:5697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 2 to 10. New cstate: current_term: 10 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:00.132416  1484 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 7 to 9, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 9 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:00.129251   431 tablet_replica.cc:333] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
W20260709 13:05:00.122715 10219 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 13:05:00.144786  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:00.150065  2309 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:05:00.155032  1668 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
I20260709 13:05:00.141592 10148 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.163357 10148 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.159780 10182 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 13:05:00.167908 10182 raft_consensus.cc:493] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.148303 10129 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:00.174026   431 raft_consensus.cc:2243] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:00.153951 10132 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.178377  1668 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:05:00.178620  1740 raft_consensus.cc:1275] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:05:00.176592  1556 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:00.184098  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.184931  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:05:00.186036  2315 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:05:00.200140  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb611776ed864c47b9f843e0746b3683" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.178524 10143 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:00.203979  1557 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.197652 10021 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 13:05:00.200364 10202 consensus_queue.cc:1048] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.011s
I20260709 13:05:00.213959  2318 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 13:05:00.214157 10139 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.214709 10139 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.185038 10197 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader election won for term 23
I20260709 13:05:00.219167  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:05:00.252337  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.251236  9259 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:05:00.201256 10148 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.185670 10132 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.283728  2315 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 23.
W20260709 13:05:00.267817 10224 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 13:05:00.257332 10197 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:05:00.300226   431 raft_consensus.cc:2272] T e7f90c9079e24cf2a4c9ab3dd55172ea P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:00.264858  9656 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.299299 10139 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:00.306769  1557 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102854us
I20260709 13:05:00.277846 10228 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:00.168630 10182 raft_consensus.cc:515] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.253296 10199 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.336333 10173 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:05:00.363387  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.351351 10223 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 13:05:00.310993  9656 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:00.402313 10226 log.cc:927] Time spent T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 13:05:00.403421 10225 log.cc:927] Time spent T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 13:05:00.413807 10132 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:00.463140  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 47531.6us
I20260709 13:05:00.319262 10197 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:00.471915  1738 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 13:05:00.473091  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9768bad280eb4cdf8f1a090752fc5ddc" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.379654 10173 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.497920  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.499946  1557 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.500597 10139 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 13:05:00.502197 10231 log.cc:927] Time spent T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 13:05:00.505792 10173 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:00.515506  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.526058  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8029bb31c40d4c01bebc7c65c2301a85" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.562342  9656 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.561600  1556 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.590355 10197 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:00.593694  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.595340   431 tablet_replica.cc:333] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:00.596786   431 raft_consensus.cc:2243] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:00.598943 10199 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.603173 10182 leader_election.cc:290] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.565749 10127 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:00.610175 10127 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.517323 10143 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 13:05:00.614493   431 raft_consensus.cc:2272] T dfe67343bf1842888d6176ed2ffe1419 P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:00.580945 10212 log.cc:927] Time spent T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
W20260709 13:05:00.621707  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:00.622246  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 79786us
I20260709 13:05:00.615454 10127 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:00.633858  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 86204.3us
I20260709 13:05:00.642189  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:00.642899  1737 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:00.645022  1741 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.646976  1742 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.648317  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.649075  1735 raft_consensus.cc:2393] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 25: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:05:00.615677  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 50130.9us
I20260709 13:05:00.667574  1737 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:05:00.693454  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.694264  1739 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 15.
I20260709 13:05:00.724135  1668 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:00.725188  1668 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:00.715939  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:00.716988  1557 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.727777  9259 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 13:05:00.728197  9259 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:00.728540  9259 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:05:00.730888  1634 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:00.732805  1734 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 13:05:00.733789  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "87a64e3accd242188d2126865daef114" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.735077  1667 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 13:05:00.731410  1634 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:05:00.740272  1556 leader_election.cc:304] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:00.740065  1557 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:05:00.743963 10021 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 13:05:00.744238 10182 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:00.753779 10182 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:00.755321   431 tablet_replica.cc:333] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:00.756458   431 raft_consensus.cc:2243] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:00.762703 10021 raft_consensus.cc:2749] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:00.763106   431 raft_consensus.cc:2272] T 648ab3b5826e49fe81c3363d752d8af8 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:00.767520  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:00.768427  1737 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:05:00.785566  1660 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:00.756532 10182 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.761144 10127 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:05:00.800339  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.779153 10227 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:05:00.810173  1668 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 24.
I20260709 13:05:00.791326 10129 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 13:05:00.806547  1557 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f; no voters: 
I20260709 13:05:00.813299  9259 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:00.804641  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:00.817075  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.840973  9259 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:00.821775 10208 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 13:05:00.818671 10228 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:05:00.835817 10227 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:05:00.853662 10228 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:00.864813  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e223adb74e71405ea633f364ffa8b6db" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:00.865579  1737 raft_consensus.cc:3060] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 15 FOLLOWER]: Advancing to term 16
W20260709 13:05:00.870729  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.876539 10127 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:00.877566  1660 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 25.
I20260709 13:05:00.865193 10228 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:05:00.892644  1737 raft_consensus.cc:2468] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 16.
W20260709 13:05:00.881742  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.906782  1632 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:00.919034  1667 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:05:00.923105   431 tablet_replica.cc:333] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:00.924266   431 raft_consensus.cc:2243] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:00.925045   431 raft_consensus.cc:2272] T 93491efa22a449f3aa27354cf02d7047 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:00.926124 10227 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:00.912595  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.934638 10127 raft_consensus.cc:2804] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 13:05:00.935253 10127 raft_consensus.cc:697] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:05:00.936159 10127 consensus_queue.cc:237] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:00.971418  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:00.984872   431 tablet_replica.cc:333] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:00.986251   431 raft_consensus.cc:2243] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Raft consensus shutting down.
W20260709 13:05:01.006114  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:01.007563  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:00.957204 10143 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 13:05:01.021023 10228 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:01.027195 10228 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:01.014797   431 raft_consensus.cc:2272] T 200c02cbe43c4af8b4c59f21f210a9bc P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.040438  2310 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 13:05:01.042861   431 tablet_replica.cc:333] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.044001   431 raft_consensus.cc:2243] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.044682   431 raft_consensus.cc:2272] T 92fb459de10a4e038d6cbee7b88dc3ba P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.047502 10132 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:05:01.053051  1659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:01.053941  1659 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:05:01.014346  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:01.060875  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 46563.5us
I20260709 13:05:01.084980  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:01.085721  1740 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:05:01.096724  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:01.094357  1659 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 26.
I20260709 13:05:01.087596 10132 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:01.111070   431 tablet_replica.cc:333] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.138765  1556 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:01.145107 10228 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:05:01.145821 10228 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 LEADER]: Becoming Leader. State: Replica: 316f3bf18ed94e32b238130a6bfedf1a, State: Running, Role: LEADER
I20260709 13:05:01.152362  1740 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316f3bf18ed94e32b238130a6bfedf1a in term 26.
I20260709 13:05:01.141307 10132 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:01.164898 10132 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:01.168231 10228 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:01.180429  1737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:01.189394  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
W20260709 13:05:01.181072  1634 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:01.234212  1556 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:01.237493   431 raft_consensus.cc:2243] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus shutting down.
W20260709 13:05:01.237600  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:01.241140  1634 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:05:01.242359  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:01.243785 10228 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:01.246694 10257 log.cc:927] Time spent T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 13:05:01.246974   431 raft_consensus.cc:2272] T 2dffe36aa52848e8962262b0e01bc346 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.252288   431 tablet_replica.cc:333] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.253445   431 raft_consensus.cc:2243] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.254659   431 raft_consensus.cc:2272] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.261765   431 tablet_replica.cc:333] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.263074   431 raft_consensus.cc:2243] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.265952   431 pending_rounds.cc:70] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.266305   431 pending_rounds.cc:77] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 11 } timestamp: 7305634866212069376 op_type: NO_OP noop_request { }
I20260709 13:05:01.266789   431 raft_consensus.cc:2889] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.267158   431 raft_consensus.cc:2272] T 4e74a6fd31c14c17a7248fe6eb410c77 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.270270   431 tablet_replica.cc:333] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.271271   431 raft_consensus.cc:2243] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.272362   431 pending_rounds.cc:70] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.272758   431 pending_rounds.cc:77] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 26 index: 14 } timestamp: 7305635025630085120 op_type: NO_OP noop_request { }
I20260709 13:05:01.273190   431 raft_consensus.cc:2889] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.273685   431 raft_consensus.cc:2272] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.289345 10139 raft_consensus.cc:493] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:01.290015 10139 raft_consensus.cc:515] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:01.292284 10262 consensus_queue.cc:1175] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "316f3bf18ed94e32b238130a6bfedf1a" status { last_received { term: 26 index: 14 } last_committed_idx: 13 last_received_current_leader { term: 26 index: 14 } }
I20260709 13:05:01.293542  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:01.303593 10139 leader_election.cc:290] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:01.304812  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c15d2efd3c74bc9a5e8d3a23d50cced" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:01.306891  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:01.307076  1557 leader_election.cc:304] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:01.308383 10139 raft_consensus.cc:2749] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:01.323669   431 tablet_replica.cc:333] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.324831   431 raft_consensus.cc:2243] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.325663   431 raft_consensus.cc:2272] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.329329   431 tablet_replica.cc:333] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.362159   431 raft_consensus.cc:2243] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.363042   431 raft_consensus.cc:2272] T 0621716cd2fa4c0cb66038edb4352f37 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.366322   431 tablet_replica.cc:333] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.369812   431 raft_consensus.cc:2243] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.385830  1667 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:05:01.388171 10228 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 13:05:01.397886  1739 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Refusing update from remote peer 316f3bf18ed94e32b238130a6bfedf1a: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 13:05:01.402614 10228 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:05:01.409107   431 pending_rounds.cc:70] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 2 pending ops.
I20260709 13:05:01.411218   431 pending_rounds.cc:77] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 12 } timestamp: 7305634964985032704 op_type: NO_OP noop_request { }
I20260709 13:05:01.411701   431 raft_consensus.cc:2889] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.412041   431 pending_rounds.cc:77] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 19 index: 11 } timestamp: 7305634910576779264 op_type: NO_OP noop_request { }
I20260709 13:05:01.412415   431 raft_consensus.cc:2889] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.422487   431 raft_consensus.cc:2272] T 8029bb31c40d4c01bebc7c65c2301a85 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.429536   431 tablet_replica.cc:333] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.430716   431 raft_consensus.cc:2243] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.431910   431 raft_consensus.cc:2272] T 7f9cc5e1ffa141e295028cdd523da631 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.443897   431 tablet_replica.cc:333] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.444998   431 raft_consensus.cc:2243] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.449332   431 raft_consensus.cc:2272] T 3d40d5b254644257a6d39df949366e1b P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.464165   431 tablet_replica.cc:333] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.465176   431 raft_consensus.cc:2243] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.466434   431 raft_consensus.cc:2272] T eed985f1202940a1ab59ea7b7b3c745a P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.492897   431 tablet_replica.cc:333] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.491777  1485 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 11 to 12, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 12 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:01.493954   431 raft_consensus.cc:2243] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.494670   431 raft_consensus.cc:2272] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:01.496843  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:01.502125   431 tablet_replica.cc:333] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.500841 10228 raft_consensus.cc:493] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:01.503423   431 raft_consensus.cc:2243] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.503142 10228 raft_consensus.cc:515] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:01.506064 10228 leader_election.cc:290] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:01.507172   431 raft_consensus.cc:2272] T bf77b84dc20e4118ad47baa1808a85d9 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.508113  1739 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:01.514945  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f303f4bc5e84b46b9bca2a44911578b" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:01.516850  1556 leader_election.cc:304] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:01.517966 10228 raft_consensus.cc:2749] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:05:01.522775  1485 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 9, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 9 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:01.527736   431 tablet_replica.cc:333] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.528847   431 raft_consensus.cc:2243] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.529595   431 pending_rounds.cc:70] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.530280   431 pending_rounds.cc:77] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 11 } timestamp: 7305634900558528512 op_type: NO_OP noop_request { }
I20260709 13:05:01.530675   431 raft_consensus.cc:2889] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.531054   431 raft_consensus.cc:2272] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.546975   431 tablet_replica.cc:333] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.548015   431 raft_consensus.cc:2243] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.554415   431 pending_rounds.cc:70] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.554756   431 pending_rounds.cc:77] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 15 } timestamp: 7305634972130742272 op_type: NO_OP noop_request { }
I20260709 13:05:01.555163   431 raft_consensus.cc:2889] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.555547   431 raft_consensus.cc:2272] T 873c3a32540d4c2cb6d882298cb1eedc P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:01.557767  1484 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 229) took 112541 ms (client timeout 15000 ms). Trace:
I20260709 13:05:01.561859   431 tablet_replica.cc:333] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.563002   431 raft_consensus.cc:2243] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.567319   431 raft_consensus.cc:2272] T 24ec87af7f274132adaaf5e59ef12b7b P 316f3bf18ed94e32b238130a6bfedf1a [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.571491   431 tablet_replica.cc:333] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.572609   431 raft_consensus.cc:2243] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.573369   431 raft_consensus.cc:2272] T 7a5331c1cc5e4cbf8847232f0d5835c7 P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.578223   431 tablet_replica.cc:333] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
W20260709 13:05:01.578755  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:01.579039   431 raft_consensus.cc:2243] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.580754   431 raft_consensus.cc:2272] T 162373eeadc2444e87ffd67df6a836e5 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:01.558362  1484 rpcz_store.cc:269] 0709 13:03:09.016889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:09.017115 (+   226us) service_pool.cc:224] Handling call
0709 13:05:01.557663 (+112540548us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:00.685644 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:00.686643 (+   999us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:00.686701 (+    58us) write_op.cc:435] Acquired schema lock
0709 13:05:00.686733 (+    32us) tablet.cc:662] Decoding operations
0709 13:05:00.688024 (+  1291us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:00.743719 (+ 55695us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:00.746252 (+  2533us) tablet.cc:685] Acquiring locks for 6 operations
0709 13:05:00.825659 (+ 79407us) tablet.cc:701] Row locks acquired
0709 13:05:00.825683 (+    24us) write_op.cc:260] PREPARE: finished
0709 13:05:00.842199 (+ 16516us) write_op.cc:270] Start()
0709 13:05:00.851896 (+  9697us) write_op.cc:276] Timestamp: P: 1783602300838260 usec, L: 0
0709 13:05:00.853386 (+  1490us) op_driver.cc:348] REPLICATION: starting
0709 13:05:00.892845 (+ 39459us) log.cc:844] Serialized 2482 byte log entry
0709 13:05:01.028497 (+135652us) spinlock_profiling.cc:243] Waited 6.86 ms on lock 0x7b180043b280. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:05:01.064895 (+ 36398us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b54002d2ba0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedf52dbe 00007fcdedf5d930 00007fcdedf5d8d8 00007fcdedf5d858 00007fcdedf5d813 00007fcdedf5ca9c 00007fcdf3e48626 00007fcdf3e42dd8 00007fcdedf997cb 00007fcdedf9c2c0
0709 13:05:01.163079 (+ 98184us) op_driver.cc:464] REPLICATION: finished
0709 13:05:01.240836 (+ 77757us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf46571 00007fcdedfda32a 00007fcdedf5ff14 00007fcdedf5fec3 00007fcdedf5fe38 00007fcdedf5fdf3 00007fcdedf5f0dc 00007fcdedf88e16 00007fcdedf70cbb 00007fcdedf5b39d 00007fcdedf5ed75
0709 13:05:01.309237 (+ 68401us) write_op.cc:301] APPLY: starting
0709 13:05:01.309341 (+   104us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:01.320518 (+ 11177us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:01.320542 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:01.335705 (+ 15163us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:01.340334 (+  4629us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 13:05:01.340405 (+    71us) write_op.cc:312] APPLY: finished
0709 13:05:01.340695 (+   290us) log.cc:844] Serialized 65 byte log entry
0709 13:05:01.341684 (+   989us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.341822 (+   138us) write_op.cc:454] Released schema lock
0709 13:05:01.341868 (+    46us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:01.367040 (+ 25172us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.367074 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":134964,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":136256,"apply.run_cpu_time_us":9506,"apply.run_wall_time_us":57678,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29219,"num_ops":6,"prepare.queue_time_us":30288,"prepare.run_cpu_time_us":25508,"prepare.run_wall_time_us":290506,"raft.queue_time_us":78356,"raft.run_cpu_time_us":17329,"raft.run_wall_time_us":97212,"replication_time_us":301780,"spinlock_wait_cycles":41243520,"thread_start_us":241383,"threads_started":4,"wal-append.queue_time_us":73121}]]}
I20260709 13:05:01.594821   431 tablet_replica.cc:333] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.595849   431 raft_consensus.cc:2243] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.596974   431 raft_consensus.cc:2272] T 421876549c8f408587a97543338bca1d P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.599881   431 tablet_replica.cc:333] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.600780   431 raft_consensus.cc:2243] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.601403   431 raft_consensus.cc:2272] T e165279fd0cb4e84a8c9e244d087f8db P 316f3bf18ed94e32b238130a6bfedf1a [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.605036   431 tablet_replica.cc:333] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.606114   431 raft_consensus.cc:2243] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.606914   431 pending_rounds.cc:70] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.607179   431 pending_rounds.cc:77] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 21 index: 12 } timestamp: 7305634865935032320 op_type: NO_OP noop_request { }
I20260709 13:05:01.607571   431 raft_consensus.cc:2889] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.607921   431 raft_consensus.cc:2272] T d66c6d05a0074fe2a7b7b2466f6182d7 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.611645   431 tablet_replica.cc:333] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.612515   431 raft_consensus.cc:2243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.613672   431 raft_consensus.cc:2272] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.630692   431 tablet_replica.cc:333] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.631747   431 raft_consensus.cc:2243] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.632979   431 raft_consensus.cc:2272] T fbe322e4447d486680550d1490266b17 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.645665  1739 raft_consensus.cc:1275] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 13:05:01.645855  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:01.658294 10249 consensus_queue.cc:1048] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
W20260709 13:05:01.694409  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:01.708710  1487 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 16 to 21, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 21 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:01.712747  1487 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 13 to 16, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:01.715672  1487 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 13 to 21, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 21 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:01.745234   431 tablet_replica.cc:333] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.748104   431 raft_consensus.cc:2243] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.749464   431 raft_consensus.cc:2272] T 249474541a174b90a099d77784ed4801 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.768870  1487 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 13 to 16, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:01.771878  1487 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 16 to 19, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 19 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:01.792394   431 tablet_replica.cc:333] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.794706   431 raft_consensus.cc:2243] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.800334   431 raft_consensus.cc:2272] T 5a44e16fd1e74bb585aa62b3d15e7384 P 316f3bf18ed94e32b238130a6bfedf1a [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.801556  1487 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 9 to 17, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 17 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:01.806932  1487 catalog_manager.cc:5697] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 10 to 16. New cstate: current_term: 16 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:01.810993  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:01.819324   431 tablet_replica.cc:333] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.820775   431 raft_consensus.cc:2243] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.821390   431 pending_rounds.cc:70] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 2 pending ops.
I20260709 13:05:01.821673   431 pending_rounds.cc:77] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 25 index: 13 } timestamp: 7305634909039783936 op_type: NO_OP noop_request { }
I20260709 13:05:01.822037   431 raft_consensus.cc:2889] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.822355   431 pending_rounds.cc:77] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 23 index: 12 } timestamp: 7305634859666411520 op_type: NO_OP noop_request { }
I20260709 13:05:01.810829  1487 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 15 to 17, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 17 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:01.822676   431 raft_consensus.cc:2889] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.833732   431 raft_consensus.cc:2272] T 9768bad280eb4cdf8f1a090752fc5ddc P 316f3bf18ed94e32b238130a6bfedf1a [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.847553   431 tablet_replica.cc:333] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.848456   431 raft_consensus.cc:2243] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:01.849686   431 raft_consensus.cc:2272] T a9c383ed9b7c4624b8fa546326e129ff P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.862808   431 tablet_replica.cc:333] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.864797   431 raft_consensus.cc:2243] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.865849   431 raft_consensus.cc:2272] T 6c15d2efd3c74bc9a5e8d3a23d50cced P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.869479   431 tablet_replica.cc:333] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.870775 10228 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:01.871395 10228 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:01.876262  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:01.878114  1634 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 284 similar messages]
I20260709 13:05:01.879242  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:01.880882  1556 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 316f3bf18ed94e32b238130a6bfedf1a; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:01.881973 10227 raft_consensus.cc:2749] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:05:01.884297  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "316f3bf18ed94e32b238130a6bfedf1a" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:05:01.886134  1485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 238) took 117201 ms (client timeout 15000 ms). Trace:
I20260709 13:05:01.886472 10228 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:01.886685  1485 rpcz_store.cc:269] 0709 13:03:04.685082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:04.685340 (+   258us) service_pool.cc:224] Handling call
0709 13:03:05.996845 (+1311505us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb9640 00007fcdf3d90b76 00007fcdf3e9ac3e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc 00007fcdf235a392 00007fcdec31848c 00007fcdec3178c6
0709 13:05:01.886034 (+115889189us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:01.584556 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:01.585164 (+   608us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:01.585199 (+    35us) write_op.cc:435] Acquired schema lock
0709 13:05:01.585220 (+    21us) tablet.cc:662] Decoding operations
0709 13:05:01.585696 (+   476us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:01.585777 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:01.585804 (+    27us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:05:01.586146 (+   342us) tablet.cc:701] Row locks acquired
0709 13:05:01.586164 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:05:01.586347 (+   183us) write_op.cc:270] Start()
0709 13:05:01.586478 (+   131us) write_op.cc:276] Timestamp: P: 1783602301586297 usec, L: 0
0709 13:05:01.586497 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 13:05:01.588174 (+  1677us) log.cc:844] Serialized 924 byte log entry
0709 13:05:01.590214 (+  2040us) op_driver.cc:464] REPLICATION: finished
0709 13:05:01.591268 (+  1054us) write_op.cc:301] APPLY: starting
0709 13:05:01.591354 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:01.592677 (+  1323us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:01.592773 (+    96us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:01.594106 (+  1333us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:01.594301 (+   195us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 13:05:01.594358 (+    57us) write_op.cc:312] APPLY: finished
0709 13:05:01.595347 (+   989us) log.cc:844] Serialized 33 byte log entry
0709 13:05:01.595528 (+   181us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.612746 (+ 17218us) write_op.cc:454] Released schema lock
0709 13:05:01.612856 (+   110us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:01.613368 (+   512us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.613399 (+    31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3109285,"reported_tablets":52,"spinlock_wait_cycles":28083456,"child_traces":[["op",{"apply.queue_time_us":818,"apply.run_cpu_time_us":4097,"apply.run_wall_time_us":22047,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8364,"num_ops":2,"prepare.queue_time_us":17041,"prepare.run_cpu_time_us":3051,"prepare.run_wall_time_us":3951,"replication_time_us":3579,"thread_start_us":12177,"threads_started":1}]]}
I20260709 13:05:01.870433   431 raft_consensus.cc:2243] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.892295   431 pending_rounds.cc:70] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 1 pending ops.
I20260709 13:05:01.892581   431 pending_rounds.cc:77] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 23 index: 14 } timestamp: 7305634869623615488 op_type: NO_OP noop_request { }
I20260709 13:05:01.892953   431 raft_consensus.cc:2889] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.893338   431 raft_consensus.cc:2272] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.898241   431 tablet_replica.cc:333] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.913785   431 raft_consensus.cc:2243] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.914546   431 raft_consensus.cc:2272] T c89f1823e773445c99646fed221320df P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.920636   431 tablet_replica.cc:333] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.927119   431 raft_consensus.cc:2243] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.927888   431 raft_consensus.cc:2272] T bb611776ed864c47b9f843e0746b3683 P 316f3bf18ed94e32b238130a6bfedf1a [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.931085   431 tablet_replica.cc:333] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.931829   431 raft_consensus.cc:2243] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.958572   431 pending_rounds.cc:70] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Trying to abort 3 pending ops.
I20260709 13:05:01.958899   431 pending_rounds.cc:77] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 23 index: 16 } timestamp: 7305634944411373568 op_type: NO_OP noop_request { }
I20260709 13:05:01.959254   431 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.959529   431 pending_rounds.cc:77] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 22 index: 15 } timestamp: 7305634900628541440 op_type: NO_OP noop_request { }
I20260709 13:05:01.961829   431 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.962114   431 pending_rounds.cc:77] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a: Aborting op as it isn't in flight: id { term: 20 index: 14 } timestamp: 7305634847794085888 op_type: NO_OP noop_request { }
I20260709 13:05:01.962432   431 raft_consensus.cc:2889] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:05:01.962779   431 raft_consensus.cc:2272] T 5a57c404b2c145bf95efa4a0649c0e33 P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:01.972469   431 tablet_replica.cc:333] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:01.973306   431 raft_consensus.cc:2243] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:01.973946   431 raft_consensus.cc:2272] T 87a64e3accd242188d2126865daef114 P 316f3bf18ed94e32b238130a6bfedf1a [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:02.007107  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:02.009585   431 tablet_replica.cc:333] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:02.010353   431 raft_consensus.cc:2243] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:02.011114   431 raft_consensus.cc:2272] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:02.033531   431 tablet_replica.cc:333] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:02.034384   431 raft_consensus.cc:2243] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:02.034978   431 raft_consensus.cc:2272] T b8e75da90507436ea52296522f44c15e P 316f3bf18ed94e32b238130a6bfedf1a [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:02.050284  1487 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 231) took 58743 ms (client timeout 15000 ms). Trace:
I20260709 13:05:02.044276  1488 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:02.054174  1487 rpcz_store.cc:269] 0709 13:04:03.307516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:03.311940 (+  4424us) service_pool.cc:224] Handling call
0709 13:05:02.050185 (+58738245us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:01.852593 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:01.853250 (+   657us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:01.853298 (+    48us) write_op.cc:435] Acquired schema lock
0709 13:05:01.853319 (+    21us) tablet.cc:662] Decoding operations
0709 13:05:01.854352 (+  1033us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:01.854419 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:01.854456 (+    37us) tablet.cc:685] Acquiring locks for 8 operations
0709 13:05:01.855210 (+   754us) tablet.cc:701] Row locks acquired
0709 13:05:01.855229 (+    19us) write_op.cc:260] PREPARE: finished
0709 13:05:01.855398 (+   169us) write_op.cc:270] Start()
0709 13:05:01.855531 (+   133us) write_op.cc:276] Timestamp: P: 1783602301855352 usec, L: 0
0709 13:05:01.855554 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 13:05:01.856494 (+   940us) log.cc:844] Serialized 3258 byte log entry
0709 13:05:01.860907 (+  4413us) op_driver.cc:464] REPLICATION: finished
0709 13:05:01.861340 (+   433us) write_op.cc:301] APPLY: starting
0709 13:05:01.861437 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:01.865749 (+  4312us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:01.865852 (+   103us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:01.888791 (+ 22939us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:01.889034 (+   243us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 13:05:01.889079 (+    45us) write_op.cc:312] APPLY: finished
0709 13:05:01.889376 (+   297us) log.cc:844] Serialized 81 byte log entry
0709 13:05:01.891084 (+  1708us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.891240 (+   156us) write_op.cc:454] Released schema lock
0709 13:05:01.891303 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:01.894706 (+  3403us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:01.894753 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":86419,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":146,"apply.run_cpu_time_us":10818,"apply.run_wall_time_us":33320,"cfile_cache_hit":32,"cfile_cache_hit_bytes":33456,"num_ops":8,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":4135,"prepare.run_wall_time_us":4483,"replication_time_us":5184}]]}
I20260709 13:05:02.057768   431 tablet_replica.cc:333] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a: stopping tablet replica
I20260709 13:05:02.062302  1488 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 12 to 15, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 15 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.078601   431 raft_consensus.cc:2243] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:02.079685   431 raft_consensus.cc:2272] T e676e54141f7470db4551f4de439c017 P 316f3bf18ed94e32b238130a6bfedf1a [term 16 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:02.080173  1488 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 6 to 17. New cstate: current_term: 17 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:02.086053  1488 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 9 to 13, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 13 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.089483  1488 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 5 to 15. New cstate: current_term: 15 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:02.188066  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:02.257121  1488 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 230) took 87177 ms (client timeout 15000 ms). Trace:
W20260709 13:05:02.257836  1488 rpcz_store.cc:269] 0709 13:03:35.080001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:35.080187 (+   186us) service_pool.cc:224] Handling call
0709 13:05:02.257028 (+87176841us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:02.117497 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:02.118151 (+   654us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:02.118198 (+    47us) write_op.cc:435] Acquired schema lock
0709 13:05:02.118216 (+    18us) tablet.cc:662] Decoding operations
0709 13:05:02.118946 (+   730us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:02.119015 (+    69us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:02.119046 (+    31us) tablet.cc:685] Acquiring locks for 5 operations
0709 13:05:02.119665 (+   619us) tablet.cc:701] Row locks acquired
0709 13:05:02.119681 (+    16us) write_op.cc:260] PREPARE: finished
0709 13:05:02.119865 (+   184us) write_op.cc:270] Start()
0709 13:05:02.120004 (+   139us) write_op.cc:276] Timestamp: P: 1783602302119794 usec, L: 0
0709 13:05:02.120026 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:05:02.120842 (+   816us) log.cc:844] Serialized 2089 byte log entry
0709 13:05:02.125051 (+  4209us) op_driver.cc:464] REPLICATION: finished
0709 13:05:02.125651 (+   600us) write_op.cc:301] APPLY: starting
0709 13:05:02.125766 (+   115us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:02.129866 (+  4100us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:02.130007 (+   141us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:02.137346 (+  7339us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:02.182275 (+ 44929us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=0,mrs_lookups=0
0709 13:05:02.182331 (+    56us) write_op.cc:312] APPLY: finished
0709 13:05:02.182604 (+   273us) log.cc:844] Serialized 57 byte log entry
0709 13:05:02.191359 (+  8755us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:02.191504 (+   145us) write_op.cc:454] Released schema lock
0709 13:05:02.191562 (+    58us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:02.192011 (+   449us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:02.197454 (+  5443us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7845,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":306,"apply.run_cpu_time_us":9201,"apply.run_wall_time_us":66309,"cfile_cache_hit":20,"cfile_cache_hit_bytes":20910,"num_ops":5,"prepare.queue_time_us":9404,"prepare.run_cpu_time_us":3665,"prepare.run_wall_time_us":3666,"replication_time_us":4869}]]}
I20260709 13:05:02.260560  1484 catalog_manager.cc:5697] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 22, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 22 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:02.285733  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.266937  1484 catalog_manager.cc:5697] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.291448  1484 catalog_manager.cc:5697] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 21 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.293041  1484 catalog_manager.cc:5697] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 25 to 29. New cstate: current_term: 29 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.314772  1484 catalog_manager.cc:5697] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.316638  1484 catalog_manager.cc:5697] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 22, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.333653  1484 catalog_manager.cc:5697] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.338925  1484 catalog_manager.cc:5697] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 15 to 25. New cstate: current_term: 25 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.342679  1484 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 25, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 25 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.351562  1484 catalog_manager.cc:5697] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 26, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 26 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:02.370814  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.358338  1484 catalog_manager.cc:5697] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 20, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 20 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.374019  1484 catalog_manager.cc:5697] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.375695  1484 catalog_manager.cc:5697] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 24, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 24 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.378594  1484 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 18, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 18 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.381536  1484 catalog_manager.cc:5697] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.383422  1484 catalog_manager.cc:5697] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 20 to 23. New cstate: current_term: 23 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.454509 10254 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.455271 10254 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.458722 10254 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.458945  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.477872 10254 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.478552 10254 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.484262  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.481480 10254 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:02.562031  1634 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.562772  1634 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:05:02.564049  1634 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.564529  1634 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:05:02.564612 10154 raft_consensus.cc:2749] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:02.565469 10154 raft_consensus.cc:2749] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:05:02.585129  9879 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:05:02.585547  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.586043  9879 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.587016 10146 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.587704 10146 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.589874 10146 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.592927  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.593753  1662 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 27.
W20260709 13:05:02.596757  1708 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.606675  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:02.599745  1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 135) took 5113 ms. Trace:
I20260709 13:05:02.611984 10146 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 13:05:02.621903 10146 raft_consensus.cc:493] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.622359 10146 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Advancing to term 28
W20260709 13:05:02.615803  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:02.620245  1482 catalog_manager.cc:5697] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 13 to 22. New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.630788  1482 catalog_manager.cc:5697] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 17 to 24, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:02.635351  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.635913  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:02.632295  1482 catalog_manager.cc:5697] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:02.638763  1708 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.638458  1482 catalog_manager.cc:5697] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 15 to 24. New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.639814  1482 catalog_manager.cc:5697] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 17 to 22, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.641558  1482 catalog_manager.cc:5697] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 11 to 19, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 19 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.644150 10235 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.644800 10235 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.647183 10235 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.617717  1484 rpcz_store.cc:276] 0709 13:04:57.493333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:01.585276 (+4091943us) service_pool.cc:224] Handling call
0709 13:05:02.599636 (+1014360us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:02.391845 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:02.392403 (+   558us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:02.392452 (+    49us) write_op.cc:435] Acquired schema lock
0709 13:05:02.392471 (+    19us) tablet.cc:662] Decoding operations
0709 13:05:02.393860 (+  1389us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:02.393923 (+    63us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:02.393952 (+    29us) tablet.cc:685] Acquiring locks for 16 operations
0709 13:05:02.398926 (+  4974us) tablet.cc:701] Row locks acquired
0709 13:05:02.398947 (+    21us) write_op.cc:260] PREPARE: finished
0709 13:05:02.399103 (+   156us) write_op.cc:270] Start()
0709 13:05:02.399237 (+   134us) write_op.cc:276] Timestamp: P: 1783602302399055 usec, L: 0
0709 13:05:02.399259 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:05:02.400049 (+   790us) log.cc:844] Serialized 6278 byte log entry
0709 13:05:02.403408 (+  3359us) op_driver.cc:464] REPLICATION: finished
0709 13:05:02.403806 (+   398us) write_op.cc:301] APPLY: starting
0709 13:05:02.403912 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:02.423430 (+ 19518us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:02.423567 (+   137us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:02.533926 (+110359us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:02.534207 (+   281us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 13:05:02.534256 (+    49us) write_op.cc:312] APPLY: finished
0709 13:05:02.534653 (+   397us) log.cc:844] Serialized 148 byte log entry
0709 13:05:02.535507 (+   854us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:02.535687 (+   180us) write_op.cc:454] Released schema lock
0709 13:05:02.535750 (+    63us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:02.580175 (+ 44425us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:02.580218 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":77505,"reported_tablets":23,"child_traces":[["op",{"apply.queue_time_us":147,"apply.run_cpu_time_us":22987,"apply.run_wall_time_us":176268,"cfile_cache_hit":64,"cfile_cache_hit_bytes":66912,"num_ops":16,"prepare.queue_time_us":119,"prepare.run_cpu_time_us":5003,"prepare.run_wall_time_us":11189,"replication_time_us":4014,"spinlock_wait_cycles":44288}]]}
I20260709 13:05:02.650765  9879 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.654565  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.655403  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.657604  1482 catalog_manager.cc:5697] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 15 to 22, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.679631  1662 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:05:02.679313  1482 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 18 to 21. New cstate: current_term: 21 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:02.683411  9879 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.683838  1482 catalog_manager.cc:5697] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 15 to 24. New cstate: current_term: 24 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.685564  1482 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 13 to 21. New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.688372 10235 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.688536  1482 catalog_manager.cc:5697] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 19 to 24, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.689013 10235 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:02.690613  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.690438  1482 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 15 to 20, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 20 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.684401  9879 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.691952  1482 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 17 to 24, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 24 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.693727  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.694430  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.694463  1482 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 8 to 23, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 23 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.702507  9879 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.703102  1482 catalog_manager.cc:5697] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 21 to 25, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 25 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.704245 10235 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.705163  1482 catalog_manager.cc:5697] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 10 to 19, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 19 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.707203  1482 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 18 to 23, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 23 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:02.708453  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.709051  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:02.710111 10140 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.710698 10140 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.712975 10140 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.714589 10140 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:02.714800  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.717283 10254 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.717865 10254 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.719938 10254 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.708624  1482 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 17 to 25. New cstate: current_term: 25 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.720939  1482 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 16 to 23, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 23 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.722402  1482 catalog_manager.cc:5697] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 14 to 22, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.724104  1482 catalog_manager.cc:5697] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 16 to 21, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.725679 10254 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
W20260709 13:05:02.726161  1634 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.726301 10254 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.734606  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:02.738183 10254 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:02.738482  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.740566 10140 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:05:02.741020 10140 raft_consensus.cc:493] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.741413 10140 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:02.750185 10140 raft_consensus.cc:515] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.752518 10140 leader_election.cc:290] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.754089  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "200c02cbe43c4af8b4c59f21f210a9bc" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:02.754803  1667 raft_consensus.cc:3060] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:02.739236  1482 catalog_manager.cc:5697] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 18 to 21, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 21 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.758658  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.760336  1482 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 21 to 28, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 28 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.761838  1482 catalog_manager.cc:5697] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.764173  1482 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.765420  1482 catalog_manager.cc:5697] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 16 to 22. New cstate: current_term: 22 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.766894  1482 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 16 to 23, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 23 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:02.768615  1482 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 14 to 18, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 18 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:02.769096  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.770273  1735 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:05:02.790129 10239 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.790985 10239 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.771442 10254 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.795830 10254 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.799180  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.799958  1662 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
W20260709 13:05:02.801628  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.802207  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.804540  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:02.806373  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:02.807461  1632 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:05:02.808547  2283 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.815397  2283 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:05:02.805574 10239 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:05:02.821242  1667 raft_consensus.cc:2468] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:05:02.822464  2281 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:02.822629  1705 leader_election.cc:304] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:02.824296 10146 raft_consensus.cc:515] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.830175 10276 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:05:02.831115 10276 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:02.820940 10154 raft_consensus.cc:2749] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:02.825740 10239 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.842932 10271 raft_consensus.cc:2804] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:05:02.852339 10271 raft_consensus.cc:697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:02.852020 10239 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.851270 10146 leader_election.cc:290] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.854779  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0444300de6924432844fa6519c92b570" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
W20260709 13:05:02.854904  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 47520.5us
I20260709 13:05:02.855436  1667 raft_consensus.cc:3060] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Advancing to term 28
I20260709 13:05:02.856520  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:02.859205  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:05:02.859285  2283 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:02.860766  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:02.862321  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.859970  2310 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:05:02.853878 10254 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:02.873273  1667 raft_consensus.cc:2468] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
I20260709 13:05:02.873081 10239 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:02.877205 10254 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.892848 10254 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.895758 10254 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:02.895982 10274 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.889340 10245 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:05:02.896677 10274 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:02.897338 10245 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.897825 10245 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:02.891570  1705 leader_election.cc:304] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:05:02.901144  1708 leader_election.cc:336] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:02.902087  1708 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:02.902554  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:02.903154  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:02.903563  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.903944  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:02.919152  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.921538  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:02.922326  2310 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 22.
I20260709 13:05:02.925567  1632 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:02.915091 10273 raft_consensus.cc:2804] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader election won for term 28
I20260709 13:05:02.923807 10146 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:02.853329 10271 consensus_queue.cc:237] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.940048  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:02.944247  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:05:02.960331  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:02.961112  1708 leader_election.cc:336] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:02.963773 10274 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:02.932629 10273 raft_consensus.cc:697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 28 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:02.930823 10280 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.966905 10273 consensus_queue.cc:237] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.967391 10280 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.943207 10254 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:05:02.970614 10254 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.971079 10254 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
W20260709 13:05:02.972581  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:02.973548  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:02.974161 10278 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.974756 10278 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.973548 10239 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:05:02.976475 10239 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.976888 10239 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:05:02.977028 10278 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.984103  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:02.991284 10281 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.992002 10281 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.994041 10280 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:02.995775  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:02.996093 10281 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:02.997354 10275 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:02.997961 10275 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.000218 10275 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:03.003311  1708 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.005776  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:03.006677  1666 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:05:03.008422  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.010659 10275 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:05:03.010947  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:03.011089 10275 raft_consensus.cc:493] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.011720 10275 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:03.011803  1734 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 20.
I20260709 13:05:03.013152  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:03.020849  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.023886  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.024242  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:03.025888 10278 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 13:05:03.026571 10278 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.027091 10278 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Advancing to term 21
W20260709 13:05:03.024662  1634 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.023784 10271 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.039738 10271 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.042408 10239 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.044231  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:03.045022  1666 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
I20260709 13:05:03.046409 10278 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.051594 10278 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:03.052919 10270 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.053097  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eed985f1202940a1ab59ea7b7b3c745a" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:05:03.053539  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.057372  1734 raft_consensus.cc:3060] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Advancing to term 21
I20260709 13:05:03.057119 10270 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:02.977412 10245 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.059463 10270 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:03.060561 10245 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:03.060838  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40d5b254644257a6d39df949366e1b" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
W20260709 13:05:03.061486  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:03.061753  1735 raft_consensus.cc:3060] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:03.062435 10239 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:05:03.068472 10274 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.069154 10274 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.075044 10274 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:03.077946  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:03.078857  1667 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:05:03.080119  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.081380 10274 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 13:05:03.081830 10274 raft_consensus.cc:493] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.082193 10274 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:05:03.084787  1734 raft_consensus.cc:2468] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 21.
I20260709 13:05:03.078027  1735 raft_consensus.cc:2468] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 25.
I20260709 13:05:03.078078  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:03.086277  1666 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 24.
I20260709 13:05:03.086179  1632 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.087512  1632 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.088200 10271 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.091218 10245 raft_consensus.cc:2804] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 13:05:03.091946 10245 raft_consensus.cc:697] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:05:03.093008 10245 consensus_queue.cc:237] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.095374 10254 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.095621 10278 raft_consensus.cc:2804] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:05:03.095398  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.097797 10278 raft_consensus.cc:697] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:05:03.098373 10270 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:05:03.098855 10270 raft_consensus.cc:493] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.099241 10270 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
W20260709 13:05:03.103047  1634 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.103662  1634 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:05:03.104018  2280 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 80901.4us
I20260709 13:05:03.104895 10239 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 13:05:03.099435 10278 consensus_queue.cc:237] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:03.116727  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:03.118223  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:03.118731  1708 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:03.123828  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55102.9us
I20260709 13:05:03.125165  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:03.124123  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:03.126981  2309 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 18.
I20260709 13:05:03.125344 10282 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:03.126049  2310 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 22.
W20260709 13:05:03.129539  1708 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.132397 10254 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.135072  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:05:03.135970  2310 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 26: Already voted for candidate e6adef9da7da44f6af0af5695cf9b704 in this term.
I20260709 13:05:03.113447 10270 raft_consensus.cc:515] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.139515  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.140363 10270 leader_election.cc:290] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:03.140836  1706 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.141732  9879 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 13:05:03.142199  9879 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.142589  9879 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Advancing to term 19
W20260709 13:05:03.146231  1708 leader_election.cc:336] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.147262 10273 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 13:05:03.147766 10273 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.148151 10273 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:05:03.152621  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:03.159803  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "421876549c8f408587a97543338bca1d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:03.160600  1662 raft_consensus.cc:3060] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:03.165160 10275 raft_consensus.cc:515] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.168172  1740 raft_consensus.cc:2393] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ff44a4643e01482cb6b6a79b099fa766 in current term 25: Already voted for candidate df6a18af06d543b2931837f5e6bab95f in this term.
W20260709 13:05:03.176590  1708 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.182724  9879 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.183403 10274 raft_consensus.cc:515] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.186443  1667 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "249474541a174b90a099d77784ed4801" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:03.187053  1667 raft_consensus.cc:3060] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Advancing to term 26
I20260709 13:05:03.192162  1662 raft_consensus.cc:2468] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:05:03.193418  1705 leader_election.cc:304] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.194290 10146 raft_consensus.cc:2804] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:05:03.194957 10146 raft_consensus.cc:697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:03.195928 10146 consensus_queue.cc:237] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.200309  1667 raft_consensus.cc:2468] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 26.
W20260709 13:05:03.205276  1708 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.205979  1666 raft_consensus.cc:2393] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate e6adef9da7da44f6af0af5695cf9b704 in current term 26: Already voted for candidate ff44a4643e01482cb6b6a79b099fa766 in this term.
I20260709 13:05:03.201854  1705 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 
W20260709 13:05:03.206624  1708 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.208446 10275 leader_election.cc:290] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:03.210693  1634 leader_election.cc:336] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.211194  1634 leader_election.cc:304] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:05:03.211900  9879 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.212203 10281 raft_consensus.cc:2749] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:05:03.215320  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "026ea522d0f44f06bfab3c6eddc5e0b3" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:03.215984  1662 raft_consensus.cc:3060] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:03.216145  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f9cc5e1ffa141e295028cdd523da631" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 19 candidate_status { last_received { term: 18 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:05:03.216744  2310 raft_consensus.cc:3060] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Advancing to term 19
I20260709 13:05:03.216167 10270 raft_consensus.cc:2804] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader election won for term 26
I20260709 13:05:03.219311 10270 raft_consensus.cc:697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 26 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:03.219609 10273 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.220232 10270 consensus_queue.cc:237] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.212898 10274 leader_election.cc:290] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:03.227417  1634 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:03.229055  1634 leader_election.cc:336] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.229703  1634 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
I20260709 13:05:03.230139  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:03.232865  1708 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.232959 10276 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 13:05:03.231192  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.234508  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:03.238637  1662 raft_consensus.cc:2468] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 25.
I20260709 13:05:03.240109 10282 raft_consensus.cc:2749] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:03.245293 10273 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.250540  1705 leader_election.cc:304] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.250952  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:03.251794  1666 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:05:03.252277 10274 raft_consensus.cc:2804] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:05:03.256640 10274 raft_consensus.cc:697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:03.262074  2310 raft_consensus.cc:2468] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 19.
I20260709 13:05:03.264184 10274 consensus_queue.cc:237] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.269181  1706 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.272881 10280 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.273484 10280 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.275296  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "162373eeadc2444e87ffd67df6a836e5" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:05:03.275906  2310 raft_consensus.cc:3060] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Advancing to term 23
I20260709 13:05:03.286036  9879 raft_consensus.cc:2804] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader election won for term 19
I20260709 13:05:03.286742  9879 raft_consensus.cc:697] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:03.287786  9879 consensus_queue.cc:237] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 18.17, Last appended by leader: 17, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.289551 10280 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.296288  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:05:03.294896  1634 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 21 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.297077  2309 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 23.
W20260709 13:05:03.297603  1634 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:03.299016  1634 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.299487  1634 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a, e6adef9da7da44f6af0af5695cf9b704
I20260709 13:05:03.296299  2310 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:05:03.302593  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.308159 10280 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 13:05:03.308640 10280 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 316f3bf18ed94e32b238130a6bfedf1a)
I20260709 13:05:03.309069 10280 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:03.315466  1666 raft_consensus.cc:2468] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 23.
I20260709 13:05:03.317148  1705 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
W20260709 13:05:03.319248  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:03.323526 10154 raft_consensus.cc:2749] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 13:05:03.327497 10275 raft_consensus.cc:2804] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Leader election won for term 23
W20260709 13:05:03.385392  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:03.371690 10154 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:03.394750  1666 raft_consensus.cc:1275] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 13:05:03.394584  1484 catalog_manager.cc:5697] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 23 to 27. New cstate: current_term: 27 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:03.402343  1484 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 13 to 16, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:03.405491   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:03.405227 10154 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.416628  1484 catalog_manager.cc:5697] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 17 to 23. New cstate: current_term: 23 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:03.422109  1484 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 24, leader changed from e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:03.427739  1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 242) took 6147 ms. Trace:
I20260709 13:05:03.432711  1482 rpcz_store.cc:276] 0709 13:04:57.275556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:59.991251 (+2715695us) service_pool.cc:224] Handling call
0709 13:05:03.423194 (+3431943us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:02.829968 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:02.830595 (+   627us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:02.830636 (+    41us) write_op.cc:435] Acquired schema lock
0709 13:05:02.830647 (+    11us) tablet.cc:662] Decoding operations
0709 13:05:02.833967 (+  3320us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:02.834038 (+    71us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:02.834076 (+    38us) tablet.cc:685] Acquiring locks for 28 operations
0709 13:05:02.839924 (+  5848us) tablet.cc:701] Row locks acquired
0709 13:05:02.839947 (+    23us) write_op.cc:260] PREPARE: finished
0709 13:05:02.840126 (+   179us) write_op.cc:270] Start()
0709 13:05:02.840281 (+   155us) write_op.cc:276] Timestamp: P: 1783602302840067 usec, L: 0
0709 13:05:02.840308 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 13:05:02.850409 (+ 10101us) log.cc:844] Serialized 11009 byte log entry
0709 13:05:02.895591 (+ 45182us) op_driver.cc:464] REPLICATION: finished
0709 13:05:02.897742 (+  2151us) write_op.cc:301] APPLY: starting
0709 13:05:02.897839 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:03.075858 (+178019us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:03.075970 (+   112us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:03.248114 (+172144us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:03.248553 (+   439us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=56,key_file_lookups=56,delta_file_lookups=0,mrs_lookups=0
0709 13:05:03.248620 (+    67us) write_op.cc:312] APPLY: finished
0709 13:05:03.249215 (+   595us) log.cc:844] Serialized 244 byte log entry
0709 13:05:03.272678 (+ 23463us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:03.272961 (+   283us) write_op.cc:454] Released schema lock
0709 13:05:03.273039 (+    78us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:03.273449 (+   410us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:03.273484 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1331,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":1884,"apply.run_cpu_time_us":36812,"apply.run_wall_time_us":375678,"cfile_cache_hit":112,"cfile_cache_hit_bytes":117096,"num_ops":28,"prepare.queue_time_us":165,"prepare.run_cpu_time_us":7582,"prepare.run_wall_time_us":20784,"replication_time_us":55123}]]}
I20260709 13:05:03.427477  1484 catalog_manager.cc:5697] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:03.431401 10273 consensus_queue.cc:1048] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 13:05:03.447104  1484 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 14 to 22. New cstate: current_term: 22 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:03.449863  1484 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 23, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 23 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:03.461880  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.457271 10154 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:03.469374  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:03.478417 10280 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:03.489639 10280 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:03.491191  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9c383ed9b7c4624b8fa546326e129ff" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
I20260709 13:05:03.491961  2310 raft_consensus.cc:3060] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Advancing to term 24
I20260709 13:05:03.446491 10275 raft_consensus.cc:697] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Becoming Leader. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:03.506899 10275 consensus_queue.cc:237] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:03.508764  1634 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.516739  2310 raft_consensus.cc:2468] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 24.
I20260709 13:05:03.518023  1632 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:03.522753 10282 raft_consensus.cc:2804] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 13:05:03.537705 10282 raft_consensus.cc:697] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:05:03.538792 10282 consensus_queue.cc:237] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:03.545322  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:03.545960  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:03.546535  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:03.546954  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:03.559113  1667 raft_consensus.cc:1275] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 13:05:03.559682  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:03.560370  1734 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 26.
I20260709 13:05:03.566447  1735 raft_consensus.cc:1275] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 13:05:03.582701  1634 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.594173  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.595222 10154 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:05:03.595621 10154 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:03.595968 10154 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:05:03.612771  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:03.616578  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 45324.7us
I20260709 13:05:03.623764  1735 raft_consensus.cc:1275] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 13:05:03.625198 10245 consensus_queue.cc:1048] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 13:05:03.634806  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:03.671813  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 102345us
I20260709 13:05:03.631325 10295 consensus_queue.cc:1048] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 13:05:03.693110 10154 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.699385 10154 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:03.634740 10292 consensus_queue.cc:1048] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
W20260709 13:05:03.712538  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:03.713222  1634 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:03.713798  1734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a654178d0194a7388155af12a626d35" candidate_uuid: "ff44a4643e01482cb6b6a79b099fa766" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:03.714519  1734 raft_consensus.cc:3060] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:05:03.712443  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:03.773315  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:03.797305  1484 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:49282 (request call id 136) took 1007 ms. Trace:
I20260709 13:05:03.815284  1484 rpcz_store.cc:276] 0709 13:05:02.789842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:02.793633 (+  3791us) service_pool.cc:224] Handling call
0709 13:05:03.797218 (+1003585us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:03.573258 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:03.578784 (+  5526us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:03.578846 (+    62us) write_op.cc:435] Acquired schema lock
0709 13:05:03.578866 (+    20us) tablet.cc:662] Decoding operations
0709 13:05:03.579840 (+   974us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:03.579923 (+    83us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:03.579960 (+    37us) tablet.cc:685] Acquiring locks for 7 operations
0709 13:05:03.580750 (+   790us) tablet.cc:701] Row locks acquired
0709 13:05:03.580772 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:05:03.580940 (+   168us) write_op.cc:270] Start()
0709 13:05:03.581073 (+   133us) write_op.cc:276] Timestamp: P: 1783602303580883 usec, L: 0
0709 13:05:03.581097 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 13:05:03.613991 (+ 32894us) log.cc:844] Serialized 2961 byte log entry
0709 13:05:03.704977 (+ 90986us) op_driver.cc:464] REPLICATION: finished
0709 13:05:03.720717 (+ 15740us) write_op.cc:301] APPLY: starting
0709 13:05:03.720838 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:03.724703 (+  3865us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:03.724825 (+   122us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:03.781198 (+ 56373us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:03.781443 (+   245us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 13:05:03.781517 (+    74us) write_op.cc:312] APPLY: finished
0709 13:05:03.781858 (+   341us) log.cc:844] Serialized 73 byte log entry
0709 13:05:03.782060 (+   202us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:03.782194 (+   134us) write_op.cc:454] Released schema lock
0709 13:05:03.782246 (+    52us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:03.782672 (+   426us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:03.782701 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":45372,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":7791,"apply.run_cpu_time_us":11146,"apply.run_wall_time_us":63123,"cfile_cache_hit":28,"cfile_cache_hit_bytes":29274,"num_ops":7,"prepare.queue_time_us":106665,"prepare.run_cpu_time_us":4529,"prepare.run_wall_time_us":53125,"replication_time_us":123684,"thread_start_us":71818,"threads_started":1}]]}
W20260709 13:05:03.848304  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:03.852643  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:03.857146  1734 raft_consensus.cc:2468] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff44a4643e01482cb6b6a79b099fa766 in term 27.
I20260709 13:05:03.858619  1632 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:03.862226  1662 raft_consensus.cc:1275] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 13:05:03.870085  1659 raft_consensus.cc:1275] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 13:05:03.876959  1659 raft_consensus.cc:1275] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 13:05:03.878840 10275 consensus_queue.cc:1048] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 13:05:03.882114  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:03.874392 10282 raft_consensus.cc:2804] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:05:03.890194 10282 raft_consensus.cc:697] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 27 LEADER]: Becoming Leader. State: Replica: ff44a4643e01482cb6b6a79b099fa766, State: Running, Role: LEADER
I20260709 13:05:03.891189 10282 consensus_queue.cc:237] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:03.894295  2310 raft_consensus.cc:1275] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 17. Preceding OpId from leader: term: 19 index: 18. (index mismatch)
W20260709 13:05:03.883172  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:03.898468 10275 consensus_queue.cc:1048] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 13:05:03.907500 10293 consensus_queue.cc:1048] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:05:03.915473  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:03.932931  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:03.905651 10275 consensus_queue.cc:1048] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 13:05:04.048408  2310 raft_consensus.cc:1275] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 13:05:04.052878  1490 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 255) took 125799 ms (client timeout 15000 ms). Trace:
W20260709 13:05:04.053540  1490 rpcz_store.cc:269] 0709 13:02:58.251842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:02:58.269711 (+ 17869us) service_pool.cc:224] Handling call
0709 13:05:04.050693 (+125780982us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":900215,"reported_tablets":51}
I20260709 13:05:04.062582  1668 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 13:05:04.062628 10281 consensus_queue.cc:1048] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 13:05:04.067821  1486 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 8 to 20, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193). New cstate: current_term: 20 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:04.068742  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49576
I20260709 13:05:04.069617  1735 raft_consensus.cc:606] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Received request to transfer leadership
W20260709 13:05:04.073781  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:04.081077  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:04.083178 10296 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 13:05:04.085294  1486 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 14 to 25. New cstate: current_term: 25 leader_uuid: "316f3bf18ed94e32b238130a6bfedf1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:04.092134  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:04.097754  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49576
I20260709 13:05:04.098685  1735 raft_consensus.cc:606] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Received request to transfer leadership
W20260709 13:05:04.118605  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:04.129395  2310 raft_consensus.cc:1275] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 13:05:04.132676  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:04.132341 10273 consensus_queue.cc:1048] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 13:05:04.147285  1735 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:49576
I20260709 13:05:04.148109  1735 raft_consensus.cc:606] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Received request to transfer leadership
I20260709 13:05:04.159802  1486 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P 316f3bf18ed94e32b238130a6bfedf1a reported cstate change: term changed from 15 to 23, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 23 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:04.187542  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:04.217367  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:04.250514  1539 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 13:05:04.251062  1539 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for e676e54141f7470db4551f4de439c017 already in progress
W20260709 13:05:04.251374  1539 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for e676e54141f7470db4551f4de439c017 already in progress
I20260709 13:05:04.329991  1735 raft_consensus.cc:1275] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 13:05:04.332811 10292 consensus_queue.cc:1048] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:05:04.337888  1484 catalog_manager.cc:5697] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 18 to 19, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 19 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:04.368046  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:04.404860  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:04.405988   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:04.420492  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:04.431533  1489 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 13 to 16, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 16 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:04.435405  1486 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 232) took 20865 ms (client timeout 15000 ms). Trace:
W20260709 13:05:04.439862  1486 rpcz_store.cc:269] 0709 13:04:43.570615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:52.149944 (+8579329us) service_pool.cc:224] Handling call
0709 13:05:04.435315 (+12285371us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:04.240791 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:04.241448 (+   657us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:04.241486 (+    38us) write_op.cc:435] Acquired schema lock
0709 13:05:04.241535 (+    49us) tablet.cc:662] Decoding operations
0709 13:05:04.243134 (+  1599us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:04.243222 (+    88us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:04.243249 (+    27us) tablet.cc:685] Acquiring locks for 3 operations
0709 13:05:04.266768 (+ 23519us) tablet.cc:701] Row locks acquired
0709 13:05:04.266794 (+    26us) write_op.cc:260] PREPARE: finished
0709 13:05:04.267004 (+   210us) write_op.cc:270] Start()
0709 13:05:04.267135 (+   131us) write_op.cc:276] Timestamp: P: 1783602304266945 usec, L: 0
0709 13:05:04.267160 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:05:04.268144 (+   984us) log.cc:844] Serialized 1317 byte log entry
0709 13:05:04.310166 (+ 42022us) op_driver.cc:464] REPLICATION: finished
0709 13:05:04.318934 (+  8768us) write_op.cc:301] APPLY: starting
0709 13:05:04.319031 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:04.321772 (+  2741us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:04.321906 (+   134us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:04.324686 (+  2780us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:04.324947 (+   261us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 13:05:04.325016 (+    69us) write_op.cc:312] APPLY: finished
0709 13:05:04.325287 (+   271us) log.cc:844] Serialized 41 byte log entry
0709 13:05:04.325554 (+   267us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.325677 (+   123us) write_op.cc:454] Released schema lock
0709 13:05:04.325731 (+    54us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:04.326262 (+   531us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.326299 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16045,"reported_tablets":51,"spinlock_wait_cycles":2304,"child_traces":[["op",{"apply.queue_time_us":8469,"apply.run_cpu_time_us":6829,"apply.run_wall_time_us":7251,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16673,"num_ops":3,"prepare.queue_time_us":37020,"prepare.run_cpu_time_us":3864,"prepare.run_wall_time_us":27761,"replication_time_us":42833,"thread_start_us":44226,"threads_started":2}]]}
W20260709 13:05:04.452946  1476 connection.cc:466] server connection from 127.0.0.1:59532 torn down before Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59532 (request call id 232) could send its response
I20260709 13:05:04.446254  1489 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.474071 10295 raft_consensus.cc:993] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: : Instructing follower e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:05:04.474705 10295 raft_consensus.cc:1081] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Signalling peer e6adef9da7da44f6af0af5695cf9b704 to start an election
I20260709 13:05:04.502483  2309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e676e54141f7470db4551f4de439c017"
dest_uuid: "e6adef9da7da44f6af0af5695cf9b704"
 from {username='slave'} at 127.0.0.1:58316
I20260709 13:05:04.503269  2309 raft_consensus.cc:493] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:05:04.503657  2309 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:05:04.547956  2309 raft_consensus.cc:515] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:04.555501  1666 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:04.556201  1666 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Advancing to term 17
I20260709 13:05:04.556831  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e676e54141f7470db4551f4de439c017" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: true dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:04.557564  1740 raft_consensus.cc:3055] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Stepping down as leader of term 16
I20260709 13:05:04.557904  1740 raft_consensus.cc:740] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 LEADER]: Becoming Follower/Learner. State: Replica: df6a18af06d543b2931837f5e6bab95f, State: Running, Role: LEADER
I20260709 13:05:04.558888  1740 consensus_queue.cc:260] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:04.559824  2309 leader_election.cc:290] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Requested vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:04.560622  1740 raft_consensus.cc:3060] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Advancing to term 17
W20260709 13:05:04.560761  2283 leader_election.cc:336] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:04.576126  1666 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:05:04.645277  1740 raft_consensus.cc:2468] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 17.
I20260709 13:05:04.647328  2281 leader_election.cc:304] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:04.651469 10279 raft_consensus.cc:2804] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 13:05:04.652047 10279 raft_consensus.cc:697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:05:04.652981 10279 consensus_queue.cc:237] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:04.672189  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:04.671188  1489 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 240) took 60161 ms (client timeout 15000 ms). Trace:
W20260709 13:05:04.691465  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 44208.8us
W20260709 13:05:04.684696  1489 rpcz_store.cc:269] 0709 13:04:04.509578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:14.453648 (+9944070us) service_pool.cc:224] Handling call
0709 13:04:14.687262 (+233614us) spinlock_profiling.cc:243] Waited 2.73 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb90b2 00007fcdf3e32a89 00007fcdf3dcada6 00007fcdf3d9c088 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc
0709 13:04:14.690400 (+  3138us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b54002904d4. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdedf47090 00007fcdedfe5f3a 00007fcdf3d9c1cc 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc 00007fcdf235a392 00007fcdec31848c 00007fcdec3178c6
0709 13:05:04.671079 (+49980679us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:04.507557 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:04.508226 (+   669us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:04.508270 (+    44us) write_op.cc:435] Acquired schema lock
0709 13:05:04.508288 (+    18us) tablet.cc:662] Decoding operations
0709 13:05:04.508811 (+   523us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:04.508878 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:04.508918 (+    40us) tablet.cc:685] Acquiring locks for 2 operations
0709 13:05:04.509324 (+   406us) tablet.cc:701] Row locks acquired
0709 13:05:04.509342 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:05:04.509540 (+   198us) write_op.cc:270] Start()
0709 13:05:04.509668 (+   128us) write_op.cc:276] Timestamp: P: 1783602304509468 usec, L: 0
0709 13:05:04.509688 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 13:05:04.510389 (+   701us) log.cc:844] Serialized 929 byte log entry
0709 13:05:04.516120 (+  5731us) op_driver.cc:464] REPLICATION: finished
0709 13:05:04.518863 (+  2743us) write_op.cc:301] APPLY: starting
0709 13:05:04.518968 (+   105us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:04.520479 (+  1511us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:04.520592 (+   113us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:04.522138 (+  1546us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:04.522344 (+   206us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 13:05:04.522398 (+    54us) write_op.cc:312] APPLY: finished
0709 13:05:04.522661 (+   263us) log.cc:844] Serialized 33 byte log entry
0709 13:05:04.522871 (+   210us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.522980 (+   109us) write_op.cc:454] Released schema lock
0709 13:05:04.523033 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:04.531688 (+  8655us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b18003959e0. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdf3db8de1 00007fcdf3dc55b1 00007fcdeaef0522 00007fcdeaef27c1 00007fcdeaef2779 00007fcdeaef2709 00007fcdeaef26d1 00007fcdeaef19cd 000000000038f354 000000000038f199 00007fcdeaed383a
0709 13:05:04.531834 (+   146us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.531869 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":14735,"reported_tablets":52,"spinlock_wait_cycles":9490048,"child_traces":[["op",{"apply.queue_time_us":2502,"apply.run_cpu_time_us":4562,"apply.run_wall_time_us":4564,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8364,"num_ops":2,"prepare.queue_time_us":4648,"prepare.run_cpu_time_us":3250,"prepare.run_wall_time_us":17009,"replication_time_us":6293,"spinlock_wait_cycles":10332416}]]}
I20260709 13:05:04.718034  1484 catalog_manager.cc:5697] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 23 to 24, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 24 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:04.723140  1484 catalog_manager.cc:5697] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 22 to 23, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 23 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:04.742326  1491 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 256) took 92717 ms (client timeout 15000 ms). Trace:
W20260709 13:05:04.742904  1491 rpcz_store.cc:269] 0709 13:03:32.025390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:03:32.025682 (+   292us) service_pool.cc:224] Handling call
0709 13:03:32.043266 (+ 17584us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b5400002210. stack: 00007fcdeaebe343 00007fcdeaebe240 00007fcdea84d1d4 00007fcdf458b304 00007fcdf458b299 00007fcdead78db1 00007fcdeaeb8b71 00007fcdeaeb90b2 00007fcdf3e32a89 00007fcdf3dcada6 00007fcdf3d9c088 00007fcdf3e9a37e 00007fcdf235b2a4 00007fcdf235b232 00007fcdf235b161 00007fcdf235b0dc
0709 13:05:04.742071 (+92698805us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26994,"reported_tablets":51,"spinlock_wait_cycles":8766848}
I20260709 13:05:04.773661  1483 catalog_manager.cc:5697] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 14 to 21, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:04.778920  1483 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 16 to 23, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 23 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:04.781097  1483 catalog_manager.cc:5697] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 23 to 25, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 25 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:04.784737  1483 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: term changed from 25 to 26, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 26 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.822069  1484 catalog_manager.cc:5697] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 16 to 17, leader changed from df6a18af06d543b2931837f5e6bab95f (127.0.107.195) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 17 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.862721  1482 catalog_manager.cc:5697] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 20 to 21, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:04.864706  1482 catalog_manager.cc:5697] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 24 to 25, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 25 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:04.867287  1482 catalog_manager.cc:5697] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 23 to 25, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 25 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:04.872991  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:04.875053  1482 catalog_manager.cc:5697] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 23 to 26, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 26 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
W20260709 13:05:04.883080  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:04.885576  1482 catalog_manager.cc:5697] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 23 to 24, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 24 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.887775  1482 catalog_manager.cc:5697] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 25 to 28, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 28 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.918545  1482 catalog_manager.cc:5697] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 23 to 24, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 24 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.923331  1483 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59536 (request call id 259) took 9960 ms. Trace:
I20260709 13:05:04.924175  1482 catalog_manager.cc:5697] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 26 to 28, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 28 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:04.923861  1483 rpcz_store.cc:276] 0709 13:04:54.962957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:04:56.756479 (+1793522us) service_pool.cc:224] Handling call
0709 13:05:04.802654 (+8046175us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.802697 (+    43us) write_op.cc:454] Released schema lock
0709 13:05:04.923221 (+120524us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:04.789312 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:04.789929 (+   617us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:04.789958 (+    29us) write_op.cc:435] Acquired schema lock
0709 13:05:04.789975 (+    17us) tablet.cc:662] Decoding operations
0709 13:05:04.790593 (+   618us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:04.790665 (+    72us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:04.790697 (+    32us) tablet.cc:685] Acquiring locks for 4 operations
0709 13:05:04.791142 (+   445us) tablet.cc:701] Row locks acquired
0709 13:05:04.791164 (+    22us) write_op.cc:260] PREPARE: finished
0709 13:05:04.791344 (+   180us) write_op.cc:270] Start()
0709 13:05:04.791465 (+   121us) write_op.cc:276] Timestamp: P: 1783602304791290 usec, L: 0
0709 13:05:04.791490 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 13:05:04.792191 (+   701us) log.cc:844] Serialized 1706 byte log entry
0709 13:05:04.794690 (+  2499us) op_driver.cc:464] REPLICATION: finished
0709 13:05:04.795146 (+   456us) write_op.cc:301] APPLY: starting
0709 13:05:04.795232 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:04.797655 (+  2423us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:04.797747 (+    92us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:04.800258 (+  2511us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:04.800464 (+   206us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 13:05:04.800510 (+    46us) write_op.cc:312] APPLY: finished
0709 13:05:04.800782 (+   272us) log.cc:844] Serialized 49 byte log entry
0709 13:05:04.801739 (+   957us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.801864 (+   125us) write_op.cc:454] Released schema lock
0709 13:05:04.801916 (+    52us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":26578,"reported_tablets":51,"spinlock_wait_cycles":185728,"child_traces":[["op",{"apply.queue_time_us":254,"apply.run_cpu_time_us":6336,"apply.run_wall_time_us":7094,"cfile_cache_hit":16,"cfile_cache_hit_bytes":16728,"num_ops":4,"prepare.queue_time_us":135,"prepare.run_cpu_time_us":3197,"prepare.run_wall_time_us":5140,"replication_time_us":3078}]]}
I20260709 13:05:04.926559  1482 catalog_manager.cc:5697] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 15 to 16, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 16 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
I20260709 13:05:04.929039  1482 catalog_manager.cc:5697] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 20 to 25, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to df6a18af06d543b2931837f5e6bab95f (127.0.107.195). New cstate: current_term: 25 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } }
I20260709 13:05:05.005726  1482 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:59548 (request call id 243) took 1039 ms. Trace:
I20260709 13:05:05.006232  1482 rpcz_store.cc:276] 0709 13:05:03.966553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 13:05:03.973931 (+  7378us) service_pool.cc:224] Handling call
0709 13:05:05.005631 (+1031700us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 13:05:04.946498 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 13:05:04.947037 (+   539us) write_op.cc:432] Acquiring schema lock in shared mode
0709 13:05:04.947075 (+    38us) write_op.cc:435] Acquired schema lock
0709 13:05:04.947093 (+    18us) tablet.cc:662] Decoding operations
0709 13:05:04.948044 (+   951us) write_op.cc:620] Acquiring the partition lock for write op
0709 13:05:04.948122 (+    78us) write_op.cc:641] Partition lock acquired for write op
0709 13:05:04.948167 (+    45us) tablet.cc:685] Acquiring locks for 10 operations
0709 13:05:04.948898 (+   731us) tablet.cc:701] Row locks acquired
0709 13:05:04.948916 (+    18us) write_op.cc:260] PREPARE: finished
0709 13:05:04.949073 (+   157us) write_op.cc:270] Start()
0709 13:05:04.949198 (+   125us) write_op.cc:276] Timestamp: P: 1783602304949021 usec, L: 0
0709 13:05:04.949220 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 13:05:04.949932 (+   712us) log.cc:844] Serialized 4030 byte log entry
0709 13:05:04.952368 (+  2436us) op_driver.cc:464] REPLICATION: finished
0709 13:05:04.952724 (+   356us) write_op.cc:301] APPLY: starting
0709 13:05:04.952810 (+    86us) tablet.cc:1367] starting BulkCheckPresence
0709 13:05:04.959109 (+  6299us) tablet.cc:1370] finished BulkCheckPresence
0709 13:05:04.959225 (+   116us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 13:05:04.980115 (+ 20890us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 13:05:04.980356 (+   241us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 13:05:04.980400 (+    44us) write_op.cc:312] APPLY: finished
0709 13:05:04.980710 (+   310us) log.cc:844] Serialized 97 byte log entry
0709 13:05:04.983767 (+  3057us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.983949 (+   182us) write_op.cc:454] Released schema lock
0709 13:05:04.984009 (+    60us) write_op.cc:341] FINISH: Updating metrics
0709 13:05:04.984411 (+   402us) write_op.cc:489] Releasing partition, row and schema locks
0709 13:05:04.984437 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24791,"reported_tablets":31,"child_traces":[["op",{"apply.queue_time_us":114,"apply.run_cpu_time_us":12987,"apply.run_wall_time_us":31648,"cfile_cache_hit":40,"cfile_cache_hit_bytes":41820,"num_ops":10,"prepare.queue_time_us":1565,"prepare.run_cpu_time_us":3773,"prepare.run_wall_time_us":6315,"replication_time_us":3031}]]}
I20260709 13:05:05.056511  1482 catalog_manager.cc:5697] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f reported cstate change: term changed from 17 to 27, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194). New cstate: current_term: 27 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } }
W20260709 13:05:05.072702  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.085558  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.087658  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.153002  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.166644  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:05.185580  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:05.198344  1662 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:05:05.200626 10276 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:05.228907  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:05.254563  1735 raft_consensus.cc:1275] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 13:05:05.256955 10276 consensus_queue.cc:1048] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:05.257369  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.275928  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.317129  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.335052  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:05.340363 10276 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:05:05.375324 10276 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:05.385203 10276 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
W20260709 13:05:05.391909  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:05.399514  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766" is_pre_election: true
I20260709 13:05:05.400310  1662 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 26.
I20260709 13:05:05.402557  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 316f3bf18ed94e32b238130a6bfedf1a
I20260709 13:05:05.403702 10277 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 13:05:05.404229 10277 raft_consensus.cc:493] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:05:05.404626 10277 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Advancing to term 27
W20260709 13:05:05.406482   431 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:05.448554 10277 raft_consensus.cc:515] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:05.473694  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.484453  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:05.488090  1662 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbe322e4447d486680550d1490266b17" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "ff44a4643e01482cb6b6a79b099fa766"
I20260709 13:05:05.488806  1662 raft_consensus.cc:3060] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Advancing to term 27
I20260709 13:05:05.498219 10277 leader_election.cc:290] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267)
I20260709 13:05:05.504350  1662 raft_consensus.cc:2468] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 27.
I20260709 13:05:05.505960  2280 leader_election.cc:304] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766; no voters: 
I20260709 13:05:05.510213 10277 raft_consensus.cc:2804] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 13:05:05.510824 10277 raft_consensus.cc:697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 27 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:05:05.511857 10277 consensus_queue.cc:237] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:05.517675  2283 leader_election.cc:336] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:05.565074  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.585727  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:05.601893  1483 catalog_manager.cc:5697] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 25 to 27, leader changed from 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 27 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:05.611793  1660 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:05:05.617093  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.647802  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.665339  1632 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4db9dd8c47ea465786759da787f2b1ab. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:05.672971  1735 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 13:05:05.685302 10245 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:05:05.703262  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:05.710856  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:05.720760 10292 raft_consensus.cc:2955] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Committing config change with OpId 21.12: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:05:05.730000  1740 raft_consensus.cc:2955] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Committing config change with OpId 21.12: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:05.748450  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.780691  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:05.781635  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:05.808501  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:05.829103  1483 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:05.852608  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:05.854552  1660 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:05:05.880191  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:05.908205  1662 raft_consensus.cc:1275] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 13:05:05.910082 10277 consensus_queue.cc:1048] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 13:05:05.963522  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:05.973649  1632 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22ef4803b7a94ddcaacb46f38d689c5d. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:05.983726  1734 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 13:05:05.984905  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:05.987802 10245 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 13:05:05.998225  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:06.018333  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:06.088584 10245 raft_consensus.cc:2955] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 LEADER]: Committing config change with OpId 28.16: config changed from index -1 to 16, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:06.098515  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:06.101218  1735 raft_consensus.cc:2955] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Committing config change with OpId 28.16: config changed from index -1 to 16, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:06.108949  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:06.128468  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:06.146253 10354 ts_tablet_manager.cc:933] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:06.149847  1737 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:06.155925 10354 tablet_copy_client.cc:323] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:05:06.156049  1482 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 16, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 28 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 13:05:06.198644  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 4db9dd8c47ea465786759da787f2b1ab from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:05:06.199344  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 4db9dd8c47ea465786759da787f2b1ab from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-4db9dd8c47ea465786759da787f2b1ab
I20260709 13:05:06.213740  1677 tablet_copy_source_session.cc:215] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:05:06.226514 10354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4db9dd8c47ea465786759da787f2b1ab. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:06.227301  1706 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e74a6fd31c14c17a7248fe6eb410c77. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:06.230712  1662 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 13:05:06.234707 10298 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 13:05:06.239960  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:06.263422 10354 tablet_copy_client.cc:806] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:05:06.264410 10354 tablet_copy_client.cc:670] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:05:06.279170 10354 tablet_copy_client.cc:538] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:05:06.294235  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:06.300624 10331 raft_consensus.cc:2955] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Committing config change with OpId 23.12: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:05:06.308207  1662 raft_consensus.cc:2955] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Committing config change with OpId 23.12: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:06.335671  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:06.340221  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:06.361529  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:06.365365  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:06.376116 10354 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
W20260709 13:05:06.385088  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:06.419227  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:06.454226   431 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:06.465025  1482 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index -1 to 12, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 23 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 13:05:06.629423  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:06.628221  1739 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:05:06.652613  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:06.667485 10354 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:06.674006 10354 tablet_bootstrap.cc:492] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
W20260709 13:05:06.695955  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:06.678023 10354 ts_tablet_manager.cc:1403] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.302s	user 0.055s	sys 0.011s
W20260709 13:05:06.692040  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 47606.6us
W20260709 13:05:06.712023  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:06.722620  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:06.733675  1706 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): Couldn't send request to peer e6adef9da7da44f6af0af5695cf9b704. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a44e16fd1e74bb585aa62b3d15e7384. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:06.735098  1666 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 13:05:06.746846 10367 ts_tablet_manager.cc:933] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:06.754390 10354 raft_consensus.cc:359] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:05:06.759042  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:06.760092  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:06.755689 10354 raft_consensus.cc:740] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:05:06.772957 10367 tablet_copy_client.cc:323] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.194:37489
I20260709 13:05:06.777655  1677 tablet_copy_service.cc:140] P ff44a4643e01482cb6b6a79b099fa766: Received BeginTabletCopySession request for tablet 22ef4803b7a94ddcaacb46f38d689c5d from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:56492)
I20260709 13:05:06.778641  1677 tablet_copy_service.cc:161] P ff44a4643e01482cb6b6a79b099fa766: Beginning new tablet copy session on tablet 22ef4803b7a94ddcaacb46f38d689c5d from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:56492: session id = e6adef9da7da44f6af0af5695cf9b704-22ef4803b7a94ddcaacb46f38d689c5d
I20260709 13:05:06.782228 10354 consensus_queue.cc:260] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:06.828773 10354 ts_tablet_manager.cc:1434] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.130s	user 0.000s	sys 0.010s
I20260709 13:05:06.844445  1676 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-4db9dd8c47ea465786759da787f2b1ab received from {username='slave'} at 127.0.0.1:56492
I20260709 13:05:06.845149  1676 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-4db9dd8c47ea465786759da787f2b1ab on tablet 4db9dd8c47ea465786759da787f2b1ab with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:05:06.861402 10370 ts_tablet_manager.cc:933] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:06.882416  1634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 480 similar messages]
I20260709 13:05:06.888406 10362 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 13:05:06.894835  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:06.896421 10370 tablet_copy_client.cc:323] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:05:06.899322  1677 tablet_copy_source_session.cc:215] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:05:06.900182  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 4e74a6fd31c14c17a7248fe6eb410c77 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:05:06.901003  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 4e74a6fd31c14c17a7248fe6eb410c77 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-4e74a6fd31c14c17a7248fe6eb410c77
I20260709 13:05:06.904901 10367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22ef4803b7a94ddcaacb46f38d689c5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:06.936556  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:07.016582 10367 tablet_copy_client.cc:806] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:05:07.021463 10367 tablet_copy_client.cc:670] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:05:07.027145 10367 tablet_copy_client.cc:538] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:05:07.047856  1752 tablet_copy_source_session.cc:215] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 13:05:07.052830 10367 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:05:07.054276 10331 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:05:07.086414  1660 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Committing config change with OpId 18.11: config changed from index -1 to 11, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
W20260709 13:05:07.090444  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:07.097898 10370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e74a6fd31c14c17a7248fe6eb410c77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:05:07.126696  2310 raft_consensus.cc:1217] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEARNER]: Deduplicated request from leader. Original: 21.11->[21.12-21.12]   Dedup: 21.12->[]
W20260709 13:05:07.135648  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:07.153052  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.238920  1483 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index -1 to 11, NON_VOTER e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) added. New cstate: current_term: 18 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } } }
I20260709 13:05:07.319032 10354 ts_tablet_manager.cc:933] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Initiating tablet copy from peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:07.377315 10354 tablet_copy_client.cc:323] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Beginning tablet copy session from remote peer at address 127.0.107.195:33737
I20260709 13:05:07.379683 10370 tablet_copy_client.cc:806] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:05:07.391546  1752 tablet_copy_service.cc:140] P df6a18af06d543b2931837f5e6bab95f: Received BeginTabletCopySession request for tablet 5a44e16fd1e74bb585aa62b3d15e7384 from peer e6adef9da7da44f6af0af5695cf9b704 ({username='slave'} at 127.0.0.1:42028)
I20260709 13:05:07.392570  1752 tablet_copy_service.cc:161] P df6a18af06d543b2931837f5e6bab95f: Beginning new tablet copy session on tablet 5a44e16fd1e74bb585aa62b3d15e7384 from peer e6adef9da7da44f6af0af5695cf9b704 at {username='slave'} at 127.0.0.1:42028: session id = e6adef9da7da44f6af0af5695cf9b704-5a44e16fd1e74bb585aa62b3d15e7384
I20260709 13:05:07.413604 10370 tablet_copy_client.cc:670] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:05:07.429682  1752 tablet_copy_source_session.cc:215] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 13:05:07.437071  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.438530 10370 tablet_copy_client.cc:538] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 13:05:07.440819 10354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a44e16fd1e74bb585aa62b3d15e7384. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:05:07.455523   431 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:07.476343 10370 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
W20260709 13:05:07.486955  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:07.497042  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:07.519510  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.547155 10292 raft_consensus.cc:1064] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:05:07.552189 10292 consensus_queue.cc:237] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 10, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:07.574267  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:07.624119  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.645591 10367 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=4 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:07.646878 10367 tablet_bootstrap.cc:492] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:05:07.647910 10367 ts_tablet_manager.cc:1403] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.595s	user 0.058s	sys 0.029s
I20260709 13:05:07.653052 10367 raft_consensus.cc:359] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:07.666972 10367 raft_consensus.cc:740] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:05:07.667886 10367 consensus_queue.cc:260] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:07.700541 10367 ts_tablet_manager.cc:1434] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.052s	user 0.004s	sys 0.005s
I20260709 13:05:07.715078  1677 tablet_copy_service.cc:342] P ff44a4643e01482cb6b6a79b099fa766: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-22ef4803b7a94ddcaacb46f38d689c5d received from {username='slave'} at 127.0.0.1:56492
I20260709 13:05:07.715601  1677 tablet_copy_service.cc:434] P ff44a4643e01482cb6b6a79b099fa766: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-22ef4803b7a94ddcaacb46f38d689c5d on tablet 22ef4803b7a94ddcaacb46f38d689c5d with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:05:07.726742  2310 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:05:07.729863 10354 tablet_copy_client.cc:806] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 0 data blocks...
I20260709 13:05:07.730623 10354 tablet_copy_client.cc:670] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Starting download of 1 WAL segments...
I20260709 13:05:07.732431 10350 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:07.747077  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:07.765373  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:07.788017  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 52601.8us
I20260709 13:05:07.788836 10354 tablet_copy_client.cc:538] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 13:05:07.790151  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.830856  1739 raft_consensus.cc:1275] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 13:05:07.842041 10350 consensus_queue.cc:1048] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:07.851671  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:07.867748  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:07.891594  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:07.929282 10370 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:07.986903 10370 tablet_bootstrap.cc:492] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:05:07.988145 10370 ts_tablet_manager.cc:1403] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.520s	user 0.065s	sys 0.009s
I20260709 13:05:08.039779 10370 raft_consensus.cc:359] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:08.041288 10370 raft_consensus.cc:740] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:05:08.042385 10370 consensus_queue.cc:260] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
W20260709 13:05:08.070791  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:08.081957 10370 ts_tablet_manager.cc:1434] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.093s	user 0.010s	sys 0.000s
I20260709 13:05:08.084887 10354 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap starting.
I20260709 13:05:08.096167  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-4e74a6fd31c14c17a7248fe6eb410c77 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:05:08.096967  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-4e74a6fd31c14c17a7248fe6eb410c77 on tablet 4e74a6fd31c14c17a7248fe6eb410c77 with peer e6adef9da7da44f6af0af5695cf9b704
I20260709 13:05:08.101468 10361 raft_consensus.cc:2955] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Committing config change with OpId 21.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:08.109748  2310 raft_consensus.cc:1217] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 LEARNER]: Deduplicated request from leader. Original: 28.15->[28.16-28.16]   Dedup: 28.16->[]
I20260709 13:05:08.131757  2310 raft_consensus.cc:2955] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Committing config change with OpId 21.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:08.138455  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 62212.2us
W20260709 13:05:08.168769  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:08.173285  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 78000.7us
W20260709 13:05:08.283049  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.299690  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:08.323422  1482 catalog_manager.cc:5697] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 21 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:08.334764  1740 raft_consensus.cc:2955] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Committing config change with OpId 21.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:08.385870  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 63816us
W20260709 13:05:08.433365  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.438086  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.439241  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.440084  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.450626  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.451586  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.456095   431 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:08.457434  2310 raft_consensus.cc:1217] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEARNER]: Deduplicated request from leader. Original: 23.11->[23.12-23.12]   Dedup: 23.12->[]
W20260709 13:05:08.460366  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.496469  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:08.501293  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.543566  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:08.544092 10354 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=2 applied=11 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:05:08.545634 10354 tablet_bootstrap.cc:492] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Bootstrap complete.
I20260709 13:05:08.548970 10390 raft_consensus.cc:1064] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:05:08.555962 10390 consensus_queue.cc:237] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 14, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:08.564257 10354 ts_tablet_manager.cc:1403] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Time spent bootstrapping tablet: real 0.480s	user 0.055s	sys 0.004s
I20260709 13:05:08.578758 10354 raft_consensus.cc:359] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:08.581216 10354 raft_consensus.cc:740] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEARNER]: Becoming Follower/Learner. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Initialized, Role: LEARNER
I20260709 13:05:08.583843 10354 consensus_queue.cc:260] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: NON_VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: true } }
I20260709 13:05:08.596489  2315 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 LEARNER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 13:05:08.598507 10392 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:05:08.621385  1739 raft_consensus.cc:1275] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Refusing update from remote peer ff44a4643e01482cb6b6a79b099fa766: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 13:05:08.623646 10390 consensus_queue.cc:1048] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:05:08.638818 10354 ts_tablet_manager.cc:1434] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: Time spent starting tablet: real 0.072s	user 0.006s	sys 0.003s
W20260709 13:05:08.642398  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:08.649670  1752 tablet_copy_service.cc:342] P df6a18af06d543b2931837f5e6bab95f: Request end of tablet copy session e6adef9da7da44f6af0af5695cf9b704-5a44e16fd1e74bb585aa62b3d15e7384 received from {username='slave'} at 127.0.0.1:42028
I20260709 13:05:08.650238  1752 tablet_copy_service.cc:434] P df6a18af06d543b2931837f5e6bab95f: ending tablet copy session e6adef9da7da44f6af0af5695cf9b704-5a44e16fd1e74bb585aa62b3d15e7384 on tablet 5a44e16fd1e74bb585aa62b3d15e7384 with peer e6adef9da7da44f6af0af5695cf9b704
W20260709 13:05:08.677054  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:08.705384  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.679927  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:08.718869  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:08.765959  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:08.794243 10383 raft_consensus.cc:2955] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 LEADER]: Committing config change with OpId 28.17: config changed from index 16 to 17, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:08.803956  1739 raft_consensus.cc:2955] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Committing config change with OpId 28.17: config changed from index 16 to 17, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:08.879720 10391 raft_consensus.cc:1064] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:05:08.883517 10391 consensus_queue.cc:237] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 10, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:08.898491  2315 raft_consensus.cc:1217] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEARNER]: Deduplicated request from leader. Original: 18.10->[18.11-18.11]   Dedup: 18.11->[]
W20260709 13:05:08.909307  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 62033.1us
W20260709 13:05:08.940161  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:08.957082  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 81948.6us
I20260709 13:05:08.975312  1483 catalog_manager.cc:5697] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 reported cstate change: config changed from index 16 to 17, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 28 leader_uuid: "ff44a4643e01482cb6b6a79b099fa766" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 13:05:08.996702  2315 raft_consensus.cc:2955] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Committing config change with OpId 28.17: config changed from index 16 to 17, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:09.015321  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:09.113202  1660 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 13:05:09.115204 10368 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:09.126024  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.151139  2315 raft_consensus.cc:1275] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 13:05:09.151731  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 75614.8us
W20260709 13:05:09.152104  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:09.190501 10362 consensus_queue.cc:1048] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 13:05:09.244233  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.244944  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.253674  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:09.254365  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.258253  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:09.289855  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:09.338804  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:09.357518 10363 raft_consensus.cc:1064] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: attempting to promote NON_VOTER e6adef9da7da44f6af0af5695cf9b704 to VOTER
I20260709 13:05:09.363818 10404 raft_consensus.cc:2955] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Committing config change with OpId 23.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.361099 10363 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 9, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:09.392323  2315 raft_consensus.cc:2955] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Committing config change with OpId 23.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:09.422474  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.434559  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:09.437855  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 87625.4us
I20260709 13:05:09.444240  1660 raft_consensus.cc:2955] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Committing config change with OpId 23.13: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:09.456636   431 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 1614)
I20260709 13:05:09.469861  2310 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEARNER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:05:09.472191 10404 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 13:05:09.483446  1666 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 13:05:09.485323 10368 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 13:05:09.496556  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.497741  1708 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:09.493016  1484 catalog_manager.cc:5697] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 12 to 13, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 23 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:09.526904  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:09.527853  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:09.565258  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:09.639822 10404 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Committing config change with OpId 18.12: config changed from index 11 to 12, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.667398  2315 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Committing config change with OpId 18.12: config changed from index 11 to 12, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.684602  1662 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Committing config change with OpId 18.12: config changed from index 11 to 12, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.700325  1483 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 11 to 12, e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196) changed from NON_VOTER to VOTER. New cstate: current_term: 18 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:09.720799  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:09.786811  1735 consensus_queue.cc:237] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:09.807570  2315 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:05:09.815320 10368 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:05:09.824270  1660 raft_consensus.cc:1275] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Refusing update from remote peer df6a18af06d543b2931837f5e6bab95f: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 13:05:09.834376 10401 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.840214 10363 consensus_queue.cc:1048] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 13:05:09.848874  2315 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
I20260709 13:05:09.861197  1660 raft_consensus.cc:2955] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Committing config change with OpId 18.13: config changed from index 12 to 13, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } } }
W20260709 13:05:09.886498  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:09.919413  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:09.924657  1476 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 5a44e16fd1e74bb585aa62b3d15e7384 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 13:05:09.965283  1483 catalog_manager.cc:5697] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f reported cstate change: config changed from index 12 to 13, VOTER 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193) evicted. New cstate: current_term: 18 leader_uuid: "df6a18af06d543b2931837f5e6bab95f" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:10.038975  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:10.048723  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.137336  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:10.192018  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 54750.8us
W20260709 13:05:10.254911  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.259469  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 61921.3us
W20260709 13:05:10.274708  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:10.285585  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.281937  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.299522  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.361390  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.397940  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:10.400756  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.458112   431 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:10.803364  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.812412  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:10.827075  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:10.841427  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:10.853256  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:10.865355  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:10.875921  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:10.982857  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:10.961939  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.017372  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.042160  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.045998  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.117261  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.135283  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.148424  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:11.140151  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:11.177812  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.220114  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.244302  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.281833  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:11.330200  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:11.333048  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:11.458732   431 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:11.470719  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.498811  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:11.597759  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.629045  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.663591  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.691771  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:11.692556  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.729256  1632 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 60422us
W20260709 13:05:11.740379  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:11.762921  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.865885  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:11.886233  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:11.889185  1634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 492 similar messages]
W20260709 13:05:11.943120  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:11.947353  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:12.114362  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.164700  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:12.267059  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.341598  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.376392  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:12.404464  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:12.459735   431 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:12.538458  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.694295  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.798470  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 52750.8us
W20260709 13:05:12.801002  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:12.854820  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.855492  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:12.884461  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.933452  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:12.935712  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.068974  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.087118  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:13.134671  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:13.291046  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.305958  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.333873  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:13.382316  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:13.375029  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:13.403105  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.417670  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.460379   431 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:13.462580  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:13.491926  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.529366  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.547384  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.574446  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.756515  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.853817  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:13.871213  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.877446  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:13.884639  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.925773  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:13.947533  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:13.964411  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:13.995441  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:14.029309  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:14.030373  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:14.066066  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:14.192858  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:14.264765  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.294371  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.321527  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:14.461002   431 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:14.477726  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.484416  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:14.527138  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.537454  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:14.583302  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:14.613641  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.621434  1478 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 74061us
W20260709 13:05:14.645911  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:14.643090  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:14.729789  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:14.749329  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:14.877637  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:14.966789  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.039377  1706 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:35615, user_credentials={real_user=slave}} blocked reactor thread for 67139.4us
W20260709 13:05:15.200184  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:15.227504  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 48363.2us
W20260709 13:05:15.282153  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.305583  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:15.311718  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.415195  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.461597   431 thread.cc:527] Waited for 13000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:15.502297  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.567844  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.664431  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:15.716146  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.730443  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:15.781872  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.789443  1634 consensus_peers.cc:597] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:15.844748  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:15.965344  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:15.993072  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:15.993943  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.008057  1634 consensus_peers.cc:597] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:16.067293  1634 consensus_peers.cc:597] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:16.102504  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.106046  1634 consensus_peers.cc:597] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.106639  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.121299  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:16.216538  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:16.251106  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.321488  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.332275  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:16.369395  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:16.462314   431 thread.cc:527] Waited for 14000ms trying to join with diag-logger (tid 1614)
W20260709 13:05:16.537657  1634 consensus_peers.cc:597] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:16.545213  1634 consensus_peers.cc:597] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:16.565304  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.592682  1634 consensus_peers.cc:597] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:16.605825  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.619212  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:16.877485  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 51569.7us
W20260709 13:05:16.893275  2283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 473 similar messages]
W20260709 13:05:16.924901  1634 consensus_peers.cc:597] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:16.946038  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 55428us
W20260709 13:05:16.923403  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:16.982393  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.003666  1634 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 53322.3us
I20260709 13:05:17.048972   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:05:17.075052  1634 consensus_peers.cc:597] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:17.078656  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:17.102896  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 81364.8us
W20260709 13:05:17.104534  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:17.114043  1634 consensus_peers.cc:597] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:17.056559  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.129088  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 72570.4us
W20260709 13:05:17.134158  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:17.134207  1634 consensus_peers.cc:597] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:17.142023  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:17.150629  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:17.149716  1634 consensus_peers.cc:597] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.276505  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:17.286765  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.353711  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:17.669663   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:05:17.682260  1705 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.734956  1705 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.746124  1705 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.750782  1705 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.754132  2280 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.766651  1705 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.781415  1705 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.791464  1705 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.797538  1705 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.807766  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:17.831000  1705 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.861778  2280 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.865571  1634 consensus_peers.cc:597] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:17.899137  1705 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.907392  2280 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.915624  1705 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.916611   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:05:17.918591  1705 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.920274   431 tablet_replica.cc:333] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:17.921700   431 raft_consensus.cc:2243] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 16 LEADER]: Raft consensus shutting down.
I20260709 13:05:17.923516   431 raft_consensus.cc:2272] T e223adb74e71405ea633f364ffa8b6db P ff44a4643e01482cb6b6a79b099fa766 [term 16 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:17.923570  1705 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.939947  1705 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.947000  2280 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.947331   431 tablet_replica.cc:333] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:17.948648   431 raft_consensus.cc:2243] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
W20260709 13:05:17.948789  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:17.952333   431 raft_consensus.cc:2272] T 567113a84c1d465597e948caf9276c34 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:17.963145  2280 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.966176  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.967922  1634 consensus_peers.cc:597] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:17.974601  2280 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.977572   431 tablet_replica.cc:333] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:17.980497   431 raft_consensus.cc:2243] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 LEADER]: Raft consensus shutting down.
W20260709 13:05:17.983664  2280 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.984853   431 raft_consensus.cc:2272] T 5ae371f2c53648c2992a835477d85184 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:17.988680  1634 consensus_peers.cc:597] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:17.988958   431 tablet_replica.cc:333] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
W20260709 13:05:17.991120  1705 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:17.992904  1705 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:17.993698   431 raft_consensus.cc:2243] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:17.995908   431 raft_consensus.cc:2272] T 01669b686f2848abb4ec03f67aa748ea P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:17.999631   431 tablet_replica.cc:333] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.000805   431 raft_consensus.cc:2243] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.001951   431 raft_consensus.cc:2272] T 7dc3659523cf47fa8918889f5129c8ba P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.006531   431 tablet_replica.cc:333] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.007638   431 raft_consensus.cc:2243] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.008666   431 raft_consensus.cc:2272] T 9a860695ab4c493b83dfcd08d8bee8de P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.012352   431 tablet_replica.cc:333] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.014047   431 raft_consensus.cc:2243] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.014787   431 raft_consensus.cc:2272] T 0621716cd2fa4c0cb66038edb4352f37 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.020210   431 tablet_replica.cc:333] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.021411   431 raft_consensus.cc:2243] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.022456   431 raft_consensus.cc:2272] T 882fdd7ea10640a78914e0647598ad9b P ff44a4643e01482cb6b6a79b099fa766 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.027956   431 tablet_replica.cc:333] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
W20260709 13:05:18.031832  1705 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:18.034981  2280 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:18.036432   431 raft_consensus.cc:2243] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.037618   431 raft_consensus.cc:2272] T d857fff084d3451d99a39db56ccd9a37 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.059034  2280 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:18.061733   431 tablet_replica.cc:333] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.088840   431 raft_consensus.cc:2243] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.091311   431 raft_consensus.cc:2272] T 88e78eac7b5f4006a15379f62bc8c536 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.099071  1705 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:18.111255   431 tablet_replica.cc:333] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
W20260709 13:05:18.118206  1705 consensus_peers.cc:597] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:18.129877  1705 consensus_peers.cc:597] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:18.124015   431 raft_consensus.cc:2243] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.142833   431 raft_consensus.cc:2272] T 422771fe34aa4b29ba2702785d38559a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.166435  1705 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:18.177398  1634 consensus_peers.cc:597] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:18.180287   431 tablet_replica.cc:333] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.182286   431 raft_consensus.cc:2243] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.194273   431 raft_consensus.cc:2272] T 648ab3b5826e49fe81c3363d752d8af8 P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.241204  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:18.245769  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:18.252611   431 tablet_replica.cc:333] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.253801   431 raft_consensus.cc:2243] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.255110   431 raft_consensus.cc:2272] T 93491efa22a449f3aa27354cf02d7047 P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.261824   431 tablet_replica.cc:333] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.262998   431 raft_consensus.cc:2243] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.272585   431 raft_consensus.cc:2272] T dfe67343bf1842888d6176ed2ffe1419 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.281440  2280 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:18.282253   431 tablet_replica.cc:333] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.283524   431 raft_consensus.cc:2243] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.284626   431 raft_consensus.cc:2272] T 4e74a6fd31c14c17a7248fe6eb410c77 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.287729   431 tablet_replica.cc:333] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
W20260709 13:05:18.294561  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:18.295215  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:18.303759   431 raft_consensus.cc:2243] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.304956   431 raft_consensus.cc:2272] T 110eba2eb7b34ea1b70ed63aca2feda8 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.310153   431 tablet_replica.cc:333] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.311324   431 raft_consensus.cc:2243] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.312126   431 raft_consensus.cc:2272] T 200c02cbe43c4af8b4c59f21f210a9bc P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.320125  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:18.340590  1634 consensus_peers.cc:597] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:18.344558  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 13:05:18.369422   431 tablet_replica.cc:333] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.370713   431 raft_consensus.cc:2243] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.371904   431 raft_consensus.cc:2272] T 0444300de6924432844fa6519c92b570 P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.378937   431 tablet_replica.cc:333] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.379997   431 raft_consensus.cc:2243] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.380765   431 raft_consensus.cc:2272] T 2dffe36aa52848e8962262b0e01bc346 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.388365  1634 consensus_peers.cc:597] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:18.390957   431 tablet_replica.cc:333] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.392009   431 raft_consensus.cc:2243] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.392982   431 raft_consensus.cc:2272] T 162373eeadc2444e87ffd67df6a836e5 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.397262   431 tablet_replica.cc:333] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.398537   431 raft_consensus.cc:2243] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.400074   431 raft_consensus.cc:2272] T 8029bb31c40d4c01bebc7c65c2301a85 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.403465   431 tablet_replica.cc:333] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.404435   431 raft_consensus.cc:2243] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.405119   431 raft_consensus.cc:2272] T b9c6fca959014bb6a785fac033e5fe47 P ff44a4643e01482cb6b6a79b099fa766 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.411131   431 tablet_replica.cc:333] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.412050   431 raft_consensus.cc:2243] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.414031   431 raft_consensus.cc:2272] T 4db9dd8c47ea465786759da787f2b1ab P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.417287   431 tablet_replica.cc:333] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.418242   431 raft_consensus.cc:2243] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus shutting down.
W20260709 13:05:18.436725  1634 consensus_peers.cc:597] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:18.444074   431 raft_consensus.cc:2272] T bf77b84dc20e4118ad47baa1808a85d9 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.447528   431 tablet_replica.cc:333] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.448504   431 raft_consensus.cc:2243] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.449891   431 raft_consensus.cc:2272] T 8a654178d0194a7388155af12a626d35 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.452926   431 tablet_replica.cc:333] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.454025   431 raft_consensus.cc:2243] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.455469   431 raft_consensus.cc:2272] T 278074adc39c4830a2791129a7815557 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.459583  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:18.466074   431 tablet_replica.cc:333] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.467849   431 raft_consensus.cc:2243] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.469319   431 raft_consensus.cc:2272] T 3d40d5b254644257a6d39df949366e1b P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.483596   431 tablet_replica.cc:333] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.484988   431 raft_consensus.cc:2243] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.486723   431 raft_consensus.cc:2272] T eed985f1202940a1ab59ea7b7b3c745a P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.491202   431 tablet_replica.cc:333] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.492297   431 raft_consensus.cc:2243] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.494228   431 raft_consensus.cc:2272] T 22ef4803b7a94ddcaacb46f38d689c5d P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.497752   431 tablet_replica.cc:333] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.498788   431 raft_consensus.cc:2243] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.500427   431 raft_consensus.cc:2272] T f5a2ef920c894818b430473b4809d61d P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.503595   431 tablet_replica.cc:333] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.504472   431 raft_consensus.cc:2243] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.505236   431 raft_consensus.cc:2272] T 421876549c8f408587a97543338bca1d P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.508597   431 tablet_replica.cc:333] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.509475   431 raft_consensus.cc:2243] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.515900   431 raft_consensus.cc:2272] T e165279fd0cb4e84a8c9e244d087f8db P ff44a4643e01482cb6b6a79b099fa766 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.519114   431 tablet_replica.cc:333] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.520136   431 raft_consensus.cc:2243] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.520797   431 raft_consensus.cc:2272] T c9e456da7a864a1b97fb4c468e75980d P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.523761   431 tablet_replica.cc:333] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.524802   431 raft_consensus.cc:2243] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.526211   431 raft_consensus.cc:2272] T d66c6d05a0074fe2a7b7b2466f6182d7 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.529692   431 tablet_replica.cc:333] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.530648   431 raft_consensus.cc:2243] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.531441   431 raft_consensus.cc:2272] T 24ec87af7f274132adaaf5e59ef12b7b P ff44a4643e01482cb6b6a79b099fa766 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.535276   431 tablet_replica.cc:333] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.536408   431 raft_consensus.cc:2243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.537853   431 raft_consensus.cc:2272] T 026ea522d0f44f06bfab3c6eddc5e0b3 P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.541400   431 tablet_replica.cc:333] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.542421   431 raft_consensus.cc:2243] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.543201   431 raft_consensus.cc:2272] T fbe322e4447d486680550d1490266b17 P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.546612   431 tablet_replica.cc:333] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.550242   431 raft_consensus.cc:2243] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.556653   431 raft_consensus.cc:2272] T 249474541a174b90a099d77784ed4801 P ff44a4643e01482cb6b6a79b099fa766 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.564374   431 tablet_replica.cc:333] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.568562   431 raft_consensus.cc:2243] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.579449   431 raft_consensus.cc:2272] T 5a44e16fd1e74bb585aa62b3d15e7384 P ff44a4643e01482cb6b6a79b099fa766 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.583288   431 tablet_replica.cc:333] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.584540   431 raft_consensus.cc:2243] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.586078   431 raft_consensus.cc:2272] T 9768bad280eb4cdf8f1a090752fc5ddc P ff44a4643e01482cb6b6a79b099fa766 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.592634   431 tablet_replica.cc:333] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.593621   431 raft_consensus.cc:2243] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.596741   431 raft_consensus.cc:2272] T 87a64e3accd242188d2126865daef114 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.624985   431 tablet_replica.cc:333] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.626034   431 raft_consensus.cc:2243] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.626896   431 raft_consensus.cc:2272] T 9f303f4bc5e84b46b9bca2a44911578b P ff44a4643e01482cb6b6a79b099fa766 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.631285   431 tablet_replica.cc:333] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.632189   431 raft_consensus.cc:2243] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.633890   431 raft_consensus.cc:2272] T d4817dbc3c2e469195aa3f5e05a1534a P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.641109   431 tablet_replica.cc:333] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.641995   431 raft_consensus.cc:2243] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.642764   431 raft_consensus.cc:2272] T 6c15d2efd3c74bc9a5e8d3a23d50cced P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.645749   431 tablet_replica.cc:333] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.646524   431 raft_consensus.cc:2243] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.647279   431 raft_consensus.cc:2272] T 337669b46a6b4f75a35716542d1e20ae P ff44a4643e01482cb6b6a79b099fa766 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.650295   431 tablet_replica.cc:333] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.651087   431 raft_consensus.cc:2243] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.652429   431 raft_consensus.cc:2272] T 5a57c404b2c145bf95efa4a0649c0e33 P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.664697   431 tablet_replica.cc:333] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.665743   431 raft_consensus.cc:2243] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.666504   431 raft_consensus.cc:2272] T b8e75da90507436ea52296522f44c15e P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.669687   431 tablet_replica.cc:333] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.670481   431 raft_consensus.cc:2243] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.671247   431 raft_consensus.cc:2272] T e676e54141f7470db4551f4de439c017 P ff44a4643e01482cb6b6a79b099fa766 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.674315   431 tablet_replica.cc:333] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.675079   431 raft_consensus.cc:2243] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:18.676765   431 raft_consensus.cc:2272] T a9c383ed9b7c4624b8fa546326e129ff P ff44a4643e01482cb6b6a79b099fa766 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.680203   431 tablet_replica.cc:333] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.680987   431 raft_consensus.cc:2243] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:18.681639   431 raft_consensus.cc:2272] T c89f1823e773445c99646fed221320df P ff44a4643e01482cb6b6a79b099fa766 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:18.684656   431 tablet_replica.cc:333] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766: stopping tablet replica
I20260709 13:05:18.685492   431 raft_consensus.cc:2243] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 LEADER]: Raft consensus shutting down.
W20260709 13:05:18.683290  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:18.686774   431 raft_consensus.cc:2272] T bb611776ed864c47b9f843e0746b3683 P ff44a4643e01482cb6b6a79b099fa766 [term 21 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:18.734081  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:18.810838  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:18.837602  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:18.846746  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:18.947474 10457 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:18.948274 10457 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:18.960888  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:18.961259 10457 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:18.965148  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:18.982810 10457 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:18.983610 10457 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:18.991751  1705 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:18.992339  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:18.993439 10464 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 13:05:18.995069 10457 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.000917  1705 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:19.011060  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.014961  1708 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.017817 10464 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:05:19.024014  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:19.047709  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:19.111896  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:19.169775  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:19.184684 10457 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.185392 10457 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:19.206130  1705 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.207928 10457 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.209365  1708 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.222638  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.230276 10464 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:19.239178  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:19.261492  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:19.409394 10457 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.410025 10457 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.412221 10457 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.426879  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:19.431566  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.445781 10438 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.449805 10438 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.457537 10438 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.454440  1705 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.476306  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.477401 10458 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:19.491079 10458 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.491845 10458 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.494975 10458 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:19.498761 10460 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.507309 10460 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:19.475337  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:19.499675  2280 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.519359  1735 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
W20260709 13:05:19.542984  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.543713  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.548189 10438 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:05:19.550423  1705 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.564312 10460 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:19.576571 10458 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.577411 10458 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.583532 10457 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.584250 10457 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.586587 10457 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:19.593466 10460 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.594323 10460 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.602669  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:19.602998 10460 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:19.603436  1740 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 21.
I20260709 13:05:19.606896 10458 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:19.588145  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.616367  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:19.623596  2280 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.624171  2280 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:19.625979  2283 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.630173 10438 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 13:05:19.633581 10457 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.634187 10457 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.634814 10472 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:05:19.636616  1708 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 48508.1us
W20260709 13:05:19.637161  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:19.642526  2283 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.645563  2310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:19.646554  2310 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:05:19.645790  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:19.648895 10457 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:19.652522  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:19.661556  1705 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:19.662277  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.662367  1705 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.664906  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:19.665767  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.666201  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.667114 10458 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:19.668151 10458 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:05:19.691835  2280 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.692483  2280 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:19.693800  1705 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.694344  1705 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: e6adef9da7da44f6af0af5695cf9b704, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.695729 10457 raft_consensus.cc:2749] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:19.697044 10460 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:19.701457 10472 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.702416 10472 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.701809 10464 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.705788 10464 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.708288 10464 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:19.704628 10457 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.709034 10457 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.711628 10457 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:19.714591 10472 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.722533  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:19.723054  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.723544  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.727421  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:19.728812  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.732983  1705 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.733559  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.734344 10472 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:19.735483  1705 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.737623  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.740284 10472 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:05:19.741536  1705 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.742185  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.746243 10472 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:19.769009 10472 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.769662 10472 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.771948 10472 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.779892  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.783588 10457 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.784335 10457 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.788928  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:19.789683  2315 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
I20260709 13:05:19.794289 10457 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:19.800029  1708 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.801780 10472 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.802363 10472 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.804752 10472 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.806396  1705 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.806874  1705 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.813930 10472 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:05:19.823303  1705 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.823930  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:19.824764  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.829401  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:19.830271 10472 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:05:19.831741  1705 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.832211  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.833004 10472 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:05:19.841674 10460 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.842438 10460 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:19.845570 10472 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.846204 10472 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:19.848503 10472 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.854161  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:19.859771  2280 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.861404 10460 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:19.866019   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1689)
W20260709 13:05:19.869992  1705 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.870682  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.871507 10457 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:19.871945  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:19.872515  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.873951 10460 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:19.927778  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:19.958756 10460 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.959362 10460 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:19.975068  2280 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:19.980608  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f" is_pre_election: true
I20260709 13:05:19.981316  1740 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 24.
I20260709 13:05:19.983121  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:19.984422 10438 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 13:05:19.984854 10438 raft_consensus.cc:493] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting leader election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:19.985224 10438 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:20.014401 10472 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.015206 10472 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:20.024587 10472 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:20.045035  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:20.051012  1705 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:20.051733  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:20.054167 10460 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:20.055284 10460 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.056866 10464 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:20.142985  1705 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:37489, user_credentials={real_user=slave}} blocked reactor thread for 65364.9us
I20260709 13:05:20.143654 10438 raft_consensus.cc:515] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:20.144387  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 77971.3us
I20260709 13:05:20.155272 10438 leader_election.cc:290] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Requested vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:20.167860  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:20.176927 10464 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.177649 10464 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:20.193001 10464 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:20.157022 10460 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:20.196928  1708 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:20.198833 10460 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:20.207394  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:20.210840  2280 leader_election.cc:336] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:20.211578  2280 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.217209  2280 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:20.217908  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:20.219027 10460 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:20.219069  1740 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4817dbc3c2e469195aa3f5e05a1534a" candidate_uuid: "e6adef9da7da44f6af0af5695cf9b704" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "df6a18af06d543b2931837f5e6bab95f"
I20260709 13:05:20.219892  1740 raft_consensus.cc:3060] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Advancing to term 25
I20260709 13:05:20.230530  1740 raft_consensus.cc:2468] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6adef9da7da44f6af0af5695cf9b704 in term 25.
I20260709 13:05:20.236807  2281 leader_election.cc:304] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:20.239882  1705 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:20.240528  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:20.239959 10460 raft_consensus.cc:2804] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 13:05:20.242024 10460 raft_consensus.cc:697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Becoming Leader. State: Replica: e6adef9da7da44f6af0af5695cf9b704, State: Running, Role: LEADER
I20260709 13:05:20.243098 10460 consensus_queue.cc:237] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:20.244542 10480 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:20.260721  2280 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.276400  1705 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.277122  2280 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.280732  1705 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.285269  1705 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.294143  2280 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.302179  1705 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.304354  1705 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.314244  1705 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.316550  1705 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:20.335685  1483 catalog_manager.cc:5697] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 reported cstate change: term changed from 24 to 25, leader changed from ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194) to e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196). New cstate: current_term: 25 leader_uuid: "e6adef9da7da44f6af0af5695cf9b704" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 13:05:20.340467  1705 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.341681  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:20.343803  1705 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.425800  1705 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.440910  2280 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.451749  2280 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.473219  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:20.475273  1705 consensus_peers.cc:597] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.483145  1705 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.484179  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:20.485543  1705 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.505779  1705 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.527176  1705 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.584288  2280 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.604548  1705 consensus_peers.cc:597] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.632236  1705 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.647446  1705 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.650358  1705 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.653549  2280 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:20.662009 10480 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.662763 10480 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:20.666492  1735 raft_consensus.cc:1275] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Refusing update from remote peer e6adef9da7da44f6af0af5695cf9b704: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 13:05:20.669554 10460 consensus_queue.cc:1048] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 13:05:20.682158 10480 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:20.685168  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:20.694183  1705 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:20.694876  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:20.701588 10481 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 13:05:20.708891  1705 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.711803  2280 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.712949  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:20.732491  2280 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.740922  1705 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:20.776610  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:20.778147  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 13:05:20.779251 10481 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.779959 10481 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:20.783758 10481 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:20.800122  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:20.836621  1705 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:20.849253  2280 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:20.851555  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:20.857405 10464 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:05:20.866619   431 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1689)
W20260709 13:05:20.878000  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:20.893338  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:20.935240  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:20.947494  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:20.991770 10481 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:20.992461 10481 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:20.995206 10481 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.001555  1705 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.003563  1708 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.004038  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.007349 10464 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:21.035251  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:21.262395  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:21.298336 10464 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.299166 10464 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:21.308550  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.310990 10464 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:21.320515  1705 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.320567  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:21.321712  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:21.321714  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.322579  2315 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
I20260709 13:05:21.323890 10464 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:21.385656 10480 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.386407 10480 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:21.399274 10480 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:21.402809  1705 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:21.404569  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:21.412746  2315 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
W20260709 13:05:21.426800  1708 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.427531  1708 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.428607 10460 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.430195 10460 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:21.432911 10460 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.434414  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:21.434984 10480 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:05:21.446120  2280 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.447224  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.447721  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.448825 10460 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:05:21.453375  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:21.482342 10464 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.482988 10464 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.485499 10464 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.499198  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:21.502341  1705 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:21.505053  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:21.509403  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:21.510102 10464 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 13:05:21.524554 10464 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.525415 10464 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.528380 10464 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.537788  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:21.550395  1705 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:21.551013  1705 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:21.551641  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:21.553731 10481 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:21.567610 10481 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.568303 10481 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.578814 10481 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:21.579550 10464 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.581923 10464 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.584882 10464 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.566200  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:21.599216  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:21.601207  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:21.601830  1705 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:21.602780  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.613458 10464 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:05:21.615024  1705 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:21.615651  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.616616 10464 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:21.634106 10464 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.634960 10464 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:21.654253  1705 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.655293  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.655748  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.656904 10481 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:05:21.658825 10464 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:21.688244 10464 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.689399 10464 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.692871 10464 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:21.687480 10481 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.694675 10481 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.698765 10481 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.723941  1705 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.728788  1705 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.729143  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.730903  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:21.732263  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.736528  1708 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.732852 10481 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:05:21.739799 10481 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:21.760424  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 13:05:21.811929  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:21.856709  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:21.861157  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:21.867172   431 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1689)
I20260709 13:05:21.886050 10481 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.886765 10481 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:21.889321 10481 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.896881  1708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 697 similar messages]
W20260709 13:05:21.905105  1705 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.906248  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.906819  1708 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.907789 10464 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 13:05:21.951495 10464 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:21.952847 10464 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:21.955221  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:21.960278 10464 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:21.975646  1705 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:21.978448  1708 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:21.979504  1708 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:21.989434 10464 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:22.017238  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:22.043260 10461 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:22.043973 10461 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:22.051553 10464 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:22.052273 10464 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:22.054535  2280 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:22.054858 10464 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:22.055356 10461 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:22.059473  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:22.060212  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:22.061615 10461 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 13:05:22.063937  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:22.075434  1705 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:22.076563  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:22.084882 10464 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:22.121275 10464 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:22.121922 10464 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:22.124312 10464 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:22.132977  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:22.135897  1705 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:22.138836  1705 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:22.139879 10481 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:22.177871  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:22.205605  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:22.240981  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:22.245070  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 13:05:22.339157 10461 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:22.339843 10461 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:22.342247 10461 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:22.348806  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:22.385578  2280 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:22.389282  2280 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:22.397850 10461 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:22.435364  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:22.561168  2281 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33737, user_credentials={real_user=slave}} blocked reactor thread for 61536.8us
W20260709 13:05:22.627754  2280 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.670687  1705 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.673075  2280 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.750525  1705 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.752715  2280 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.780596  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:22.782145  1705 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.786551  1705 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.797489  2280 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.799924  2280 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.801973  1705 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.817080  2280 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.818861  1705 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.841913  1705 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.874959   431 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1689)
W20260709 13:05:22.910995  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:22.911891  1705 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.942590  1705 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.953142  1705 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:22.980111  1705 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:22.997787 10461 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:22.998368 10461 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:23.021965 10461 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:23.027719  2280 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:23.027719  1705 consensus_peers.cc:597] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.039769  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.040472  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.041973 10461 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:05:23.078588  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.094496  1705 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.115715  1705 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.116741  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:23.115816  2280 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.120751  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:23.128145  1705 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:23.136282 10464 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.137048 10464 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:23.142195 10464 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:23.142789  2280 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.145319  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:23.151527  1705 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:23.152110  1705 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.153239 10481 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:23.153851  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:23.160619  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:23.161382  2315 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
W20260709 13:05:23.166162  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 13:05:23.174491  2280 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.180629  1705 consensus_peers.cc:597] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.204947  1705 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.228428  2280 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:23.236552  1705 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.243304  1705 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.254864  1705 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:05:23.269237 10464 raft_consensus.cc:493] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.269898 10464 raft_consensus.cc:515] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:23.272138 10464 leader_election.cc:290] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:23.277673  1708 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:23.297695  2280 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.325525  1705 leader_election.cc:336] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:23.326191  1705 leader_election.cc:304] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.327008 10481 raft_consensus.cc:2749] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 13:05:23.365955 10464 raft_consensus.cc:493] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.366694 10464 raft_consensus.cc:515] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:23.376075  1705 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:23.378762 10464 leader_election.cc:290] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:23.380534  1708 leader_election.cc:336] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.381480  1708 leader_election.cc:304] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.383055 10464 raft_consensus.cc:2749] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:23.414479  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.433728  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.438699  1705 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:23.448988  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:23.486936  1708 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.504613  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:23.583884  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:23.648860 10481 raft_consensus.cc:493] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.649513 10481 raft_consensus.cc:515] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:23.651916 10481 leader_election.cc:290] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:23.660085  1705 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:23.661118  1708 leader_election.cc:336] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.661541  1708 leader_election.cc:304] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.662789 10464 raft_consensus.cc:2749] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 13:05:23.829916 10481 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.830765 10481 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:23.845230  2309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22ef4803b7a94ddcaacb46f38d689c5d" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
W20260709 13:05:23.845700  1708 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.846100  2309 raft_consensus.cc:2468] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 28.
I20260709 13:05:23.850400 10493 raft_consensus.cc:493] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.850626 10481 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
I20260709 13:05:23.851122 10493 raft_consensus.cc:515] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:23.852792 10481 raft_consensus.cc:493] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:23.853387 10481 raft_consensus.cc:515] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:23.856817 10493 leader_election.cc:290] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:23.873818 10481 leader_election.cc:290] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:23.875969  1705 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:23.876132   431 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1689)
W20260709 13:05:23.878350  1708 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.877249  1705 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f, e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.881544 10481 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:05:23.886003  1705 leader_election.cc:336] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:23.886191  1708 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:23.886588  1705 leader_election.cc:304] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.888260 10481 raft_consensus.cc:2749] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:23.902380  1708 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.903478  1705 leader_election.cc:336] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:23.904119  1705 leader_election.cc:304] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:23.905700 10481 raft_consensus.cc:2749] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:05:23.925684  1708 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.927423  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:23.948396  1708 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.963593  1708 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:23.993453  1708 consensus_peers.cc:597] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:24.156087  1708 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:24.173923 10493 raft_consensus.cc:493] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.174643 10493 raft_consensus.cc:515] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.177030 10493 leader_election.cc:290] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:24.183621 10493 raft_consensus.cc:493] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:05:24.184299  1708 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.184391 10493 raft_consensus.cc:515] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.187188 10493 leader_election.cc:290] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.203015  1705 leader_election.cc:336] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.207444  1705 leader_election.cc:304] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:24.202071  1708 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.211434 10481 raft_consensus.cc:2749] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 13:05:24.244352  1705 leader_election.cc:336] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.245054  1705 leader_election.cc:304] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.246423 10481 raft_consensus.cc:2749] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:24.276298  1708 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:24.304420  1708 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 13:05:24.327998  1708 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:24.364177  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:24.367280 10481 raft_consensus.cc:493] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.368017 10481 raft_consensus.cc:515] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
W20260709 13:05:24.381363  1708 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.386390 10481 leader_election.cc:290] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.393880  1705 leader_election.cc:336] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.394443  1705 leader_election.cc:304] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.397289 10493 raft_consensus.cc:2749] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:05:24.403100 10492 raft_consensus.cc:493] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.403916 10492 raft_consensus.cc:515] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:24.422772  2280 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.426604 10492 leader_election.cc:290] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.431623  2283 leader_election.cc:336] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.432992  2283 leader_election.cc:304] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.437815 10492 raft_consensus.cc:2749] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 13:05:24.514655  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:24.577595  1708 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 13:05:24.671980  2283 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:41267, user_credentials={real_user=slave}} blocked reactor thread for 45213.8us
W20260709 13:05:24.676662  1708 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 13:05:24.707239  1708 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:24.763888 10494 raft_consensus.cc:493] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.764686 10494 raft_consensus.cc:515] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.768257 10494 leader_election.cc:290] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.771871  1708 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:24.785218  1705 leader_election.cc:336] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.785938  1705 leader_election.cc:304] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.786847 10494 raft_consensus.cc:2749] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 13:05:24.811029 10481 raft_consensus.cc:493] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.811637 10481 raft_consensus.cc:515] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.813823 10481 leader_election.cc:290] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.818317  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:24.825582  1708 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:24.827147  1705 leader_election.cc:336] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.827708  1705 leader_election.cc:304] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.828613 10481 raft_consensus.cc:2749] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:24.868232  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:05:24.897681 10481 raft_consensus.cc:493] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.898533 10481 raft_consensus.cc:515] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.900674 10481 leader_election.cc:290] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:24.905032 10492 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
W20260709 13:05:24.905306  1708 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.905864 10492 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:24.906550  1705 leader_election.cc:336] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.908633 10492 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:24.909240  1705 leader_election.cc:304] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.910441 10481 raft_consensus.cc:2749] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:05:24.919253  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:24.921447  2280 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.922644  2280 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.927268 10481 raft_consensus.cc:493] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.927533 10492 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:24.927892 10481 raft_consensus.cc:515] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.930066 10481 leader_election.cc:290] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.936362  1705 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:24.949640 10481 raft_consensus.cc:493] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:24.950637 10481 raft_consensus.cc:515] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:24.952673 10481 leader_election.cc:290] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:24.961863  2280 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:24.962054  1705 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:24.979314  1708 leader_election.cc:336] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.980113  1708 leader_election.cc:304] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.981000 10481 raft_consensus.cc:2749] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:24.990298  1708 leader_election.cc:336] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:24.992158   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 13:05:24.992458  1708 leader_election.cc:304] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:24.995683 10481 raft_consensus.cc:2749] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 13:05:25.012547 10481 raft_consensus.cc:493] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:25.013166 10481 raft_consensus.cc:515] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } }
I20260709 13:05:25.015625 10481 leader_election.cc:290] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:25.027741  1705 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:25.071249  1708 leader_election.cc:336] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:25.072541  1708 leader_election.cc:304] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:25.074575 10481 raft_consensus.cc:2749] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 13:05:25.127081  2280 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.133477  1705 consensus_peers.cc:597] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.146641  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:25.251220  1705 consensus_peers.cc:597] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.263319  1705 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.278832  1705 consensus_peers.cc:597] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.280066  1705 consensus_peers.cc:597] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.288213  2280 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.289664  1705 consensus_peers.cc:597] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.298373  2280 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.315116  2280 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.349700  2280 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.392201  1705 consensus_peers.cc:597] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.412911   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 13:05:25.438787  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:05:25.439985 10492 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:25.440622 10492 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:25.449321  2281 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.456379  2280 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:25.457109  2283 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:25.458122 10492 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:25.457729  2283 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:25.463414  2281 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 13:05:25.464303 10492 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:05:25.464546  1705 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.470805  2281 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.493067  1705 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.493801  1705 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.522948  1705 consensus_peers.cc:597] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.527976  2281 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.543462  1705 consensus_peers.cc:597] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.548982  2281 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.564579  1705 consensus_peers.cc:597] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.566128  2281 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.562487  2280 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.571435  1705 consensus_peers.cc:597] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.601836  2281 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.604489  1705 consensus_peers.cc:597] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.605674  2281 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.625883  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:25.631134  2281 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.644145  2280 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.633877  2281 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.646106  2281 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.670523  1705 consensus_peers.cc:597] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.673854  1708 consensus_peers.cc:597] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 13:05:25.677636 10464 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:25.682271 10464 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:25.685338 10464 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615)
W20260709 13:05:25.693116  1705 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:25.695562  2315 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4db9dd8c47ea465786759da787f2b1ab" candidate_uuid: "df6a18af06d543b2931837f5e6bab95f" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "e6adef9da7da44f6af0af5695cf9b704" is_pre_election: true
I20260709 13:05:25.700273  2315 raft_consensus.cc:2468] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df6a18af06d543b2931837f5e6bab95f in term 21.
W20260709 13:05:25.707116  1708 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:25.707679  1708 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: df6a18af06d543b2931837f5e6bab95f; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:25.711799 10464 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:25.713342  2280 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:05:25.713896  2280 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.736735   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 13:05:25.737072  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:25.740360   431 tablet_replica.cc:333] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.741473   431 raft_consensus.cc:2243] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.742244   431 raft_consensus.cc:2272] T ce1eb834927c4d3caa029344d6f69de5 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.746268   431 tablet_replica.cc:333] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.751910   431 raft_consensus.cc:2243] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.753374   431 raft_consensus.cc:2272] T 01669b686f2848abb4ec03f67aa748ea P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.760185   431 tablet_replica.cc:333] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.761150   431 raft_consensus.cc:2243] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.761891   431 raft_consensus.cc:2272] T e223adb74e71405ea633f364ffa8b6db P df6a18af06d543b2931837f5e6bab95f [term 16 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:25.762984  2281 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.772042  1708 consensus_peers.cc:597] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 13:05:25.778528  1705 consensus_peers.cc:597] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.780721   431 tablet_replica.cc:333] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.781953   431 raft_consensus.cc:2243] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
W20260709 13:05:25.791405  2281 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:05:25.794521  1705 consensus_peers.cc:597] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.798024   431 raft_consensus.cc:2272] T 567113a84c1d465597e948caf9276c34 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.801298   431 tablet_replica.cc:333] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.802304   431 raft_consensus.cc:2243] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.803083   431 raft_consensus.cc:2272] T 5ae371f2c53648c2992a835477d85184 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.806504   431 tablet_replica.cc:333] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.807412   431 raft_consensus.cc:2243] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.808665   431 raft_consensus.cc:2272] T 9a860695ab4c493b83dfcd08d8bee8de P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.811534   431 tablet_replica.cc:333] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.812842   431 raft_consensus.cc:2243] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.813865   431 raft_consensus.cc:2272] T 882fdd7ea10640a78914e0647598ad9b P df6a18af06d543b2931837f5e6bab95f [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:25.819329  1708 consensus_peers.cc:597] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:25.823186   431 tablet_replica.cc:333] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
W20260709 13:05:25.825402  2280 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.826267   431 raft_consensus.cc:2243] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.827322   431 raft_consensus.cc:2272] T ce79fa9e7595481c8379fd2bdeca8a6e P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:25.828742  2281 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:25.830323   431 tablet_replica.cc:333] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.831334   431 raft_consensus.cc:2243] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
W20260709 13:05:25.842451  1705 consensus_peers.cc:597] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:25.844134  1708 consensus_peers.cc:597] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 13:05:25.833423   431 raft_consensus.cc:2272] T 0621716cd2fa4c0cb66038edb4352f37 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.848229   431 tablet_replica.cc:333] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.849136   431 raft_consensus.cc:2243] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.849785   431 raft_consensus.cc:2272] T e7f90c9079e24cf2a4c9ab3dd55172ea P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.852190   431 tablet_replica.cc:333] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.852804   431 raft_consensus.cc:2243] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.853724   431 raft_consensus.cc:2272] T dfe67343bf1842888d6176ed2ffe1419 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.856310   431 tablet_replica.cc:333] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.856920   431 raft_consensus.cc:2243] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.857468   431 raft_consensus.cc:2272] T d857fff084d3451d99a39db56ccd9a37 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.859858   431 tablet_replica.cc:333] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.860584   431 raft_consensus.cc:2243] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.861212   431 raft_consensus.cc:2272] T 422771fe34aa4b29ba2702785d38559a P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.863613   431 tablet_replica.cc:333] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.864398   431 raft_consensus.cc:2243] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.865113   431 raft_consensus.cc:2272] T 648ab3b5826e49fe81c3363d752d8af8 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.869481   431 tablet_replica.cc:333] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.871517   431 raft_consensus.cc:2243] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.873147   431 raft_consensus.cc:2272] T 93491efa22a449f3aa27354cf02d7047 P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.879379   431 tablet_replica.cc:333] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.880422   431 raft_consensus.cc:2243] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.883378   431 raft_consensus.cc:2272] T 4e74a6fd31c14c17a7248fe6eb410c77 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:25.884748  1705 consensus_peers.cc:597] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:25.886591   431 tablet_replica.cc:333] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.887584   431 raft_consensus.cc:2243] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.888276   431 raft_consensus.cc:2272] T 110eba2eb7b34ea1b70ed63aca2feda8 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.895406   431 tablet_replica.cc:333] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.896399   431 raft_consensus.cc:2243] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Raft consensus shutting down.
W20260709 13:05:25.898263  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 13:05:25.899344   431 raft_consensus.cc:2272] T 200c02cbe43c4af8b4c59f21f210a9bc P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.903707   431 tablet_replica.cc:333] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.905189   431 raft_consensus.cc:2243] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.906190   431 raft_consensus.cc:2272] T 92fb459de10a4e038d6cbee7b88dc3ba P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.909780   431 tablet_replica.cc:333] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.910742   431 raft_consensus.cc:2243] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.912825   431 raft_consensus.cc:2272] T 0444300de6924432844fa6519c92b570 P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.916646   431 tablet_replica.cc:333] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.917724   431 raft_consensus.cc:2243] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.918550   431 raft_consensus.cc:2272] T 8029bb31c40d4c01bebc7c65c2301a85 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.922107   431 tablet_replica.cc:333] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.923024   431 raft_consensus.cc:2243] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.923952   431 raft_consensus.cc:2272] T b9c6fca959014bb6a785fac033e5fe47 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.927551   431 tablet_replica.cc:333] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.928491   431 raft_consensus.cc:2243] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.929857   431 raft_consensus.cc:2272] T 7f9cc5e1ffa141e295028cdd523da631 P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.932991   431 tablet_replica.cc:333] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.933934   431 raft_consensus.cc:2243] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.934773   431 raft_consensus.cc:2272] T 3d40d5b254644257a6d39df949366e1b P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.937976   431 tablet_replica.cc:333] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.938854   431 raft_consensus.cc:2243] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.939641   431 raft_consensus.cc:2272] T eed985f1202940a1ab59ea7b7b3c745a P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.942598   431 tablet_replica.cc:333] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.943459   431 raft_consensus.cc:2243] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.944397   431 raft_consensus.cc:2272] T 22ef4803b7a94ddcaacb46f38d689c5d P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.947543   431 tablet_replica.cc:333] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.948426   431 raft_consensus.cc:2243] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.950258   431 raft_consensus.cc:2272] T 162373eeadc2444e87ffd67df6a836e5 P df6a18af06d543b2931837f5e6bab95f [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.953521   431 tablet_replica.cc:333] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.954394   431 raft_consensus.cc:2243] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.955353   431 raft_consensus.cc:2272] T 8a654178d0194a7388155af12a626d35 P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.959141   431 tablet_replica.cc:333] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.960289   431 raft_consensus.cc:2243] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.961141   431 raft_consensus.cc:2272] T 278074adc39c4830a2791129a7815557 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:25.964017  1708 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 13:05:25.964896   431 tablet_replica.cc:333] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.966589   431 raft_consensus.cc:2243] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.968226   431 raft_consensus.cc:2272] T 24ec87af7f274132adaaf5e59ef12b7b P df6a18af06d543b2931837f5e6bab95f [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.976100   431 tablet_replica.cc:333] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.977106   431 raft_consensus.cc:2243] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.978237   431 raft_consensus.cc:2272] T 7a5331c1cc5e4cbf8847232f0d5835c7 P df6a18af06d543b2931837f5e6bab95f [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.988547   431 tablet_replica.cc:333] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.989699   431 raft_consensus.cc:2243] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:25.990931   431 raft_consensus.cc:2272] T d66c6d05a0074fe2a7b7b2466f6182d7 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:25.994736   431 tablet_replica.cc:333] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:25.995796   431 raft_consensus.cc:2243] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:25.997495   431 raft_consensus.cc:2272] T f5a2ef920c894818b430473b4809d61d P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.000828   431 tablet_replica.cc:333] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.001739   431 raft_consensus.cc:2243] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.003255   431 raft_consensus.cc:2272] T 421876549c8f408587a97543338bca1d P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:26.004688  1705 consensus_peers.cc:597] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 13:05:26.006510   431 tablet_replica.cc:333] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
W20260709 13:05:26.014470  2281 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:05:26.022223   431 raft_consensus.cc:2243] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.023604   431 raft_consensus.cc:2272] T c9e456da7a864a1b97fb4c468e75980d P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.031944   431 tablet_replica.cc:333] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.032946   431 raft_consensus.cc:2243] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.034271   431 raft_consensus.cc:2272] T bf77b84dc20e4118ad47baa1808a85d9 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:26.049127  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 13:05:26.038802   431 tablet_replica.cc:333] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.055240   431 raft_consensus.cc:2243] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.062702   431 raft_consensus.cc:2272] T 4db9dd8c47ea465786759da787f2b1ab P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.066921   431 tablet_replica.cc:333] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.067860   431 raft_consensus.cc:2243] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.068679   431 raft_consensus.cc:2272] T 873c3a32540d4c2cb6d882298cb1eedc P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.072520   431 tablet_replica.cc:333] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.073459   431 raft_consensus.cc:2243] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.074326   431 raft_consensus.cc:2272] T 9768bad280eb4cdf8f1a090752fc5ddc P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.077913   431 tablet_replica.cc:333] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.078960   431 raft_consensus.cc:2243] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.080367   431 raft_consensus.cc:2272] T 026ea522d0f44f06bfab3c6eddc5e0b3 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.085096   431 tablet_replica.cc:333] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.086052   431 raft_consensus.cc:2243] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.087487   431 raft_consensus.cc:2272] T 249474541a174b90a099d77784ed4801 P df6a18af06d543b2931837f5e6bab95f [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.091974   431 tablet_replica.cc:333] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.092854   431 raft_consensus.cc:2243] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.094259   431 raft_consensus.cc:2272] T 5a44e16fd1e74bb585aa62b3d15e7384 P df6a18af06d543b2931837f5e6bab95f [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.099407   431 tablet_replica.cc:333] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.100363   431 raft_consensus.cc:2243] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.101452   431 raft_consensus.cc:2272] T d4817dbc3c2e469195aa3f5e05a1534a P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.105077   431 tablet_replica.cc:333] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.106077   431 raft_consensus.cc:2243] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.112707   431 raft_consensus.cc:2272] T 6c15d2efd3c74bc9a5e8d3a23d50cced P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.115803   431 tablet_replica.cc:333] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.116621   431 raft_consensus.cc:2243] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.117362   431 raft_consensus.cc:2272] T 219129ac643e4650b611a85f565f5990 P df6a18af06d543b2931837f5e6bab95f [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.120318   431 tablet_replica.cc:333] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.120992   431 raft_consensus.cc:2243] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.122560   431 raft_consensus.cc:2272] T c89f1823e773445c99646fed221320df P df6a18af06d543b2931837f5e6bab95f [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.129858   431 tablet_replica.cc:333] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.130617   431 raft_consensus.cc:2243] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.131731   431 raft_consensus.cc:2272] T bb611776ed864c47b9f843e0746b3683 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.134614   431 tablet_replica.cc:333] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.135164   431 raft_consensus.cc:2243] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.136333   431 raft_consensus.cc:2272] T 87a64e3accd242188d2126865daef114 P df6a18af06d543b2931837f5e6bab95f [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.138564   431 tablet_replica.cc:333] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.139179   431 raft_consensus.cc:2243] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 LEADER]: Raft consensus shutting down.
I20260709 13:05:26.140120   431 raft_consensus.cc:2272] T 9f303f4bc5e84b46b9bca2a44911578b P df6a18af06d543b2931837f5e6bab95f [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.142175   431 tablet_replica.cc:333] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.142746   431 raft_consensus.cc:2243] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.143327   431 raft_consensus.cc:2272] T e676e54141f7470db4551f4de439c017 P df6a18af06d543b2931837f5e6bab95f [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.145658   431 tablet_replica.cc:333] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.146252   431 raft_consensus.cc:2243] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.146817   431 raft_consensus.cc:2272] T 337669b46a6b4f75a35716542d1e20ae P df6a18af06d543b2931837f5e6bab95f [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:26.149010   431 tablet_replica.cc:333] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f: stopping tablet replica
I20260709 13:05:26.149668   431 raft_consensus.cc:2243] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:26.150437   431 raft_consensus.cc:2272] T 5a57c404b2c145bf95efa4a0649c0e33 P df6a18af06d543b2931837f5e6bab95f [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 13:05:26.205276  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:26.217845  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:26.778075  2283 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:26.877660  2283 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:26.917447  2281 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111) [suppressed 648 similar messages]
W20260709 13:05:27.034916  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:27.079622 10495 raft_consensus.cc:493] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:05:27.081185 10495 raft_consensus.cc:515] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.089179 10495 leader_election.cc:290] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:27.095717  2280 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:27.097127  2283 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:27.097674  2283 leader_election.cc:304] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.098610 10495 raft_consensus.cc:2749] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:05:27.109546  2281 leader_election.cc:336] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.136551 10495 raft_consensus.cc:493] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:05:27.137574 10495 raft_consensus.cc:515] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } attrs { replace: true } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.140949 10495 leader_election.cc:290] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:27.148852  2283 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:27.152591  2280 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:27.153168  2280 leader_election.cc:304] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
W20260709 13:05:27.154495  2281 leader_election.cc:336] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.155169 10495 raft_consensus.cc:2749] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 13:05:27.221938  2283 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:27.262913  2283 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:27.262966   431 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1763)
I20260709 13:05:27.313535 10496 raft_consensus.cc:493] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:05:27.314429 10496 raft_consensus.cc:515] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.320050 10496 leader_election.cc:290] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:27.322928  2281 leader_election.cc:336] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
W20260709 13:05:27.323551  2280 leader_election.cc:336] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:27.324012  2280 leader_election.cc:304] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.324908 10496 raft_consensus.cc:2749] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 13:05:27.370369 10495 raft_consensus.cc:493] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:05:27.371313 10495 raft_consensus.cc:515] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.379580 10496 raft_consensus.cc:493] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:05:27.380075  2283 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:27.381479 10495 leader_election.cc:290] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:27.391299 10505 raft_consensus.cc:493] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:27.389854 10496 raft_consensus.cc:515] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.392819 10505 raft_consensus.cc:515] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.396344 10505 leader_election.cc:290] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
I20260709 13:05:27.400079 10495 raft_consensus.cc:493] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
W20260709 13:05:27.402598  2280 leader_election.cc:336] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:27.403299  2280 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:27.403352 10495 raft_consensus.cc:515] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
W20260709 13:05:27.405277  2283 leader_election.cc:336] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:27.405773  2283 leader_election.cc:304] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.406684 10505 raft_consensus.cc:493] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader df6a18af06d543b2931837f5e6bab95f)
I20260709 13:05:27.407609 10505 raft_consensus.cc:515] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.395426 10496 leader_election.cc:290] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:27.414803 10506 raft_consensus.cc:2749] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 13:05:27.415009 10495 leader_election.cc:290] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:27.417912  2281 leader_election.cc:336] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.420554  2281 leader_election.cc:304] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:05:27.420861  2280 leader_election.cc:336] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:27.421710 10495 raft_consensus.cc:2749] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 13:05:27.422659  2281 leader_election.cc:336] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.423128  2281 leader_election.cc:304] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.423823 10505 leader_election.cc:290] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
I20260709 13:05:27.424079 10496 raft_consensus.cc:2749] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 13:05:27.425359  2280 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:27.431439  2281 leader_election.cc:336] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.431970  2281 leader_election.cc:304] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.432740 10506 raft_consensus.cc:2749] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:27.433461  2281 leader_election.cc:336] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.433974  2281 leader_election.cc:304] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: df6a18af06d543b2931837f5e6bab95f, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:27.434958 10506 raft_consensus.cc:2749] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 13:05:27.531328  2280 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.652371  2283 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:27.730247  2280 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.745383  2280 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.751266  2280 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 13:05:27.786903 10505 raft_consensus.cc:493] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:27.787938 10505 raft_consensus.cc:515] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:27.791534 10505 leader_election.cc:290] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:27.799566  2283 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:27.800593  2281 consensus_peers.cc:597] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:27.801196  2281 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
I20260709 13:05:27.801669  2281 leader_election.cc:304] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, df6a18af06d543b2931837f5e6bab95f
W20260709 13:05:27.802763  2283 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:27.806154  2280 leader_election.cc:336] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
I20260709 13:05:27.806696 10506 raft_consensus.cc:2749] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 13:05:27.829448  2280 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.861207  2281 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:27.870488  2280 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.887102  2281 consensus_peers.cc:597] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:27.942546  2280 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:27.943437  2281 consensus_peers.cc:597] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:27.975328  2281 consensus_peers.cc:597] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:27.978037  2281 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.005297  2281 consensus_peers.cc:597] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.016641  2280 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 13:05:28.027324  2281 consensus_peers.cc:597] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.049631  2283 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 13:05:28.059775  2280 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:28.118286  2283 consensus_peers.cc:597] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:28.141598  2281 consensus_peers.cc:597] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.194798  2281 consensus_peers.cc:597] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.195297  2283 consensus_peers.cc:597] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 13:05:28.237727  2281 consensus_peers.cc:597] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.242262  2281 consensus_peers.cc:597] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:05:28.308058 10505 raft_consensus.cc:493] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:28.309253 10505 raft_consensus.cc:515] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "df6a18af06d543b2931837f5e6bab95f" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33737 } attrs { replace: true } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:28.314266 10505 leader_election.cc:290] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489), df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737)
W20260709 13:05:28.319569  2280 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:28.320601  2283 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:28.322770  2283 leader_election.cc:304] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:28.324584 10506 raft_consensus.cc:2749] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 13:05:28.325337  2281 leader_election.cc:336] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111)
W20260709 13:05:28.329254  2281 consensus_peers.cc:597] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.331130  2280 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:28.361366  2280 consensus_peers.cc:597] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 -> Peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Couldn't send request to peer ff44a4643e01482cb6b6a79b099fa766. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 13:05:28.384918  2281 consensus_peers.cc:597] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 -> Peer df6a18af06d543b2931837f5e6bab95f (127.0.107.195:33737): Couldn't send request to peer df6a18af06d543b2931837f5e6bab95f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33737: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:05:28.481240  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:28.519165  2283 consensus_peers.cc:597] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 13:05:28.581146   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
W20260709 13:05:28.596421  2283 consensus_peers.cc:597] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 13:05:28.667786  2283 consensus_peers.cc:597] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 -> Peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Couldn't send request to peer 316f3bf18ed94e32b238130a6bfedf1a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 13:05:28.782034 10507 raft_consensus.cc:493] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader ff44a4643e01482cb6b6a79b099fa766)
I20260709 13:05:28.782835   431 tablet_server.cc:179] TabletServer@127.0.107.196:0 shutting down...
I20260709 13:05:28.782879 10507 raft_consensus.cc:515] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "316f3bf18ed94e32b238130a6bfedf1a" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 41267 } } peers { permanent_uuid: "ff44a4643e01482cb6b6a79b099fa766" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 37489 } } peers { permanent_uuid: "e6adef9da7da44f6af0af5695cf9b704" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 35615 } attrs { promote: false } }
I20260709 13:05:28.786139 10507 leader_election.cc:290] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267), ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489)
W20260709 13:05:28.795104  2280 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer ff44a4643e01482cb6b6a79b099fa766 (127.0.107.194:37489): Network error: Client connection negotiation failed: client connection to 127.0.107.194:37489: connect: Connection refused (error 111)
W20260709 13:05:28.796922  2283 leader_election.cc:336] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:05:28.797374  2283 leader_election.cc:304] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6adef9da7da44f6af0af5695cf9b704; no voters: 316f3bf18ed94e32b238130a6bfedf1a, ff44a4643e01482cb6b6a79b099fa766
I20260709 13:05:28.798197 10507 raft_consensus.cc:2749] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 13:05:28.850224   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:05:28.851104   431 tablet_replica.cc:333] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.852711   431 raft_consensus.cc:2243] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.854142   431 raft_consensus.cc:2272] T 22ef4803b7a94ddcaacb46f38d689c5d P e6adef9da7da44f6af0af5695cf9b704 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.858739   431 tablet_replica.cc:333] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.859597   431 raft_consensus.cc:2243] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.860975   431 raft_consensus.cc:2272] T dfe67343bf1842888d6176ed2ffe1419 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.864218   431 tablet_replica.cc:333] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.865026   431 raft_consensus.cc:2243] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.866498   431 raft_consensus.cc:2272] T 110eba2eb7b34ea1b70ed63aca2feda8 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.869467   431 tablet_replica.cc:333] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.870307   431 raft_consensus.cc:2243] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.871189   431 raft_consensus.cc:2272] T 882fdd7ea10640a78914e0647598ad9b P e6adef9da7da44f6af0af5695cf9b704 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.873881   431 tablet_replica.cc:333] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.874591   431 raft_consensus.cc:2243] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.875880   431 raft_consensus.cc:2272] T 7dc3659523cf47fa8918889f5129c8ba P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.878773   431 tablet_replica.cc:333] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.879528   431 raft_consensus.cc:2243] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.880409   431 raft_consensus.cc:2272] T 7f9cc5e1ffa141e295028cdd523da631 P e6adef9da7da44f6af0af5695cf9b704 [term 19 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.883734   431 tablet_replica.cc:333] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.884691   431 raft_consensus.cc:2243] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.885864   431 raft_consensus.cc:2272] T 162373eeadc2444e87ffd67df6a836e5 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.889708   431 tablet_replica.cc:333] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.890530   431 raft_consensus.cc:2243] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.891862   431 raft_consensus.cc:2272] T 92fb459de10a4e038d6cbee7b88dc3ba P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.895725   431 tablet_replica.cc:333] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.896674   431 raft_consensus.cc:2243] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.898341   431 raft_consensus.cc:2272] T 2dffe36aa52848e8962262b0e01bc346 P e6adef9da7da44f6af0af5695cf9b704 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.901304   431 tablet_replica.cc:333] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.902009   431 raft_consensus.cc:2243] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.903259   431 raft_consensus.cc:2272] T d4817dbc3c2e469195aa3f5e05a1534a P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.906067   431 tablet_replica.cc:333] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.906929   431 raft_consensus.cc:2243] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.907823   431 raft_consensus.cc:2272] T 4e74a6fd31c14c17a7248fe6eb410c77 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.910985   431 tablet_replica.cc:333] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.911568   431 raft_consensus.cc:2243] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.912267   431 raft_consensus.cc:2272] T c9e456da7a864a1b97fb4c468e75980d P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.915132   431 tablet_replica.cc:333] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.915743   431 raft_consensus.cc:2243] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.916244   431 raft_consensus.cc:2272] T f5a2ef920c894818b430473b4809d61d P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.918856   431 tablet_replica.cc:333] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.919466   431 raft_consensus.cc:2243] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.920979   431 raft_consensus.cc:2272] T b9c6fca959014bb6a785fac033e5fe47 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.924350   431 tablet_replica.cc:333] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.925333   431 raft_consensus.cc:2243] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.926648   431 raft_consensus.cc:2272] T 873c3a32540d4c2cb6d882298cb1eedc P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.929265   431 tablet_replica.cc:333] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.930037   431 raft_consensus.cc:2243] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.931164   431 raft_consensus.cc:2272] T 5a44e16fd1e74bb585aa62b3d15e7384 P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.934962   431 tablet_replica.cc:333] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.935621   431 raft_consensus.cc:2243] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.936859   431 raft_consensus.cc:2272] T fbe322e4447d486680550d1490266b17 P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.939961   431 tablet_replica.cc:333] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.940702   431 raft_consensus.cc:2243] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 17 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.942685   431 raft_consensus.cc:2272] T e676e54141f7470db4551f4de439c017 P e6adef9da7da44f6af0af5695cf9b704 [term 17 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.945775   431 tablet_replica.cc:333] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.946379   431 raft_consensus.cc:2243] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 24 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.947919   431 raft_consensus.cc:2272] T b8e75da90507436ea52296522f44c15e P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.950640   431 tablet_replica.cc:333] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.951259   431 raft_consensus.cc:2243] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.951764   431 raft_consensus.cc:2272] T e165279fd0cb4e84a8c9e244d087f8db P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.954384   431 tablet_replica.cc:333] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.954936   431 raft_consensus.cc:2243] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.956449   431 raft_consensus.cc:2272] T 219129ac643e4650b611a85f565f5990 P e6adef9da7da44f6af0af5695cf9b704 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.959152   431 tablet_replica.cc:333] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.959966   431 raft_consensus.cc:2243] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.960990   431 raft_consensus.cc:2272] T 4db9dd8c47ea465786759da787f2b1ab P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.964427   431 tablet_replica.cc:333] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.965072   431 raft_consensus.cc:2243] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.965940   431 raft_consensus.cc:2272] T a9c383ed9b7c4624b8fa546326e129ff P e6adef9da7da44f6af0af5695cf9b704 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.968822   431 tablet_replica.cc:333] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.969554   431 raft_consensus.cc:2243] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.970887   431 raft_consensus.cc:2272] T d857fff084d3451d99a39db56ccd9a37 P e6adef9da7da44f6af0af5695cf9b704 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.973476   431 tablet_replica.cc:333] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.974165   431 raft_consensus.cc:2243] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.975342   431 raft_consensus.cc:2272] T 7a5331c1cc5e4cbf8847232f0d5835c7 P e6adef9da7da44f6af0af5695cf9b704 [term 20 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.978188   431 tablet_replica.cc:333] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.978751   431 raft_consensus.cc:2243] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.980448   431 raft_consensus.cc:2272] T e7f90c9079e24cf2a4c9ab3dd55172ea P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.983294   431 tablet_replica.cc:333] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.983908   431 raft_consensus.cc:2243] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.985088   431 raft_consensus.cc:2272] T ce1eb834927c4d3caa029344d6f69de5 P e6adef9da7da44f6af0af5695cf9b704 [term 21 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.987600   431 tablet_replica.cc:333] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.988173   431 raft_consensus.cc:2243] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.989333   431 raft_consensus.cc:2272] T ce79fa9e7595481c8379fd2bdeca8a6e P e6adef9da7da44f6af0af5695cf9b704 [term 18 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.992095   431 tablet_replica.cc:333] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.992666   431 raft_consensus.cc:2243] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 13:05:28.993221   431 raft_consensus.cc:2272] T 88e78eac7b5f4006a15379f62bc8c536 P e6adef9da7da44f6af0af5695cf9b704 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:28.995522   431 tablet_replica.cc:333] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704: stopping tablet replica
I20260709 13:05:28.995981   431 raft_consensus.cc:2243] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 LEADER]: Raft consensus shutting down.
I20260709 13:05:28.997237   431 raft_consensus.cc:2272] T 337669b46a6b4f75a35716542d1e20ae P e6adef9da7da44f6af0af5695cf9b704 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 13:05:29.467985   431 tablet_server.cc:196] TabletServer@127.0.107.196:0 shutdown complete.
I20260709 13:05:29.547995   431 master.cc:562] Master@127.0.107.254:39071 shutting down...
W20260709 13:05:29.602849  1539 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 13:05:29.603236  1539 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:05:29.603494  1539 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 13:05:43.441118  1478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 177 similar messages]
W20260709 13:05:43.445286  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5a44e16fd1e74bb585aa62b3d15e7384: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
W20260709 13:05:48.217703  1476 catalog_manager.cc:4755] TS e6adef9da7da44f6af0af5695cf9b704 (127.0.107.196:35615): ChangeConfig:REMOVE_PEER RPC failed for tablet e676e54141f7470db4551f4de439c017: Network error: Client connection negotiation failed: client connection to 127.0.107.196:35615: connect: Connection refused (error 111)
W20260709 13:06:01.280125  1478 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260709 13:06:01.284554  1478 catalog_manager.cc:4755] TS 316f3bf18ed94e32b238130a6bfedf1a (127.0.107.193:41267): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet dfe67343bf1842888d6176ed2ffe1419: Network error: Client connection negotiation failed: client connection to 127.0.107.193:41267: connect: Connection refused (error 111)
I20260709 13:06:02.785928   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:02.787374   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:02.787828   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P e6b16382849d482ba544fc5ee58e0b0a: stopping tablet replica
I20260709 13:06:02.975057   431 master.cc:584] Master@127.0.107.254:39071 shutdown complete.
I20260709 13:06:03.039328   431 test_util.cc:182] -----------------------------------------------
I20260709 13:06:03.039709   431 test_util.cc:183] Had failures, leaving test files at /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783602010504718-431-0
[  FAILED  ] LeaderRebalancerTest.AddTserver (325310 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 13:06:03.045145   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:39861
I20260709 13:06:03.046576   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:03.054229 10519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.054242 10520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.054804 10522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.056522   431 server_base.cc:1061] running on GCE node
I20260709 13:06:03.061625   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:03.061888   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:03.062012   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602363062001 us; error 0 us; skew 500 ppm
I20260709 13:06:03.062879   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:03.067391   431 webserver.cc:533] Webserver started at http://127.0.107.254:33593/ using document root <none> and password file <none>
I20260709 13:06:03.068277   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:03.068497   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:03.068748   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:03.070294   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "2c2db47040184744a819b7db4d1a589a"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.075868   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:06:03.079946 10527 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.081032   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:06:03.081755   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "2c2db47040184744a819b7db4d1a589a"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.082126   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:03.104578   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:03.106419   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:03.160348   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:39861
I20260709 13:06:03.160529 10578 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:39861 every 8 connection(s)
I20260709 13:06:03.165745 10579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:03.182456 10579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a: Bootstrap starting.
I20260709 13:06:03.189086 10579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:03.194867 10579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a: No bootstrap required, opened a new log
I20260709 13:06:03.197649 10579 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER }
I20260709 13:06:03.198194 10579 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:03.198472 10579 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2db47040184744a819b7db4d1a589a, State: Initialized, Role: FOLLOWER
I20260709 13:06:03.199172 10579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER }
I20260709 13:06:03.199679 10579 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:03.199908 10579 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:03.200227 10579 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:03.206019 10579 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER }
I20260709 13:06:03.206671 10579 leader_election.cc:304] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c2db47040184744a819b7db4d1a589a; no voters: 
I20260709 13:06:03.208914 10579 leader_election.cc:290] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:03.209164 10582 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:03.210979 10582 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 1 LEADER]: Becoming Leader. State: Replica: 2c2db47040184744a819b7db4d1a589a, State: Running, Role: LEADER
I20260709 13:06:03.211859 10582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER }
I20260709 13:06:03.213038 10579 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:03.220335 10584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c2db47040184744a819b7db4d1a589a. Latest consensus state: current_term: 1 leader_uuid: "2c2db47040184744a819b7db4d1a589a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER } }
I20260709 13:06:03.219988 10583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c2db47040184744a819b7db4d1a589a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c2db47040184744a819b7db4d1a589a" member_type: VOTER } }
I20260709 13:06:03.221220 10584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:03.221319 10583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:03.225394 10590 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:03.232218 10590 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:03.233923   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:03.243840 10590 catalog_manager.cc:1383] Generated new cluster ID: abbdec04cabc4c67a6904a5aa502d37c
I20260709 13:06:03.244251 10590 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:03.264321 10590 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:03.265950 10590 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:03.279735 10590 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a: Generated new TSK 0
I20260709 13:06:03.280510 10590 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:03.302202   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:03.309772 10600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.311249 10601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.313853 10603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.315361   431 server_base.cc:1061] running on GCE node
I20260709 13:06:03.318413   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:03.318706   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:03.318888   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602363318863 us; error 0 us; skew 500 ppm
I20260709 13:06:03.319490   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:03.322556   431 webserver.cc:533] Webserver started at http://127.0.107.193:43975/ using document root <none> and password file <none>
I20260709 13:06:03.323125   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:03.323357   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:03.323663   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:03.325111   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.330818   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:03.335132 10608 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.336176   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:06:03.336845   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.337208   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:03.365201   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:03.366657   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:03.368415   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:03.371078   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:03.371320   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.371580   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:03.371798   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.426962   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:46307
I20260709 13:06:03.427080 10670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:46307 every 8 connection(s)
I20260709 13:06:03.433099   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:03.442222 10675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.443472 10676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.447979 10678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.449466   431 server_base.cc:1061] running on GCE node
I20260709 13:06:03.452960   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:03.453207   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:03.453466   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602363453439 us; error 0 us; skew 500 ppm
I20260709 13:06:03.454319   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:03.456089 10671 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39861
I20260709 13:06:03.456516 10671 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:03.457477 10671 heartbeater.cc:507] Master 127.0.107.254:39861 requested a full tablet report, sending...
I20260709 13:06:03.457779   431 webserver.cc:533] Webserver started at http://127.0.107.194:36811/ using document root <none> and password file <none>
I20260709 13:06:03.458379   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:03.458607   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:03.458904   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:03.460131 10544 ts_manager.cc:194] Registered new tserver with Master: bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:03.460572   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "39f0ed0891e14f20bb25882450033da4"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.462586 10544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36822
I20260709 13:06:03.466224   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.002s
I20260709 13:06:03.470482 10683 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.471417   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:06:03.471835   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "39f0ed0891e14f20bb25882450033da4"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.472172   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:03.504927   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:03.506538   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:03.508572   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:03.511355   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:03.511586   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.511879   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:03.512058   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.564150   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:45783
I20260709 13:06:03.564263 10745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:45783 every 8 connection(s)
I20260709 13:06:03.570139   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:03.579583 10749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.582139 10750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.585247   431 server_base.cc:1061] running on GCE node
W20260709 13:06:03.586905 10752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.588238 10746 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39861
I20260709 13:06:03.588893 10746 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:03.588908   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:03.589383   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:03.589624   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602363589598 us; error 0 us; skew 500 ppm
I20260709 13:06:03.589934 10746 heartbeater.cc:507] Master 127.0.107.254:39861 requested a full tablet report, sending...
I20260709 13:06:03.590433   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:03.592538 10544 ts_manager.cc:194] Registered new tserver with Master: 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:03.593880   431 webserver.cc:533] Webserver started at http://127.0.107.195:44059/ using document root <none> and password file <none>
I20260709 13:06:03.594635   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:03.594722 10544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36836
I20260709 13:06:03.594925   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:03.595553   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:03.597259   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "7133813a08fe45908399f19d1e019ca6"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.602751   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:03.607115 10757 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.608134   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:06:03.608521   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "7133813a08fe45908399f19d1e019ca6"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.608894   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:03.650094   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:03.651701   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:03.653517   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:03.656347   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:03.656634   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.656917   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:03.657124   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.714056   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:33001
I20260709 13:06:03.714186 10819 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:33001 every 8 connection(s)
I20260709 13:06:03.719704   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:03.728358 10823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.730656 10824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:03.732084 10826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:03.732987   431 server_base.cc:1061] running on GCE node
I20260709 13:06:03.734237   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:03.734611   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:03.734828   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602363734811 us; error 0 us; skew 500 ppm
I20260709 13:06:03.735706   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:03.736428 10820 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39861
I20260709 13:06:03.736836 10820 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:03.737794 10820 heartbeater.cc:507] Master 127.0.107.254:39861 requested a full tablet report, sending...
I20260709 13:06:03.740196   431 webserver.cc:533] Webserver started at http://127.0.107.196:39225/ using document root <none> and password file <none>
I20260709 13:06:03.740835 10544 ts_manager.cc:194] Registered new tserver with Master: 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:03.740968   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:03.741384   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:03.741827   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:03.742985 10544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36844
I20260709 13:06:03.743616   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-3-root/instance:
uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.749742   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:06:03.753758 10831 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.754710   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:06:03.755061   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-3-root
uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:03.755335   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:03.779991   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:03.781714   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:03.784037   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:03.787216   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:03.787458   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.787655   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:03.787799   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:03.841938   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.196:36819
I20260709 13:06:03.842021 10893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.196:36819 every 8 connection(s)
I20260709 13:06:03.858821 10894 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39861
I20260709 13:06:03.859344 10894 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:03.860357 10894 heartbeater.cc:507] Master 127.0.107.254:39861 requested a full tablet report, sending...
I20260709 13:06:03.862845 10544 ts_manager.cc:194] Registered new tserver with Master: 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:03.863524   431 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01751876s
I20260709 13:06:03.864866 10544 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36860
I20260709 13:06:03.897694 10544 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36866:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 13:06:04.189251 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 97767de992584ed3962c38a716e7122a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:06:04.191202 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97767de992584ed3962c38a716e7122a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.195057 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 69a32458d0e54f08a475b354e480c787 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:06:04.196983 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a32458d0e54f08a475b354e480c787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.199203 10711 tablet_service.cc:1511] Processing CreateTablet for tablet 8b781a7ee3874351bfa70a5ab1452a6f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:06:04.200973 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b781a7ee3874351bfa70a5ab1452a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.202301 10636 tablet_service.cc:1511] Processing CreateTablet for tablet 8b781a7ee3874351bfa70a5ab1452a6f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:06:04.203888 10636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b781a7ee3874351bfa70a5ab1452a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.203694 10708 tablet_service.cc:1511] Processing CreateTablet for tablet 193ca6746d1344849958720afeb79325 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:06:04.206871 10635 tablet_service.cc:1511] Processing CreateTablet for tablet 97767de992584ed3962c38a716e7122a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:06:04.207432 10707 tablet_service.cc:1511] Processing CreateTablet for tablet 5a8f7860dfb8447a8960a50b1e087806 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:06:04.208424 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97767de992584ed3962c38a716e7122a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.210559 10706 tablet_service.cc:1511] Processing CreateTablet for tablet 410064a793614346a815b98b7c6d5316 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:06:04.210889 10634 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff22da804864a31992dc8a587b224f2 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:06:04.212611 10634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff22da804864a31992dc8a587b224f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.214092 10705 tablet_service.cc:1511] Processing CreateTablet for tablet c394fa4e36384c6eb804d29258eb7ec1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:06:04.216022 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193ca6746d1344849958720afeb79325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.216114 10633 tablet_service.cc:1511] Processing CreateTablet for tablet 193ca6746d1344849958720afeb79325 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:06:04.217679 10859 tablet_service.cc:1511] Processing CreateTablet for tablet 8b781a7ee3874351bfa70a5ab1452a6f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 13:06:04.215821 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff22da804864a31992dc8a587b224f2 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:06:04.217891 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 69a32458d0e54f08a475b354e480c787 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:06:04.219102 10707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8f7860dfb8447a8960a50b1e087806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.219265 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b781a7ee3874351bfa70a5ab1452a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.220273 10854 tablet_service.cc:1511] Processing CreateTablet for tablet 410064a793614346a815b98b7c6d5316 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:06:04.220876 10631 tablet_service.cc:1511] Processing CreateTablet for tablet d43a3401e55d4d8687de1b5dad8edc84 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:06:04.222443 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a32458d0e54f08a475b354e480c787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.223485 10856 tablet_service.cc:1511] Processing CreateTablet for tablet 5a8f7860dfb8447a8960a50b1e087806 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:06:04.225114 10856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8f7860dfb8447a8960a50b1e087806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.226300 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410064a793614346a815b98b7c6d5316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.226542 10853 tablet_service.cc:1511] Processing CreateTablet for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:06:04.226819 10855 tablet_service.cc:1511] Processing CreateTablet for tablet d43a3401e55d4d8687de1b5dad8edc84 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:06:04.228104 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.229067 10630 tablet_service.cc:1511] Processing CreateTablet for tablet c394fa4e36384c6eb804d29258eb7ec1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:06:04.230123 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c394fa4e36384c6eb804d29258eb7ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.218860 10632 tablet_service.cc:1511] Processing CreateTablet for tablet 5a8f7860dfb8447a8960a50b1e087806 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 13:06:04.231765 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193ca6746d1344849958720afeb79325. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:04.232625 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:04.234009 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a8f7860dfb8447a8960a50b1e087806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.234222 10706 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410064a793614346a815b98b7c6d5316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.234019 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff22da804864a31992dc8a587b224f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.235378 10631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43a3401e55d4d8687de1b5dad8edc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.235641 10855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43a3401e55d4d8687de1b5dad8edc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.236773 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c394fa4e36384c6eb804d29258eb7ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.289415 10704 tablet_service.cc:1511] Processing CreateTablet for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:06:04.291911 10704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.294065 10629 tablet_service.cc:1511] Processing CreateTablet for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 13:06:04.310379 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f27b1950c54af2b8d48d3e4c7cb7e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.382226 10913 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:04.436879 10785 tablet_service.cc:1511] Processing CreateTablet for tablet 97767de992584ed3962c38a716e7122a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 13:06:04.439298 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97767de992584ed3962c38a716e7122a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.439551 10784 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff22da804864a31992dc8a587b224f2 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 13:06:04.441231 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff22da804864a31992dc8a587b224f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.437402 10783 tablet_service.cc:1511] Processing CreateTablet for tablet 69a32458d0e54f08a475b354e480c787 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 13:06:04.445647 10783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a32458d0e54f08a475b354e480c787. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.451257 10782 tablet_service.cc:1511] Processing CreateTablet for tablet 193ca6746d1344849958720afeb79325 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 13:06:04.453101 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 193ca6746d1344849958720afeb79325. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.455615 10781 tablet_service.cc:1511] Processing CreateTablet for tablet d43a3401e55d4d8687de1b5dad8edc84 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 13:06:04.456653 10778 tablet_service.cc:1511] Processing CreateTablet for tablet e46c52fd7feb4776839cc022f4ef27e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:06:04.458882 10780 tablet_service.cc:1511] Processing CreateTablet for tablet 410064a793614346a815b98b7c6d5316 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 13:06:04.461427 10779 tablet_service.cc:1511] Processing CreateTablet for tablet c394fa4e36384c6eb804d29258eb7ec1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 13:06:04.462890 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d43a3401e55d4d8687de1b5dad8edc84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.466246 10671 heartbeater.cc:499] Master 127.0.107.254:39861 was elected leader, sending a full tablet report...
I20260709 13:06:04.484412 10859 tablet_service.cc:1511] Processing CreateTablet for tablet e46c52fd7feb4776839cc022f4ef27e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:06:04.486059 10778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46c52fd7feb4776839cc022f4ef27e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.486150 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46c52fd7feb4776839cc022f4ef27e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.487371 10711 tablet_service.cc:1511] Processing CreateTablet for tablet 912716eee3434e46aead233d6f5a7c83 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:06:04.488962 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912716eee3434e46aead233d6f5a7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.494335 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c394fa4e36384c6eb804d29258eb7ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.495188 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 410064a793614346a815b98b7c6d5316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.508064 10635 tablet_service.cc:1511] Processing CreateTablet for tablet e46c52fd7feb4776839cc022f4ef27e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 13:06:04.509810 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e46c52fd7feb4776839cc022f4ef27e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.526808 10777 tablet_service.cc:1511] Processing CreateTablet for tablet 912716eee3434e46aead233d6f5a7c83 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:06:04.528806 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912716eee3434e46aead233d6f5a7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.536339 10914 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:04.547260 10913 tablet_bootstrap.cc:654] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.566749 10915 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:04.582885 10633 tablet_service.cc:1511] Processing CreateTablet for tablet a375c777784941d4a0945680b7406ed1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:06:04.595803 10916 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:04.598309 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375c777784941d4a0945680b7406ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.598531 10746 heartbeater.cc:499] Master 127.0.107.254:39861 was elected leader, sending a full tablet report...
I20260709 13:06:04.606709 10853 tablet_service.cc:1511] Processing CreateTablet for tablet 912716eee3434e46aead233d6f5a7c83 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 13:06:04.608381 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912716eee3434e46aead233d6f5a7c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.616452 10785 tablet_service.cc:1511] Processing CreateTablet for tablet a375c777784941d4a0945680b7406ed1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:06:04.618541 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375c777784941d4a0945680b7406ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.619411 10710 tablet_service.cc:1511] Processing CreateTablet for tablet a375c777784941d4a0945680b7406ed1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 13:06:04.620944 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a375c777784941d4a0945680b7406ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.619428 10854 tablet_service.cc:1511] Processing CreateTablet for tablet d50a168e65034266bc912284a133cb88 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:06:04.623598 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50a168e65034266bc912284a133cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.629798 10914 tablet_bootstrap.cc:654] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.632750 10916 tablet_bootstrap.cc:654] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.637061 10915 tablet_bootstrap.cc:654] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.639068 10629 tablet_service.cc:1511] Processing CreateTablet for tablet d50a168e65034266bc912284a133cb88 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:06:04.641315 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50a168e65034266bc912284a133cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.639029 10709 tablet_service.cc:1511] Processing CreateTablet for tablet d50a168e65034266bc912284a133cb88 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 13:06:04.646622 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d50a168e65034266bc912284a133cb88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.659976 10913 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:04.660696 10913 ts_tablet_manager.cc:1403] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.279s	user 0.013s	sys 0.005s
I20260709 13:06:04.672382 10781 tablet_service.cc:1511] Processing CreateTablet for tablet 8d56437e06784282b4cb42f93fa4094a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:06:04.674193 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d56437e06784282b4cb42f93fa4094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.677340 10913 raft_consensus.cc:359] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:04.706290 10913 raft_consensus.cc:385] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:04.706835 10913 raft_consensus.cc:740] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:04.708053 10913 consensus_queue.cc:260] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:04.746587 10820 heartbeater.cc:499] Master 127.0.107.254:39861 was elected leader, sending a full tablet report...
I20260709 13:06:04.777733 10915 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
W20260709 13:06:04.783360 10531 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 75363.7us
W20260709 13:06:04.779346 10528 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 99698.2us
I20260709 13:06:04.785256 10915 ts_tablet_manager.cc:1403] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.219s	user 0.021s	sys 0.031s
I20260709 13:06:04.787426 10914 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:04.788264 10894 heartbeater.cc:499] Master 127.0.107.254:39861 was elected leader, sending a full tablet report...
I20260709 13:06:04.789479 10635 tablet_service.cc:1511] Processing CreateTablet for tablet 8d56437e06784282b4cb42f93fa4094a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:06:04.791077 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d56437e06784282b4cb42f93fa4094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.795467 10920 raft_consensus.cc:493] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:04.795854 10915 raft_consensus.cc:359] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:04.796147 10920 raft_consensus.cc:515] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:04.796759 10915 raft_consensus.cc:385] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:04.797147 10915 raft_consensus.cc:740] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:04.799666 10859 tablet_service.cc:1511] Processing CreateTablet for tablet 8d56437e06784282b4cb42f93fa4094a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 13:06:04.810559 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d56437e06784282b4cb42f93fa4094a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.788259 10914 ts_tablet_manager.cc:1403] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.252s	user 0.015s	sys 0.013s
I20260709 13:06:04.810253 10915 consensus_queue.cc:260] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:04.820540 10914 raft_consensus.cc:359] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.823405 10914 raft_consensus.cc:385] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:04.823948 10914 raft_consensus.cc:740] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:04.822799 10913 ts_tablet_manager.cc:1434] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.162s	user 0.012s	sys 0.012s
I20260709 13:06:04.826472 10913 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:04.808748 10916 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:04.833370 10916 ts_tablet_manager.cc:1403] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.238s	user 0.018s	sys 0.024s
W20260709 13:06:04.848246 10895 tablet.cc:2406] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:06:04.827219 10914 consensus_queue.cc:260] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.849833 10916 raft_consensus.cc:359] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.851459 10916 raft_consensus.cc:385] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:04.851775 10916 raft_consensus.cc:740] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:04.853077 10916 consensus_queue.cc:260] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.858318 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 76a046317986428fbc51a3609a2bf91e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:06:04.860471 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a046317986428fbc51a3609a2bf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.862375 10914 ts_tablet_manager.cc:1434] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.005s
I20260709 13:06:04.862847 10916 ts_tablet_manager.cc:1434] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.006s
I20260709 13:06:04.872824 10926 raft_consensus.cc:493] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:04.879208 10920 leader_election.cc:290] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:04.895557 10913 tablet_bootstrap.cc:654] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.896384 10785 tablet_service.cc:1511] Processing CreateTablet for tablet 76a046317986428fbc51a3609a2bf91e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:06:04.892030 10916 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:04.899133 10915 ts_tablet_manager.cc:1434] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.111s	user 0.028s	sys 0.024s
I20260709 13:06:04.900290 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a046317986428fbc51a3609a2bf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.910827 10915 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:04.912880 10914 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:04.919155 10916 tablet_bootstrap.cc:654] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.920022 10915 tablet_bootstrap.cc:654] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.886942 10926 raft_consensus.cc:515] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.927174 10859 tablet_service.cc:1511] Processing CreateTablet for tablet 76a046317986428fbc51a3609a2bf91e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 13:06:04.929394 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a046317986428fbc51a3609a2bf91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.933765 10926 leader_election.cc:290] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:04.952234 10913 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:04.953125 10913 ts_tablet_manager.cc:1403] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.127s	user 0.009s	sys 0.013s
I20260709 13:06:04.952725 10635 tablet_service.cc:1511] Processing CreateTablet for tablet fe860f4e3a2945b28a2c390567bc043d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:06:04.954502 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860f4e3a2945b28a2c390567bc043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.955554 10710 tablet_service.cc:1511] Processing CreateTablet for tablet fe860f4e3a2945b28a2c390567bc043d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:06:04.957012 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860f4e3a2945b28a2c390567bc043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.957650 10913 raft_consensus.cc:359] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.961673 10913 raft_consensus.cc:385] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:04.962029 10913 raft_consensus.cc:740] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:04.963083 10913 consensus_queue.cc:260] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:04.966305 10913 ts_tablet_manager.cc:1434] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:04.975814 10914 tablet_bootstrap.cc:654] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:04.985997 10781 tablet_service.cc:1511] Processing CreateTablet for tablet fe860f4e3a2945b28a2c390567bc043d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 13:06:04.987936 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe860f4e3a2945b28a2c390567bc043d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.991876 10854 tablet_service.cc:1511] Processing CreateTablet for tablet c3f299da07404b66be3f547c4629f7b6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:06:04.993662 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f299da07404b66be3f547c4629f7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:04.998746 10913 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:05.024650 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:05.024870 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b781a7ee3874351bfa70a5ab1452a6f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:05.025676 10869 raft_consensus.cc:2468] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
W20260709 13:06:05.027290 10835 leader_election.cc:343] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.033203 10635 tablet_service.cc:1511] Processing CreateTablet for tablet c3f299da07404b66be3f547c4629f7b6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:06:05.027422 10684 leader_election.cc:304] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:05.034083 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b781a7ee3874351bfa70a5ab1452a6f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:05.035012 10926 raft_consensus.cc:2804] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:05.035115 10646 raft_consensus.cc:2468] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:05.035436 10926 raft_consensus.cc:493] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:05.035854 10926 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.036597 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f299da07404b66be3f547c4629f7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.037653 10709 tablet_service.cc:1511] Processing CreateTablet for tablet c3f299da07404b66be3f547c4629f7b6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 13:06:05.040060 10834 leader_election.cc:304] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:05.044039 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3f299da07404b66be3f547c4629f7b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.047140 10926 raft_consensus.cc:515] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.050035 10920 raft_consensus.cc:2804] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:05.050596 10920 raft_consensus.cc:493] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:05.051108 10920 raft_consensus.cc:3060] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.043345 10915 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:05.055140 10915 ts_tablet_manager.cc:1403] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.145s	user 0.018s	sys 0.000s
I20260709 13:06:05.042783 10916 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:05.058295 10916 ts_tablet_manager.cc:1403] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.171s	user 0.011s	sys 0.035s
I20260709 13:06:05.042588 10913 tablet_bootstrap.cc:654] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.067004 10785 tablet_service.cc:1511] Processing CreateTablet for tablet c2e822ba7018446f98e776bbd1c9e8a3 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:06:05.074620 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e822ba7018446f98e776bbd1c9e8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.071529 10916 raft_consensus.cc:359] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:05.086357 10916 raft_consensus.cc:385] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.086781 10916 raft_consensus.cc:740] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.087750 10916 consensus_queue.cc:260] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:05.090883 10916 ts_tablet_manager.cc:1434] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.000s
I20260709 13:06:05.092393 10916 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:05.074733 10926 leader_election.cc:290] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:05.093315 10914 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:05.075012 10859 tablet_service.cc:1511] Processing CreateTablet for tablet c2e822ba7018446f98e776bbd1c9e8a3 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:06:05.094022 10914 ts_tablet_manager.cc:1403] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.203s	user 0.024s	sys 0.016s
I20260709 13:06:05.094614 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e822ba7018446f98e776bbd1c9e8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.076206 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:05.079691 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 9a50b607435548d3a8c9a26b49489a9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:06:05.098678 10869 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.099788 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a50b607435548d3a8c9a26b49489a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.100569 10916 tablet_bootstrap.cc:654] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.078367 10633 tablet_service.cc:1511] Processing CreateTablet for tablet c2e822ba7018446f98e776bbd1c9e8a3 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 13:06:05.107067 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2e822ba7018446f98e776bbd1c9e8a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.097345 10781 tablet_service.cc:1511] Processing CreateTablet for tablet 9a50b607435548d3a8c9a26b49489a9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:06:05.114703 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a50b607435548d3a8c9a26b49489a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.115056 10629 tablet_service.cc:1511] Processing CreateTablet for tablet a233da3c2e7d4fe795e5191bd32c8daf (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:06:05.116945 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a233da3c2e7d4fe795e5191bd32c8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.085024 10915 raft_consensus.cc:359] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.119303 10711 tablet_service.cc:1511] Processing CreateTablet for tablet a233da3c2e7d4fe795e5191bd32c8daf (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:06:05.120901 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a233da3c2e7d4fe795e5191bd32c8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.119648 10915 raft_consensus.cc:385] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.121690 10915 raft_consensus.cc:740] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.122598 10915 consensus_queue.cc:260] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.127225 10915 ts_tablet_manager.cc:1434] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.004s
I20260709 13:06:05.126915 10777 tablet_service.cc:1511] Processing CreateTablet for tablet a233da3c2e7d4fe795e5191bd32c8daf (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 13:06:05.128553 10915 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:05.127156 10854 tablet_service.cc:1511] Processing CreateTablet for tablet 9a50b607435548d3a8c9a26b49489a9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 13:06:05.129104 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a233da3c2e7d4fe795e5191bd32c8daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.114215 10914 raft_consensus.cc:359] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.137997 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a50b607435548d3a8c9a26b49489a9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.141579 10853 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf8b7376abf4fc68a82af6386b6a6b7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:06:05.073416 10920 raft_consensus.cc:515] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.145787 10914 raft_consensus.cc:385] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.146201 10914 raft_consensus.cc:740] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.147354 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf8b7376abf4fc68a82af6386b6a6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.147120 10914 consensus_queue.cc:260] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.149991 10914 ts_tablet_manager.cc:1434] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.056s	user 0.009s	sys 0.000s
I20260709 13:06:05.150658 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b781a7ee3874351bfa70a5ab1452a6f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:05.151578 10646 raft_consensus.cc:3060] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.153712 10920 leader_election.cc:290] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:05.165107 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b781a7ee3874351bfa70a5ab1452a6f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:05.165987 10721 raft_consensus.cc:3060] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.165782 10630 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf8b7376abf4fc68a82af6386b6a6b7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:06:05.167768 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf8b7376abf4fc68a82af6386b6a6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.151060 10914 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:05.166538 10915 tablet_bootstrap.cc:654] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.181814 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:05.182144 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
W20260709 13:06:05.184160 10686 leader_election.cc:343] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:05.184934 10686 leader_election.cc:343] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.190682 10869 raft_consensus.cc:2468] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:05.192044 10927 raft_consensus.cc:493] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.192634 10927 raft_consensus.cc:515] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.212152 10684 leader_election.cc:304] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:05.213326 10646 raft_consensus.cc:2468] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:05.213675 10926 raft_consensus.cc:2804] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:05.214320 10926 raft_consensus.cc:697] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:05.215250 10927 leader_election.cc:290] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:05.215342 10926 consensus_queue.cc:237] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.231178 10834 leader_election.cc:304] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:05.232224 10920 raft_consensus.cc:2804] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:05.233250 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 4cf8b7376abf4fc68a82af6386b6a6b7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 13:06:05.234813 10721 raft_consensus.cc:2468] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:05.235190 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cf8b7376abf4fc68a82af6386b6a6b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:05.237429 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:05.238284 10779 tablet_service.cc:1511] Processing CreateTablet for tablet 38f642f247be413085d971feb65e94be (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:06:05.240869 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 38f642f247be413085d971feb65e94be (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:06:05.245452 10635 tablet_service.cc:1511] Processing CreateTablet for tablet 38f642f247be413085d971feb65e94be (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 13:06:05.255213 10922 raft_consensus.cc:493] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.255815 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f642f247be413085d971feb65e94be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.258127 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f642f247be413085d971feb65e94be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.259912 10705 tablet_service.cc:1511] Processing CreateTablet for tablet a95dffb6d91746b6bc3fd74d12cbdccc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:06:05.262140 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f642f247be413085d971feb65e94be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.267447 10913 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:05.267285 10784 tablet_service.cc:1511] Processing CreateTablet for tablet a95dffb6d91746b6bc3fd74d12cbdccc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:06:05.268062 10913 ts_tablet_manager.cc:1403] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.273s	user 0.027s	sys 0.006s
I20260709 13:06:05.268836 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95dffb6d91746b6bc3fd74d12cbdccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.255851 10922 raft_consensus.cc:515] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.272189 10913 raft_consensus.cc:359] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.272758 10922 leader_election.cc:290] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:05.273018 10913 raft_consensus.cc:385] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.277210 10913 raft_consensus.cc:740] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.283612 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95dffb6d91746b6bc3fd74d12cbdccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.291977 10857 tablet_service.cc:1511] Processing CreateTablet for tablet a95dffb6d91746b6bc3fd74d12cbdccc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 13:06:05.296463 10543 catalog_manager.cc:5697] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:05.307664 10913 consensus_queue.cc:260] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.310227 10913 ts_tablet_manager.cc:1434] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.042s	user 0.011s	sys 0.000s
I20260709 13:06:05.311342 10913 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:05.302754 10914 tablet_bootstrap.cc:654] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.314134 10916 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:05.314998 10916 ts_tablet_manager.cc:1403] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.223s	user 0.056s	sys 0.072s
I20260709 13:06:05.284539 10920 raft_consensus.cc:697] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:05.306274 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95dffb6d91746b6bc3fd74d12cbdccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.330771 10927 raft_consensus.cc:493] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.331369 10927 raft_consensus.cc:515] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:05.342432 10785 tablet_service.cc:1511] Processing CreateTablet for tablet de828056db2a4621ba3a4e4ec0cc5d53 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:06:05.342124 10855 tablet_service.cc:1511] Processing CreateTablet for tablet 017730403fb44573ae22679b6d9d461f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:06:05.344051 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de828056db2a4621ba3a4e4ec0cc5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.344300 10855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017730403fb44573ae22679b6d9d461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.347347 10915 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:05.347954 10915 ts_tablet_manager.cc:1403] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.220s	user 0.019s	sys 0.004s
I20260709 13:06:05.353945 10927 leader_election.cc:290] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:05.354398 10711 tablet_service.cc:1511] Processing CreateTablet for tablet de828056db2a4621ba3a4e4ec0cc5d53 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:06:05.356331 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de828056db2a4621ba3a4e4ec0cc5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.363648 10915 raft_consensus.cc:359] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.364579 10915 raft_consensus.cc:385] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.364976 10915 raft_consensus.cc:740] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.365877 10915 consensus_queue.cc:260] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.368386 10915 ts_tablet_manager.cc:1434] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 13:06:05.389118 10914 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:05.389846 10920 consensus_queue.cc:237] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.413622 10633 tablet_service.cc:1511] Processing CreateTablet for tablet de828056db2a4621ba3a4e4ec0cc5d53 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 13:06:05.413997 10630 tablet_service.cc:1511] Processing CreateTablet for tablet 017730403fb44573ae22679b6d9d461f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:06:05.415459 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de828056db2a4621ba3a4e4ec0cc5d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.415678 10944 raft_consensus.cc:493] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.416296 10944 raft_consensus.cc:515] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.419111 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017730403fb44573ae22679b6d9d461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.419248 10944 leader_election.cc:290] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:05.426967 10704 tablet_service.cc:1511] Processing CreateTablet for tablet 017730403fb44573ae22679b6d9d461f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 13:06:05.427625 10781 tablet_service.cc:1511] Processing CreateTablet for tablet bddfc21009464cbd8334356be1d409fb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:06:05.428581 10704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017730403fb44573ae22679b6d9d461f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.408349 10914 ts_tablet_manager.cc:1403] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.256s	user 0.016s	sys 0.006s
I20260709 13:06:05.437783 10915 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:05.431145 10916 raft_consensus.cc:359] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.453959 10853 tablet_service.cc:1511] Processing CreateTablet for tablet bddfc21009464cbd8334356be1d409fb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:06:05.455467 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddfc21009464cbd8334356be1d409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.458119 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddfc21009464cbd8334356be1d409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.462589 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:05.463667 10646 raft_consensus.cc:2468] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:05.469650 10834 leader_election.cc:304] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:05.470628 10944 raft_consensus.cc:2804] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:05.471019 10944 raft_consensus.cc:493] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:05.471366 10944 raft_consensus.cc:3060] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.449805 10916 raft_consensus.cc:385] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.477732 10916 raft_consensus.cc:740] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.478686 10916 consensus_queue.cc:260] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.480271 10629 tablet_service.cc:1511] Processing CreateTablet for tablet bddfc21009464cbd8334356be1d409fb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 13:06:05.481830 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bddfc21009464cbd8334356be1d409fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.487004 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 0681122286764dc3b7d97a1a9962e46e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:06:05.488593 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0681122286764dc3b7d97a1a9962e46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.488837 10779 tablet_service.cc:1511] Processing CreateTablet for tablet 0681122286764dc3b7d97a1a9962e46e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 13:06:05.490252 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0681122286764dc3b7d97a1a9962e46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.490705 10944 raft_consensus.cc:515] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.499151 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:05.499665 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:05.499928 10646 raft_consensus.cc:2468] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:05.501358 10914 raft_consensus.cc:359] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.501482 10854 tablet_service.cc:1511] Processing CreateTablet for tablet 0681122286764dc3b7d97a1a9962e46e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
W20260709 13:06:05.501904 10760 leader_election.cc:343] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.510426 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0681122286764dc3b7d97a1a9962e46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.510677 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:05.515002 10944 leader_election.cc:290] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:05.507937 10916 ts_tablet_manager.cc:1434] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.192s	user 0.007s	sys 0.003s
I20260709 13:06:05.512429 10914 raft_consensus.cc:385] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.530695 10760 leader_election.cc:304] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:05.531452 10705 tablet_service.cc:1511] Processing CreateTablet for tablet 25f5b1ed27b34d46b55c0e684d98d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:06:05.533437 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:05.539456 10612 leader_election.cc:343] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.537210 10784 tablet_service.cc:1511] Processing CreateTablet for tablet 25f5b1ed27b34d46b55c0e684d98d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:06:05.525151 10635 tablet_service.cc:1511] Processing CreateTablet for tablet 25f5b1ed27b34d46b55c0e684d98d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 13:06:05.532354 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:05.550518 10916 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:05.551047 10632 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73ff90c30e40eaa01d6254d379aa3a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:06:05.540082 10915 tablet_bootstrap.cc:654] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.538918 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:05.554203 10914 raft_consensus.cc:740] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.565301 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f5b1ed27b34d46b55c0e684d98d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.571529 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f5b1ed27b34d46b55c0e684d98d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.573486 10777 tablet_service.cc:1511] Processing CreateTablet for tablet cd8c76caaf4443508347a95b9085bf9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:06:05.566493 10927 raft_consensus.cc:2804] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:05.579417 10927 raft_consensus.cc:493] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:05.579820 10927 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.568023 10646 raft_consensus.cc:3060] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.574184 10858 tablet_service.cc:1511] Processing CreateTablet for tablet cd8c76caaf4443508347a95b9085bf9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:06:05.587571 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8c76caaf4443508347a95b9085bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:05.589690 10761 leader_election.cc:343] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.590329 10761 leader_election.cc:304] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:05.590704 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73ff90c30e40eaa01d6254d379aa3a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:06:05.592257 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73ff90c30e40eaa01d6254d379aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.591960 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:05.593480 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8c76caaf4443508347a95b9085bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.591203 10544 catalog_manager.cc:5697] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:05.575228 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25f5b1ed27b34d46b55c0e684d98d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.610630 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73ff90c30e40eaa01d6254d379aa3a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 13:06:05.613492 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73ff90c30e40eaa01d6254d379aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.619602 10913 tablet_bootstrap.cc:654] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.639807 10914 consensus_queue.cc:260] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.641692 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73ff90c30e40eaa01d6254d379aa3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.642866 10914 ts_tablet_manager.cc:1434] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.194s	user 0.009s	sys 0.000s
I20260709 13:06:05.645680 10914 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:05.654968 10646 raft_consensus.cc:2468] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:05.663409 10834 leader_election.cc:304] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:05.664556 10920 raft_consensus.cc:2804] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:05.665274 10920 raft_consensus.cc:697] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:05.666493 10920 consensus_queue.cc:237] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.673429 10916 tablet_bootstrap.cc:654] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.679256 10633 tablet_service.cc:1511] Processing CreateTablet for tablet cd8c76caaf4443508347a95b9085bf9a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 13:06:05.663861 10711 tablet_service.cc:1511] Processing CreateTablet for tablet d52e964d865840ee885c5cb005982d82 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:06:05.691258 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52e964d865840ee885c5cb005982d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.703735 10913 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:05.704737 10913 ts_tablet_manager.cc:1403] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.394s	user 0.018s	sys 0.017s
I20260709 13:06:05.706115 10955 raft_consensus.cc:2749] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:05.706187 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd8c76caaf4443508347a95b9085bf9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.715534 10915 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:05.716249 10915 ts_tablet_manager.cc:1403] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.281s	user 0.010s	sys 0.016s
I20260709 13:06:05.725093 10779 tablet_service.cc:1511] Processing CreateTablet for tablet d52e964d865840ee885c5cb005982d82 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260709 13:06:05.726405 10686 consensus_peers.cc:597] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:05.726823 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52e964d865840ee885c5cb005982d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.720141 10927 raft_consensus.cc:515] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.728449 10795 raft_consensus.cc:2393] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:05.731215 10853 tablet_service.cc:1511] Processing CreateTablet for tablet d52e964d865840ee885c5cb005982d82 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 13:06:05.732702 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52e964d865840ee885c5cb005982d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.726179 10915 raft_consensus.cc:359] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.732826 10609 leader_election.cc:304] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:05.734372 10913 raft_consensus.cc:359] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.741663 10913 raft_consensus.cc:385] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.741783 10922 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:05.742074 10913 raft_consensus.cc:740] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.743647 10927 leader_election.cc:290] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:05.743992 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:05.745440 10913 consensus_queue.cc:260] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.748685 10913 ts_tablet_manager.cc:1434] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.044s	user 0.010s	sys 0.002s
I20260709 13:06:05.750032 10913 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:05.751695 10922 raft_consensus.cc:2749] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:05.752308 10915 raft_consensus.cc:385] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:06:05.752923 10761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 192555us
I20260709 13:06:05.761560 10915 raft_consensus.cc:740] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.762626 10915 consensus_queue.cc:260] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.764654 10710 tablet_service.cc:1511] Processing CreateTablet for tablet fcce4acfbd7240178278200df749235a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:06:05.759418 10916 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:05.765165 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:05.760722 10629 tablet_service.cc:1511] Processing CreateTablet for tablet fcce4acfbd7240178278200df749235a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:06:05.770375 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcce4acfbd7240178278200df749235a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.771178 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcce4acfbd7240178278200df749235a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:05.775437 10761 leader_election.cc:343] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.777602 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:05.778223 10646 raft_consensus.cc:2468] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:05.778081 10926 consensus_queue.cc:1048] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:05.779562 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:05.780781 10760 leader_election.cc:304] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:05.781893 10927 raft_consensus.cc:2804] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:05.782966 10927 raft_consensus.cc:697] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:05.784139 10927 consensus_queue.cc:237] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:05.785653 10916 ts_tablet_manager.cc:1403] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.225s	user 0.013s	sys 0.019s
W20260709 13:06:05.786871 10761 leader_election.cc:343] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.786720 10784 tablet_service.cc:1511] Processing CreateTablet for tablet fcce4acfbd7240178278200df749235a (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 13:06:05.788269 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcce4acfbd7240178278200df749235a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:05.791411 10832 leader_election.cc:343] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:05.804013 10832 leader_election.cc:343] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.806526 10914 tablet_bootstrap.cc:654] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.808513 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 58ce1e65607f4e02b082593d502be0e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:06:05.810591 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ce1e65607f4e02b082593d502be0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.811776 10922 raft_consensus.cc:493] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.811884 10704 tablet_service.cc:1511] Processing CreateTablet for tablet 58ce1e65607f4e02b082593d502be0e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:06:05.812376 10922 raft_consensus.cc:515] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.805305 10630 tablet_service.cc:1511] Processing CreateTablet for tablet 58ce1e65607f4e02b082593d502be0e9 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 13:06:05.815627 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ce1e65607f4e02b082593d502be0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.816151 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:05.818427 10922 leader_election.cc:290] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:05.825001 10704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58ce1e65607f4e02b082593d502be0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.828655 10913 tablet_bootstrap.cc:654] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:05.828754 10781 tablet_service.cc:1511] Processing CreateTablet for tablet 1a0288d9025e4e1fa0e2269184186f0c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:06:05.835059 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0288d9025e4e1fa0e2269184186f0c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:05.824651 10609 leader_election.cc:343] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.847168 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 1a0288d9025e4e1fa0e2269184186f0c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:06:05.847733 10705 tablet_service.cc:1511] Processing CreateTablet for tablet ca1d8934020d46ad9b3a2a19c509cf56 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:06:05.849519 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0288d9025e4e1fa0e2269184186f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.850242 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1d8934020d46ad9b3a2a19c509cf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.855822 10636 tablet_service.cc:1511] Processing CreateTablet for tablet 1a0288d9025e4e1fa0e2269184186f0c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 13:06:05.857859 10636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0288d9025e4e1fa0e2269184186f0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.861760 10777 tablet_service.cc:1511] Processing CreateTablet for tablet ca1d8934020d46ad9b3a2a19c509cf56 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:06:05.863351 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1d8934020d46ad9b3a2a19c509cf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.865043 10926 raft_consensus.cc:493] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.866252 10926 raft_consensus.cc:515] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.867821 10854 tablet_service.cc:1511] Processing CreateTablet for tablet ca1d8934020d46ad9b3a2a19c509cf56 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 13:06:05.869369 10854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca1d8934020d46ad9b3a2a19c509cf56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.874271 10926 leader_election.cc:290] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:05.881342 10955 raft_consensus.cc:493] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:05.882148 10955 raft_consensus.cc:515] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:05.888146 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 96da319bd6aa4afb90b067caaedacf3e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:06:05.889878 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:05.891597 10761 leader_election.cc:343] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.888167 10955 leader_election.cc:290] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:05.894253 10631 tablet_service.cc:1511] Processing CreateTablet for tablet 96da319bd6aa4afb90b067caaedacf3e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:06:05.895718 10631 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96da319bd6aa4afb90b067caaedacf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.896159 10785 tablet_service.cc:1511] Processing CreateTablet for tablet 96da319bd6aa4afb90b067caaedacf3e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 13:06:05.889910 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96da319bd6aa4afb90b067caaedacf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.897547 10785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96da319bd6aa4afb90b067caaedacf3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.911875 10855 tablet_service.cc:1511] Processing CreateTablet for tablet d744005e4a084fd29059cb6140b6d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:06:05.917662 10855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d744005e4a084fd29059cb6140b6d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.919132 10916 raft_consensus.cc:359] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.942968 10915 ts_tablet_manager.cc:1434] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.226s	user 0.030s	sys 0.065s
I20260709 13:06:05.937748 10916 raft_consensus.cc:385] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:05.944417 10916 raft_consensus.cc:740] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:05.944911 10915 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:05.945564 10916 consensus_queue.cc:260] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:05.950683 10711 tablet_service.cc:1511] Processing CreateTablet for tablet d744005e4a084fd29059cb6140b6d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:06:05.952271 10711 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d744005e4a084fd29059cb6140b6d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.951223 10916 ts_tablet_manager.cc:1434] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.165s	user 0.008s	sys 0.000s
I20260709 13:06:05.956557 10916 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:05.961128 10859 tablet_service.cc:1511] Processing CreateTablet for tablet 5ce3e6e278894017aa528fe91907d31c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:06:05.962728 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce3e6e278894017aa528fe91907d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.964756 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
W20260709 13:06:05.966939 10609 leader_election.cc:343] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:05.967475 10609 leader_election.cc:304] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:05.962877 10779 tablet_service.cc:1511] Processing CreateTablet for tablet 5ce3e6e278894017aa528fe91907d31c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:06:05.968393 10922 raft_consensus.cc:2749] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:05.967768 10541 catalog_manager.cc:5697] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:05.970420 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce3e6e278894017aa528fe91907d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:05.982882 10632 tablet_service.cc:1511] Processing CreateTablet for tablet d744005e4a084fd29059cb6140b6d935 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 13:06:05.984617 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d744005e4a084fd29059cb6140b6d935. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.027232 10635 tablet_service.cc:1511] Processing CreateTablet for tablet 5ce3e6e278894017aa528fe91907d31c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 13:06:06.028319 10708 tablet_service.cc:1511] Processing CreateTablet for tablet a37f22f2791e488bb016469737ecef9d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:06:06.029160 10635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ce3e6e278894017aa528fe91907d31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.037796 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37f22f2791e488bb016469737ecef9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.042872 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:06.043701 10869 raft_consensus.cc:2468] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:06.043972 10920 consensus_queue.cc:1048] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:06.046607 10920 consensus_queue.cc:1048] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:06.062007 10914 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:06.062716 10914 ts_tablet_manager.cc:1403] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.419s	user 0.017s	sys 0.005s
I20260709 13:06:06.066920 10780 tablet_service.cc:1511] Processing CreateTablet for tablet a37f22f2791e488bb016469737ecef9d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:06:06.068588 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37f22f2791e488bb016469737ecef9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.070097 10913 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:06.070798 10684 leader_election.cc:304] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
W20260709 13:06:06.087579 10832 consensus_peers.cc:597] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:06.087747 10853 tablet_service.cc:1511] Processing CreateTablet for tablet a37f22f2791e488bb016469737ecef9d (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 13:06:06.089299 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37f22f2791e488bb016469737ecef9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.101151 10915 tablet_bootstrap.cc:654] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.115530 10913 ts_tablet_manager.cc:1403] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.366s	user 0.018s	sys 0.014s
I20260709 13:06:06.143831 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 588969ac5b174336819392cffb0cd9e7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:06:06.146996 10630 tablet_service.cc:1511] Processing CreateTablet for tablet 588969ac5b174336819392cffb0cd9e7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:06:06.148535 10630 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588969ac5b174336819392cffb0cd9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.148401 10541 catalog_manager.cc:5697] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:06.151703 10633 tablet_service.cc:1511] Processing CreateTablet for tablet 97d4e12c4db14c3397986584840b233b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:06:06.153105 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d4e12c4db14c3397986584840b233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.167785 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588969ac5b174336819392cffb0cd9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.179450 10920 consensus_queue.cc:1048] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:06.111974 10914 raft_consensus.cc:359] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.185070 10914 raft_consensus.cc:385] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.185590 10914 raft_consensus.cc:740] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.186957 10914 consensus_queue.cc:260] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:06.191452 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 120746us
I20260709 13:06:06.237010 10916 tablet_bootstrap.cc:654] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.161011 10913 raft_consensus.cc:359] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.240419 10913 raft_consensus.cc:385] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.240811 10913 raft_consensus.cc:740] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.241783 10913 consensus_queue.cc:260] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.244297 10913 ts_tablet_manager.cc:1434] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.109s	user 0.007s	sys 0.000s
W20260709 13:06:06.257128 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:06.245433 10913 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:06.292398 10782 tablet_service.cc:1511] Processing CreateTablet for tablet 79aeb4ec11254f438869b1aae326f5c6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:06:06.272385 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:06.298395 10705 tablet_service.cc:1511] Processing CreateTablet for tablet 97d4e12c4db14c3397986584840b233b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:06:06.303016 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d4e12c4db14c3397986584840b233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.306721 10855 tablet_service.cc:1511] Processing CreateTablet for tablet 97d4e12c4db14c3397986584840b233b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 13:06:06.308374 10855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d4e12c4db14c3397986584840b233b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.310462 10915 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:06.310657 10914 ts_tablet_manager.cc:1434] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.248s	user 0.009s	sys 0.019s
I20260709 13:06:06.311218 10915 ts_tablet_manager.cc:1403] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.367s	user 0.010s	sys 0.022s
I20260709 13:06:06.315776 10915 raft_consensus.cc:359] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.323431 10915 raft_consensus.cc:385] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.323844 10915 raft_consensus.cc:740] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.324198 10914 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:06.299224 10970 raft_consensus.cc:2804] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:06.334033 10970 raft_consensus.cc:493] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:06.334440 10970 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:06.336467 10633 tablet_service.cc:1511] Processing CreateTablet for tablet 79aeb4ec11254f438869b1aae326f5c6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:06:06.338114 10784 tablet_service.cc:1511] Processing CreateTablet for tablet 588969ac5b174336819392cffb0cd9e7 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 13:06:06.338629 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aeb4ec11254f438869b1aae326f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.340279 10920 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:06.341130 10920 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.341915 10859 tablet_service.cc:1511] Processing CreateTablet for tablet 79aeb4ec11254f438869b1aae326f5c6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 13:06:06.343374 10859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aeb4ec11254f438869b1aae326f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:06.345623 10758 leader_election.cc:343] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.346369 10758 leader_election.cc:304] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:06.368672 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79aeb4ec11254f438869b1aae326f5c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.376430 10916 tablet_bootstrap.cc:492] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:06.377138 10916 ts_tablet_manager.cc:1403] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.421s	user 0.023s	sys 0.023s
I20260709 13:06:06.378500 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 588969ac5b174336819392cffb0cd9e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.282938 10927 consensus_queue.cc:1048] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.081s
I20260709 13:06:06.391624 10913 tablet_bootstrap.cc:654] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.393250 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:06.397181 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa9d4273100419bb29366421f9a294c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:06:06.397814 10914 tablet_bootstrap.cc:654] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.371273 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:06.371368 10920 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:06.428304 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa9d4273100419bb29366421f9a294c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.428141 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:06.433207 10982 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:06.433876 10982 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:06.436254 10927 raft_consensus.cc:2749] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:06.442118 10835 leader_election.cc:343] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:06.454027 10761 consensus_peers.cc:597] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:06.453929 10708 tablet_service.cc:1511] Processing CreateTablet for tablet 3ebe7292f3cf4695813d0c4e643fb632 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:06:06.455915 10629 tablet_service.cc:1511] Processing CreateTablet for tablet 3ebe7292f3cf4695813d0c4e643fb632 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:06:06.381217 10916 raft_consensus.cc:359] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:06.458197 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 112870us
I20260709 13:06:06.457444 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ebe7292f3cf4695813d0c4e643fb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.382863 10915 consensus_queue.cc:260] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.473929 10964 raft_consensus.cc:493] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:06.470427 10636 tablet_service.cc:1511] Processing CreateTablet for tablet 3cbb2820ebd1414e98070a8a401c92a0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:06:06.474545 10964 raft_consensus.cc:515] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.469110 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa9d4273100419bb29366421f9a294c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 13:06:06.479067 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 3cbb2820ebd1414e98070a8a401c92a0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:06:06.479491 10970 raft_consensus.cc:515] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:06.475492 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:06.482219 10760 leader_election.cc:343] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.470072 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 3cbb2820ebd1414e98070a8a401c92a0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 13:06:06.490051 10636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cbb2820ebd1414e98070a8a401c92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.491988 10915 ts_tablet_manager.cc:1434] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.180s	user 0.013s	sys 0.001s
I20260709 13:06:06.493294 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:06.493604 10915 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:06.459530 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 72258.5us
I20260709 13:06:06.467195 10916 raft_consensus.cc:385] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.473074 10982 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:06.498060 10970 leader_election.cc:290] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:06.515017 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:06.517277 10977 raft_consensus.cc:493] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:06.518103 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cbb2820ebd1414e98070a8a401c92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.529697 10868 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:06.529812 10977 raft_consensus.cc:515] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.519701 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:06.532354 10977 leader_election.cc:290] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:06.532944 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ebe7292f3cf4695813d0c4e643fb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.534796 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:06.536149 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:06.522938 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cbb2820ebd1414e98070a8a401c92a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.536389 10853 tablet_service.cc:1511] Processing CreateTablet for tablet d61cf6b014a34a9f8758702b48e6f5e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:06:06.536849 10794 raft_consensus.cc:2468] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:06.524701 10964 leader_election.cc:290] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:06.510782 10916 raft_consensus.cc:740] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.528458 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:06.534832 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:06.539306 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61cf6b014a34a9f8758702b48e6f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:06.545913 10758 leader_election.cc:343] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.548360 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa9d4273100419bb29366421f9a294c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.548008 10632 tablet_service.cc:1511] Processing CreateTablet for tablet d61cf6b014a34a9f8758702b48e6f5e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
W20260709 13:06:06.549563 10686 leader_election.cc:343] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.541250 10780 tablet_service.cc:1511] Processing CreateTablet for tablet 6aa9d4273100419bb29366421f9a294c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
W20260709 13:06:06.561630 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 44760.7us
I20260709 13:06:06.561722 10612 leader_election.cc:304] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:06.566452 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61cf6b014a34a9f8758702b48e6f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.567662 10684 leader_election.cc:304] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:06.581614 10970 raft_consensus.cc:2804] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:06.582135 10970 raft_consensus.cc:493] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:06.582430 10779 tablet_service.cc:1511] Processing CreateTablet for tablet d61cf6b014a34a9f8758702b48e6f5e0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 13:06:06.583876 10868 raft_consensus.cc:2468] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:06.582489 10970 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:06.585307 10964 raft_consensus.cc:2749] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:06.591199 10684 leader_election.cc:304] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:06.592307 10972 raft_consensus.cc:2804] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:06.593030 10972 raft_consensus.cc:697] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:06.594110 10972 consensus_queue.cc:237] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:06.592643 10916 consensus_queue.cc:260] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:06.596577 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6aa9d4273100419bb29366421f9a294c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.596943 10795 raft_consensus.cc:2468] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:06.602180 10916 ts_tablet_manager.cc:1434] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.225s	user 0.013s	sys 0.000s
I20260709 13:06:06.564087 10758 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:06.606144 10982 raft_consensus.cc:2749] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:06.602973 10834 leader_election.cc:343] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.613909 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61cf6b014a34a9f8758702b48e6f5e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:06.607424 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 61574us
I20260709 13:06:06.608170 10834 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:06.631518 10913 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:06.642246 10913 ts_tablet_manager.cc:1403] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.397s	user 0.016s	sys 0.005s
I20260709 13:06:06.646205 10784 tablet_service.cc:1511] Processing CreateTablet for tablet 3ebe7292f3cf4695813d0c4e643fb632 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 13:06:06.640793 10970 raft_consensus.cc:515] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.655155 10985 raft_consensus.cc:2749] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:06.647179 10915 tablet_bootstrap.cc:654] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.662611 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 5f424821b5444c69b981f137e64f96fc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:06:06.663674 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f424821b5444c69b981f137e64f96fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.658577 10916 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:06.662035 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ebe7292f3cf4695813d0c4e643fb632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.671335 10782 tablet_service.cc:1511] Processing CreateTablet for tablet 5f424821b5444c69b981f137e64f96fc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:06:06.673758 10970 leader_election.cc:290] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:06.674628 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
W20260709 13:06:06.682054 10686 leader_election.cc:343] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:06.675282 10914 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:06.689087 10914 ts_tablet_manager.cc:1403] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.365s	user 0.020s	sys 0.002s
I20260709 13:06:06.690474 10777 tablet_service.cc:1511] Processing CreateTablet for tablet cd614e94b3a24216a9ae872cda1709c0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:06:06.675411 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 5f424821b5444c69b981f137e64f96fc (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 13:06:06.692408 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f424821b5444c69b981f137e64f96fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.675778 10633 tablet_service.cc:1511] Processing CreateTablet for tablet cd614e94b3a24216a9ae872cda1709c0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:06:06.677655 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:06.677631 10913 raft_consensus.cc:359] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:06.708469 10913 raft_consensus.cc:385] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.708841 10913 raft_consensus.cc:740] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.709826 10913 consensus_queue.cc:260] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:06.697171 10710 tablet_service.cc:1511] Processing CreateTablet for tablet cd614e94b3a24216a9ae872cda1709c0 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 13:06:06.706115 10795 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:06.706151 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f424821b5444c69b981f137e64f96fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.718333 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd614e94b3a24216a9ae872cda1709c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.707538 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd614e94b3a24216a9ae872cda1709c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.721751 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd614e94b3a24216a9ae872cda1709c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.729650 10914 raft_consensus.cc:359] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.734969 10914 raft_consensus.cc:385] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.735510 10914 raft_consensus.cc:740] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.736495 10914 consensus_queue.cc:260] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:06.746269 10855 tablet_service.cc:1511] Processing CreateTablet for tablet a32195885f27470e85f07731d06d85d6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:06:06.748421 10855 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32195885f27470e85f07731d06d85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.766279 10915 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:06.766983 10915 ts_tablet_manager.cc:1403] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.274s	user 0.018s	sys 0.001s
I20260709 13:06:06.766747 10629 tablet_service.cc:1511] Processing CreateTablet for tablet a32195885f27470e85f07731d06d85d6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:06:06.768416 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32195885f27470e85f07731d06d85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.776854 10915 raft_consensus.cc:359] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.778059 10915 raft_consensus.cc:385] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:06.778400 10915 raft_consensus.cc:740] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:06.783174 10915 consensus_queue.cc:260] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.791481 10795 raft_consensus.cc:2468] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:06.794225 10913 ts_tablet_manager.cc:1434] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.152s	user 0.012s	sys 0.000s
I20260709 13:06:06.795608 10913 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
W20260709 13:06:06.760221 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 157273us
I20260709 13:06:06.800010 10914 ts_tablet_manager.cc:1434] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.110s	user 0.010s	sys 0.000s
I20260709 13:06:06.804487 10709 tablet_service.cc:1511] Processing CreateTablet for tablet a32195885f27470e85f07731d06d85d6 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 13:06:06.813694 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a32195885f27470e85f07731d06d85d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.825208 10684 leader_election.cc:304] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:06.827847 10970 raft_consensus.cc:2804] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:06.828418 10970 raft_consensus.cc:697] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:06.830775 10914 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:06.838131 10779 tablet_service.cc:1511] Processing CreateTablet for tablet 95a19fc069014b2db397a4c45961acda (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:06:06.829272 10970 consensus_queue.cc:237] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:06.846400 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a19fc069014b2db397a4c45961acda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.832882 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 95a19fc069014b2db397a4c45961acda (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:06:06.852685 10915 ts_tablet_manager.cc:1434] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.085s	user 0.012s	sys 0.001s
I20260709 13:06:06.854432 10915 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:06.852855 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a19fc069014b2db397a4c45961acda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.871467 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 37b295fd9eb746c78bae881d54b57351 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:06:06.871317 10633 tablet_service.cc:1511] Processing CreateTablet for tablet 95a19fc069014b2db397a4c45961acda (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 13:06:06.872857 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b295fd9eb746c78bae881d54b57351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.873266 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95a19fc069014b2db397a4c45961acda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.896711 10645 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:06.899457 10780 tablet_service.cc:1511] Processing CreateTablet for tablet 37b295fd9eb746c78bae881d54b57351 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:06:06.901160 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b295fd9eb746c78bae881d54b57351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.906356 10543 catalog_manager.cc:5697] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:06.908285 10914 tablet_bootstrap.cc:654] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.910092 10913 tablet_bootstrap.cc:654] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.910869 10915 tablet_bootstrap.cc:654] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:06.930918 10858 tablet_service.cc:1511] Processing CreateTablet for tablet 37b295fd9eb746c78bae881d54b57351 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 13:06:06.927347 10632 tablet_service.cc:1511] Processing CreateTablet for tablet 7cf735aa97ef4935a1f8ac140c341ceb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:06:06.932407 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf735aa97ef4935a1f8ac140c341ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.933041 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37b295fd9eb746c78bae881d54b57351. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:06.988278 10916 tablet_bootstrap.cc:654] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.017814 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 80f861846a004804a0c1cca12f60514e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:06:07.018580 10784 tablet_service.cc:1511] Processing CreateTablet for tablet 7cf735aa97ef4935a1f8ac140c341ceb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:06:07.019774 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f861846a004804a0c1cca12f60514e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.020234 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf735aa97ef4935a1f8ac140c341ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.021710 10992 raft_consensus.cc:493] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.022519 10992 raft_consensus.cc:515] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.023033 10633 tablet_service.cc:1511] Processing CreateTablet for tablet 80f861846a004804a0c1cca12f60514e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:06:07.024780 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f861846a004804a0c1cca12f60514e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.007607 10708 tablet_service.cc:1511] Processing CreateTablet for tablet 7cf735aa97ef4935a1f8ac140c341ceb (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 13:06:07.026621 10708 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf735aa97ef4935a1f8ac140c341ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.027065 10992 leader_election.cc:290] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:07.027743 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:07.027881 10853 tablet_service.cc:1511] Processing CreateTablet for tablet 80f861846a004804a0c1cca12f60514e (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 13:06:07.029908 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:07.030215 10835 leader_election.cc:343] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.030660 10795 raft_consensus.cc:2468] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:07.032243 10832 leader_election.cc:304] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:07.033418 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f861846a004804a0c1cca12f60514e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.033938 10992 raft_consensus.cc:2804] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:07.034464 10992 raft_consensus.cc:493] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:07.034854 10992 raft_consensus.cc:3060] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.039736 10913 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:07.040376 10913 ts_tablet_manager.cc:1403] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.245s	user 0.018s	sys 0.006s
I20260709 13:06:07.058238 10914 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:07.059000 10914 ts_tablet_manager.cc:1403] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.229s	user 0.013s	sys 0.029s
I20260709 13:06:07.059167 10915 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:07.059751 10915 ts_tablet_manager.cc:1403] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.206s	user 0.018s	sys 0.003s
I20260709 13:06:07.061728 10913 raft_consensus.cc:359] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.062546 10913 raft_consensus.cc:385] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.062606 10914 raft_consensus.cc:359] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.062915 10915 raft_consensus.cc:359] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.063448 10914 raft_consensus.cc:385] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.063659 10915 raft_consensus.cc:385] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.063819 10914 raft_consensus.cc:740] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.064018 10915 raft_consensus.cc:740] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.058580 10992 raft_consensus.cc:515] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.064812 10915 consensus_queue.cc:260] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.067521 10915 ts_tablet_manager.cc:1434] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.006s
I20260709 13:06:07.067926 10992 leader_election.cc:290] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:07.064759 10914 consensus_queue.cc:260] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.068719 10915 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:07.062927 10913 raft_consensus.cc:740] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.071206 10913 consensus_queue.cc:260] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.072300 10914 ts_tablet_manager.cc:1434] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.013s	user 0.003s	sys 0.004s
I20260709 13:06:07.076112 10913 ts_tablet_manager.cc:1434] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.003s
I20260709 13:06:07.076134 10914 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:07.134442 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:07.136924 10795 raft_consensus.cc:3060] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.138543 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
W20260709 13:06:07.148343 10835 leader_election.cc:343] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.157363 10964 raft_consensus.cc:493] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.158725 10964 raft_consensus.cc:515] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.159986 10915 tablet_bootstrap.cc:654] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.171558 10972 raft_consensus.cc:493] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.172559 10972 raft_consensus.cc:515] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.175830 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:07.179824 10964 leader_election.cc:290] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:07.182171 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.185585 10970 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:06:07.185788 10609 leader_election.cc:343] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.192734 10779 tablet_service.cc:1511] Processing CreateTablet for tablet ec1ed15631604b35b414fe86a4c3083c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20260709 13:06:07.201122 10612 leader_election.cc:343] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.202060 10612 leader_election.cc:304] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:07.205804 10964 raft_consensus.cc:2749] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.208639 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1ed15631604b35b414fe86a4c3083c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.214805 10794 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.186622 10970 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.214808 10977 consensus_queue.cc:1048] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:06:07.216251 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.220391 10972 leader_election.cc:290] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.179059 10913 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:07.233856 10916 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:07.232841 10858 tablet_service.cc:1511] Processing CreateTablet for tablet ec1ed15631604b35b414fe86a4c3083c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:06:07.222007 10964 raft_consensus.cc:493] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.245796 10636 tablet_service.cc:1511] Processing CreateTablet for tablet ec1ed15631604b35b414fe86a4c3083c (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 13:06:07.251989 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:07.252389 10858 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1ed15631604b35b414fe86a4c3083c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.252045 10636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec1ed15631604b35b414fe86a4c3083c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.287667 10970 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.289438 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:07.291752 10686 leader_election.cc:343] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.294384 10795 raft_consensus.cc:2468] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:07.296062 10832 leader_election.cc:304] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:07.285862 10916 ts_tablet_manager.cc:1403] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.650s	user 0.016s	sys 0.013s
I20260709 13:06:07.297881 10992 raft_consensus.cc:2804] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:07.298084 10993 raft_consensus.cc:493] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.299124 10993 raft_consensus.cc:515] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.300287 10915 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:07.301052 10915 ts_tablet_manager.cc:1403] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.233s	user 0.018s	sys 0.018s
I20260709 13:06:07.303673 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:07.304646 10915 raft_consensus.cc:359] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.259478 10964 raft_consensus.cc:515] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.305560 10915 raft_consensus.cc:385] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.305948 10915 raft_consensus.cc:740] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.306329 10993 leader_election.cc:290] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.307591 10964 leader_election.cc:290] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:07.308499 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:07.304207 10913 tablet_bootstrap.cc:654] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.308852 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.310694 10793 raft_consensus.cc:2393] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:07.312075 10609 leader_election.cc:304] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:07.312608 10709 tablet_service.cc:1511] Processing CreateTablet for tablet 77087779392747febcc9e60b2f23f96f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:06:07.307235 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.309738 10915 consensus_queue.cc:260] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.304160 10916 raft_consensus.cc:359] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.315359 10916 raft_consensus.cc:385] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 13:06:07.315491 10832 leader_election.cc:343] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:07.315706 10632 tablet_service.cc:1511] Processing CreateTablet for tablet 1260e07a0f924664b09fbb38ee72b0a1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:06:07.314999 10709 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77087779392747febcc9e60b2f23f96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.317270 10632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1260e07a0f924664b09fbb38ee72b0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.304721 10645 raft_consensus.cc:2468] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:07.317835 10915 ts_tablet_manager.cc:1434] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.002s
I20260709 13:06:07.319370 10834 leader_election.cc:304] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:07.320739 10993 raft_consensus.cc:2804] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:07.321115 10993 raft_consensus.cc:493] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:07.321444 10993 raft_consensus.cc:3060] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.325155 10914 tablet_bootstrap.cc:654] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.314961 10964 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.315734 10916 raft_consensus.cc:740] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.328598 10916 consensus_queue.cc:260] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.319352 10915 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:07.341675 10528 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 64756us
I20260709 13:06:07.304466 10684 leader_election.cc:304] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:07.354177 10982 raft_consensus.cc:493] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.354861 10982 raft_consensus.cc:515] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:07.355314 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 95604.8us
I20260709 13:06:07.357918 10970 raft_consensus.cc:2749] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.359741 10982 leader_election.cc:290] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.361965 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.363138 10867 raft_consensus.cc:2468] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:07.366151 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.374331 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:07.379627 10853 tablet_service.cc:1511] Processing CreateTablet for tablet 77087779392747febcc9e60b2f23f96f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:06:07.381258 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77087779392747febcc9e60b2f23f96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.386261 10970 consensus_queue.cc:1048] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:07.387763 10760 leader_election.cc:304] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:07.394958 10544 catalog_manager.cc:5697] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:07.386488 10780 tablet_service.cc:1511] Processing CreateTablet for tablet 77087779392747febcc9e60b2f23f96f (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 13:06:07.350660 10992 raft_consensus.cc:697] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:07.408277 10710 tablet_service.cc:1511] Processing CreateTablet for tablet 1260e07a0f924664b09fbb38ee72b0a1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:06:07.409214 10992 consensus_queue.cc:237] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.427364 10915 tablet_bootstrap.cc:654] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.429554 10684 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:07.430475 10970 raft_consensus.cc:2804] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:07.430867 10970 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:07.431233 10970 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.411438 10782 tablet_service.cc:1511] Processing CreateTablet for tablet 1260e07a0f924664b09fbb38ee72b0a1 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 13:06:07.438380 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1260e07a0f924664b09fbb38ee72b0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.440522 10710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1260e07a0f924664b09fbb38ee72b0a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.407907 10914 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:07.413645 10982 raft_consensus.cc:2749] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.424063 10964 raft_consensus.cc:2749] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.416343 10993 raft_consensus.cc:515] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.460345 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77087779392747febcc9e60b2f23f96f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.419266 10705 tablet_service.cc:1511] Processing CreateTablet for tablet 3077ac317c854af19123e5a37389d01b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:06:07.464201 10914 ts_tablet_manager.cc:1403] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.391s	user 0.009s	sys 0.014s
I20260709 13:06:07.468250 10993 leader_election.cc:290] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.471140 10705 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3077ac317c854af19123e5a37389d01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.474884 10916 ts_tablet_manager.cc:1434] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.176s	user 0.006s	sys 0.003s
I20260709 13:06:07.476496 10916 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:07.415414 10857 tablet_service.cc:1511] Processing CreateTablet for tablet 3077ac317c854af19123e5a37389d01b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:06:07.472208 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:07.477319 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:07.478027 10645 raft_consensus.cc:3060] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.423393 10777 tablet_service.cc:1511] Processing CreateTablet for tablet c66cc57cc4ac4587a267846214205210 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260709 13:06:07.473376 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:07.483057 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cc57cc4ac4587a267846214205210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.487987 10913 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:07.489055 10913 ts_tablet_manager.cc:1403] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.412s	user 0.027s	sys 0.036s
I20260709 13:06:07.418522 10629 tablet_service.cc:1511] Processing CreateTablet for tablet 3077ac317c854af19123e5a37389d01b (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 13:06:07.490577 10629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3077ac317c854af19123e5a37389d01b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.498804 10857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3077ac317c854af19123e5a37389d01b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:07.497977 10832 leader_election.cc:343] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:07.463200 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 75679us
I20260709 13:06:07.536301 10633 tablet_service.cc:1511] Processing CreateTablet for tablet c66cc57cc4ac4587a267846214205210 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:06:07.537886 10633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cc57cc4ac4587a267846214205210. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:06:07.519966 10528 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 53616.4us
I20260709 13:06:07.542809 10913 raft_consensus.cc:359] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.553181 10913 raft_consensus.cc:385] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.553550 10913 raft_consensus.cc:740] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.544094 10914 raft_consensus.cc:359] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.542876 10972 consensus_queue.cc:1048] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 13:06:07.566799 10914 raft_consensus.cc:385] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.567356 10914 raft_consensus.cc:740] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.569523 10977 consensus_queue.cc:1048] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:07.570670 10913 consensus_queue.cc:260] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.572755 10915 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:07.576922 10915 ts_tablet_manager.cc:1403] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.258s	user 0.012s	sys 0.019s
I20260709 13:06:07.572770 10853 tablet_service.cc:1511] Processing CreateTablet for tablet c66cc57cc4ac4587a267846214205210 (DEFAULT_TABLE table=test-workload [id=11f9ab3bd30047a58ae8c51468b20f28]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 13:06:07.583583 10853 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c66cc57cc4ac4587a267846214205210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:07.580255 10645 raft_consensus.cc:2468] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
W20260709 13:06:07.588706 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 76201.7us
I20260709 13:06:07.583045 10720 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.609117 10541 catalog_manager.cc:5697] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:07.585040 10914 consensus_queue.cc:260] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.623773 10913 ts_tablet_manager.cc:1434] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.134s	user 0.012s	sys 0.000s
I20260709 13:06:07.629263 10915 raft_consensus.cc:359] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:07.637950 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 135803us
I20260709 13:06:07.640915 10915 raft_consensus.cc:385] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.642880 10915 raft_consensus.cc:740] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.645078 10834 leader_election.cc:304] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:07.651145 10993 raft_consensus.cc:2804] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:07.651682 10993 raft_consensus.cc:697] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:07.652580 10993 consensus_queue.cc:237] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.655207 10794 raft_consensus.cc:3060] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.656572 10914 ts_tablet_manager.cc:1434] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.161s	user 0.008s	sys 0.001s
I20260709 13:06:07.661247 10914 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:07.643987 10915 consensus_queue.cc:260] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.635800 10970 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.678102 10916 tablet_bootstrap.cc:654] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.681945 10914 tablet_bootstrap.cc:654] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.685714 10915 ts_tablet_manager.cc:1434] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.108s	user 0.010s	sys 0.000s
I20260709 13:06:07.687392 10915 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:07.719852 10913 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:07.722975 10964 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.723620 10964 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.731843 10964 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:07.733987 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:07.734383 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.735121 10721 raft_consensus.cc:2393] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:07.735142 10868 raft_consensus.cc:2468] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:07.740684 10914 tablet_bootstrap.cc:492] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:07.741284 10914 ts_tablet_manager.cc:1403] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.084s	user 0.016s	sys 0.005s
I20260709 13:06:07.742146 10913 tablet_bootstrap.cc:654] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.741968 10609 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:07.744477 10914 raft_consensus.cc:359] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.745281 10914 raft_consensus.cc:385] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.745568 10914 raft_consensus.cc:740] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.746287 10916 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:07.746876 10916 ts_tablet_manager.cc:1403] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.271s	user 0.021s	sys 0.011s
I20260709 13:06:07.748513 10914 consensus_queue.cc:260] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.751520 10914 ts_tablet_manager.cc:1434] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.010s	user 0.003s	sys 0.003s
I20260709 13:06:07.748178 10964 raft_consensus.cc:2804] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:07.752125 10964 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:07.752527 10964 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.752875 10914 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:07.760715 10914 tablet_bootstrap.cc:654] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.744293 10544 catalog_manager.cc:5697] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:07.769886 10970 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.772207 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:07.773010 10868 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.781175 10915 tablet_bootstrap.cc:654] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.809226 10916 raft_consensus.cc:359] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.810776 10916 raft_consensus.cc:385] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.811321 10916 raft_consensus.cc:740] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.812419 10916 consensus_queue.cc:260] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.785668 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:07.830255 11009 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.835147 11009 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.847596 10913 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:07.850409 10913 ts_tablet_manager.cc:1403] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.138s	user 0.013s	sys 0.003s
I20260709 13:06:07.853161 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.857854 10867 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:07.847877 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.859424 10794 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:07.861119 10609 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:07.845391 11009 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:07.874054 10964 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.875727 10645 raft_consensus.cc:2393] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate bf9fc3264f844470bc0ea04312ff2dd6 in this term.
I20260709 13:06:07.879081 10916 ts_tablet_manager.cc:1434] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.132s	user 0.009s	sys 0.008s
I20260709 13:06:07.880616 10916 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:07.882332 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:07.883322 10720 raft_consensus.cc:2393] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:07.884899 10964 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:07.887404 10868 raft_consensus.cc:2468] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:07.889837 10684 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:07.890946 10970 raft_consensus.cc:2804] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:07.891983 10970 raft_consensus.cc:697] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:07.893126 10970 consensus_queue.cc:237] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.899736 10916 tablet_bootstrap.cc:654] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.901911 11009 raft_consensus.cc:2804] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:07.902534 11009 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:07.902947 11009 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.917454 10913 raft_consensus.cc:359] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.919164 10913 raft_consensus.cc:385] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.919452 10913 raft_consensus.cc:740] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.921054 10913 consensus_queue.cc:260] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:07.910578 10609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 49565.4us
I20260709 13:06:07.929134 10914 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:07.929939 10972 raft_consensus.cc:493] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.930446 10914 ts_tablet_manager.cc:1403] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.178s	user 0.021s	sys 0.046s
I20260709 13:06:07.930624 10972 raft_consensus.cc:515] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.934909 10914 raft_consensus.cc:359] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.935973 10914 raft_consensus.cc:385] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.936314 10914 raft_consensus.cc:740] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.938686 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:07.939690 10795 raft_consensus.cc:2393] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:07.933324 10972 leader_election.cc:290] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:07.940897 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69a32458d0e54f08a475b354e480c787" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.948633 10684 leader_election.cc:304] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:07.937296 10914 consensus_queue.cc:260] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.943162 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:07.950304 10867 raft_consensus.cc:2393] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:07.949646 10915 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:07.951449 10914 ts_tablet_manager.cc:1434] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.000s
I20260709 13:06:07.951773 10915 ts_tablet_manager.cc:1403] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.265s	user 0.015s	sys 0.002s
I20260709 13:06:07.951973 10609 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:07.952328 10916 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:07.950174 10972 raft_consensus.cc:3060] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:07.953176 10916 ts_tablet_manager.cc:1403] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.073s	user 0.023s	sys 0.011s
I20260709 13:06:07.957068 11036 raft_consensus.cc:2749] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.958655 10916 raft_consensus.cc:359] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.959491 10916 raft_consensus.cc:385] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.959789 10916 raft_consensus.cc:740] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.961325 10916 consensus_queue.cc:260] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:07.963910 10916 ts_tablet_manager.cc:1434] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 13:06:07.965226 10916 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:07.963172 10915 raft_consensus.cc:359] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.965708 10915 raft_consensus.cc:385] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:07.952672 10914 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:07.966014 10915 raft_consensus.cc:740] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:07.970541 10913 ts_tablet_manager.cc:1434] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.120s	user 0.011s	sys 0.000s
I20260709 13:06:07.973052 10542 catalog_manager.cc:5697] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:07.976836 10914 tablet_bootstrap.cc:654] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.971081 10915 consensus_queue.cc:260] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:07.970661 10972 raft_consensus.cc:2749] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:07.980201 10993 consensus_queue.cc:1048] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:07.985564 10982 raft_consensus.cc:493] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.986361 10982 raft_consensus.cc:515] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:07.988052 10916 tablet_bootstrap.cc:654] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.990739 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:07.992703 10915 ts_tablet_manager.cc:1434] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.000s
I20260709 13:06:07.992707 10972 raft_consensus.cc:493] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:07.993551 10972 raft_consensus.cc:515] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:07.994596 10915 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:07.995042 10982 leader_election.cc:290] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:07.995951 10972 leader_election.cc:290] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:07.996500 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff22da804864a31992dc8a587b224f2" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:07.997573 10646 raft_consensus.cc:2393] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:07.998175 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.000142 10760 leader_election.cc:304] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:08.001497 10913 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.005250 10915 tablet_bootstrap.cc:654] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:07.948998 11009 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:08.000252 10684 leader_election.cc:343] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.002506 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.011534 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:08.019361 10868 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.011780 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:08.022367 10793 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.021596 10993 consensus_queue.cc:1048] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 13:06:08.013569 10982 raft_consensus.cc:3060] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.013033 10867 raft_consensus.cc:2468] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:08.035890 10684 leader_election.cc:304] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:08.037300 10972 raft_consensus.cc:2804] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.037988 10972 raft_consensus.cc:493] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.038364 10972 raft_consensus.cc:3060] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.044266 10915 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.045192 10915 ts_tablet_manager.cc:1403] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.051s	user 0.019s	sys 0.000s
I20260709 13:06:08.047860 10868 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:08.049736 10609 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.050827 11036 raft_consensus.cc:2804] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.051640 10793 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:08.051548 11036 raft_consensus.cc:697] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:08.055071 11036 consensus_queue.cc:237] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.051798 10972 raft_consensus.cc:515] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.009454 11009 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:08.066257 10972 leader_election.cc:290] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:08.067938 10914 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:08.068300 10982 raft_consensus.cc:2749] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.068974 10914 ts_tablet_manager.cc:1403] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.117s	user 0.035s	sys 0.046s
I20260709 13:06:08.069813 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:08.063910 10915 raft_consensus.cc:359] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.070600 10868 raft_consensus.cc:3060] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.070631 10915 raft_consensus.cc:385] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.071671 10915 raft_consensus.cc:740] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.072875 10915 consensus_queue.cc:260] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.075037 10913 tablet_bootstrap.cc:654] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.076149 10915 ts_tablet_manager.cc:1434] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 13:06:08.076987 10914 raft_consensus.cc:359] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.078112 10914 raft_consensus.cc:385] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.078930 10914 raft_consensus.cc:740] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.079778 10915 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.078178 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:08.084084 10916 tablet_bootstrap.cc:492] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.084896 10916 ts_tablet_manager.cc:1403] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.120s	user 0.018s	sys 0.018s
W20260709 13:06:08.085466 10684 leader_election.cc:343] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.084614 10914 consensus_queue.cc:260] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.089164 10914 ts_tablet_manager.cc:1434] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.020s	user 0.009s	sys 0.000s
I20260709 13:06:08.090574 10914 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:08.091259 10916 raft_consensus.cc:359] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.092088 10916 raft_consensus.cc:385] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.092360 10916 raft_consensus.cc:740] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.097095 10916 consensus_queue.cc:260] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.103724 10914 tablet_bootstrap.cc:654] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.118695 10915 tablet_bootstrap.cc:654] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.123361 10970 raft_consensus.cc:493] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.125012 10970 raft_consensus.cc:515] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.129149 10916 ts_tablet_manager.cc:1434] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.044s	user 0.011s	sys 0.000s
I20260709 13:06:08.130884 10916 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.144843 10868 raft_consensus.cc:2468] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:08.137070 10993 consensus_queue.cc:1048] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 13:06:08.159958 10992 consensus_queue.cc:1048] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.160367 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:08.161032 10970 leader_election.cc:290] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.161068 10646 raft_consensus.cc:2468] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:08.162669 10686 leader_election.cc:304] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.158804 10684 leader_election.cc:304] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:08.168222 10972 raft_consensus.cc:2804] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.168651 10972 raft_consensus.cc:493] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.169036 10972 raft_consensus.cc:3060] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.172561 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.173429 10795 raft_consensus.cc:2468] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:08.178970 11023 raft_consensus.cc:2804] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.180650 11023 raft_consensus.cc:697] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:08.181542 10916 tablet_bootstrap.cc:654] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.182262 11023 consensus_queue.cc:237] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.199129 10913 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.199993 10913 ts_tablet_manager.cc:1403] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.199s	user 0.025s	sys 0.020s
I20260709 13:06:08.208047 10914 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:08.209240 10914 ts_tablet_manager.cc:1403] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.119s	user 0.032s	sys 0.009s
I20260709 13:06:08.208410 10543 catalog_manager.cc:5697] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:08.228056 10915 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.228773 10915 ts_tablet_manager.cc:1403] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.151s	user 0.032s	sys 0.006s
I20260709 13:06:08.231184 10914 raft_consensus.cc:359] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.232245 10914 raft_consensus.cc:385] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.232573 10914 raft_consensus.cc:740] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.237260 10914 consensus_queue.cc:260] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.244215 10913 raft_consensus.cc:359] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.252511 10913 raft_consensus.cc:385] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.252982 10913 raft_consensus.cc:740] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.255484 10914 ts_tablet_manager.cc:1434] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.046s	user 0.010s	sys 0.000s
I20260709 13:06:08.255715 10913 consensus_queue.cc:260] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.253726 10972 raft_consensus.cc:515] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.255232 10982 raft_consensus.cc:493] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.266034 10982 raft_consensus.cc:515] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.267071 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:08.269030 10646 raft_consensus.cc:3060] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.267630 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c394fa4e36384c6eb804d29258eb7ec1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:08.270082 10793 raft_consensus.cc:3060] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.270742 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.270939 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "912716eee3434e46aead233d6f5a7c83" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:08.271632 10868 raft_consensus.cc:2393] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:08.258919 10915 raft_consensus.cc:359] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.260033 11036 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.274183 10758 leader_election.cc:304] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:08.276319 10915 raft_consensus.cc:385] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.273379 10982 leader_election.cc:290] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:08.281165 10915 raft_consensus.cc:740] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.283558 10914 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:08.285033 10972 leader_election.cc:290] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.274207 11036 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.292814 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.293658 10868 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:08.295154 10609 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.296134 10964 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.296548 10964 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.296918 10964 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.285871 10982 raft_consensus.cc:3060] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.287999 11009 raft_consensus.cc:493] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.289463 10542 catalog_manager.cc:5697] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:08.302122 11009 raft_consensus.cc:515] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.302438 10646 raft_consensus.cc:2468] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:08.304277 10686 leader_election.cc:304] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.306142 10970 raft_consensus.cc:2804] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.306711 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.307531 10970 raft_consensus.cc:697] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:08.307704 11036 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:08.309027 10609 leader_election.cc:343] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.308836 10970 consensus_queue.cc:237] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.309756 11009 leader_election.cc:290] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.314144 10916 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.313932 11022 raft_consensus.cc:493] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.287333 10913 ts_tablet_manager.cc:1434] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.087s	user 0.005s	sys 0.004s
I20260709 13:06:08.314862 10916 ts_tablet_manager.cc:1403] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.184s	user 0.022s	sys 0.000s
I20260709 13:06:08.314908 11022 raft_consensus.cc:515] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.323585 10793 raft_consensus.cc:2468] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:08.324709 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:08.327628 10684 leader_election.cc:343] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.329450 11022 leader_election.cc:290] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:08.286536 10915 consensus_queue.cc:260] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.318109 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:08.332580 10721 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:08.332446 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.334254 10915 ts_tablet_manager.cc:1434] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.105s	user 0.011s	sys 0.000s
I20260709 13:06:08.318995 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:08.334621 10684 leader_election.cc:343] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.335165 10684 leader_election.cc:304] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:08.335485 10915 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:08.336362 10612 leader_election.cc:343] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.336529 11022 raft_consensus.cc:2749] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.337021 10612 leader_election.cc:304] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:08.338379 11009 raft_consensus.cc:2749] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.321564 10913 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.339840 10964 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.350030 10916 raft_consensus.cc:359] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.350867 10916 raft_consensus.cc:385] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.351197 10916 raft_consensus.cc:740] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.357909 10916 consensus_queue.cc:260] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.359372 11022 consensus_queue.cc:1048] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.365494 10913 tablet_bootstrap.cc:654] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.371963 11022 consensus_queue.cc:1048] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.373436 10916 ts_tablet_manager.cc:1434] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.031s	user 0.011s	sys 0.000s
I20260709 13:06:08.374727 10915 tablet_bootstrap.cc:654] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.376928 10913 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.376397 10543 catalog_manager.cc:5697] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:08.384524 10916 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.399210 10914 tablet_bootstrap.cc:654] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.400341 10916 tablet_bootstrap.cc:654] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.387622 10913 ts_tablet_manager.cc:1403] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.066s	user 0.025s	sys 0.003s
I20260709 13:06:08.403069 10915 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.403638 10915 ts_tablet_manager.cc:1403] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.068s	user 0.014s	sys 0.002s
I20260709 13:06:08.405275 10913 raft_consensus.cc:359] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.406524 10913 raft_consensus.cc:385] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.406661 10915 raft_consensus.cc:359] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.407444 10915 raft_consensus.cc:385] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.407722 10915 raft_consensus.cc:740] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.408205 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:08.408882 10867 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.387645 10982 raft_consensus.cc:2749] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.409830 10915 consensus_queue.cc:260] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.411449 10913 raft_consensus.cc:740] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.412318 10913 consensus_queue.cc:260] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.413192 10915 ts_tablet_manager.cc:1434] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 13:06:08.414808 10913 ts_tablet_manager.cc:1434] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 13:06:08.415968 10913 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.416415 10915 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.418490 10916 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.419337 10916 ts_tablet_manager.cc:1403] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.035s	user 0.009s	sys 0.008s
I20260709 13:06:08.418442 10867 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:08.422086 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:08.423168 10720 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.423676 10915 tablet_bootstrap.cc:654] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.426445 10916 raft_consensus.cc:359] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.427273 10916 raft_consensus.cc:385] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.427590 10916 raft_consensus.cc:740] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.428519 10916 consensus_queue.cc:260] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.430166 10609 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.430840 10916 ts_tablet_manager.cc:1434] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 13:06:08.431154 11009 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.431699 11009 raft_consensus.cc:697] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:08.431990 10916 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.432600 11009 consensus_queue.cc:237] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.427670 10964 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:08.464385 10720 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:08.469973 10913 tablet_bootstrap.cc:654] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.477161 10916 tablet_bootstrap.cc:654] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.480134 10914 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:08.489467 10915 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.490548 10915 ts_tablet_manager.cc:1403] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.074s	user 0.017s	sys 0.002s
I20260709 13:06:08.493902 10915 raft_consensus.cc:359] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.494760 10915 raft_consensus.cc:385] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.495079 10915 raft_consensus.cc:740] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.495836 10914 ts_tablet_manager.cc:1403] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.219s	user 0.024s	sys 0.002s
I20260709 13:06:08.500015 10914 raft_consensus.cc:359] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.500741 10914 raft_consensus.cc:385] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.501009 10914 raft_consensus.cc:740] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.501786 10914 consensus_queue.cc:260] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.503010 10915 consensus_queue.cc:260] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.510231 10914 ts_tablet_manager.cc:1434] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.002s
I20260709 13:06:08.518851 10916 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.519521 10916 ts_tablet_manager.cc:1403] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.088s	user 0.010s	sys 0.005s
I20260709 13:06:08.528258 10915 ts_tablet_manager.cc:1434] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.037s	user 0.009s	sys 0.000s
I20260709 13:06:08.529222 10914 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:08.529587 11009 consensus_queue.cc:1048] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.531926 10541 catalog_manager.cc:5697] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:08.535223 10915 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.522819 10916 raft_consensus.cc:359] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.535697 10916 raft_consensus.cc:385] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.537664 10916 raft_consensus.cc:740] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.538744 10916 consensus_queue.cc:260] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:08.517324 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:08.536847 10913 tablet_bootstrap.cc:492] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.541636 10913 ts_tablet_manager.cc:1403] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.126s	user 0.020s	sys 0.035s
I20260709 13:06:08.543205 10916 ts_tablet_manager.cc:1434] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.001s
I20260709 13:06:08.544028 10915 tablet_bootstrap.cc:654] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.544469 10916 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.545364 10913 raft_consensus.cc:359] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.546260 10913 raft_consensus.cc:385] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.546587 10913 raft_consensus.cc:740] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.547492 10913 consensus_queue.cc:260] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.550403 10913 ts_tablet_manager.cc:1434] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:08.555721 10916 tablet_bootstrap.cc:654] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.559784 10913 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.563187 11036 consensus_queue.cc:1048] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.568719 10913 tablet_bootstrap.cc:654] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.572976 10970 consensus_queue.cc:1048] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.576869 11011 raft_consensus.cc:493] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.577425 11011 raft_consensus.cc:515] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.580812 11011 leader_election.cc:290] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.583128 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:08.583184 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.584432 10720 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:08.584702 10793 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:08.586468 10835 leader_election.cc:304] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:08.589336 11011 raft_consensus.cc:2804] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.589474 10913 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.589856 11011 raft_consensus.cc:493] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.590306 10913 ts_tablet_manager.cc:1403] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.031s	user 0.016s	sys 0.002s
I20260709 13:06:08.590328 11011 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.590723 11022 raft_consensus.cc:493] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.602432 11022 raft_consensus.cc:515] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.606086 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:08.606320 11022 leader_election.cc:290] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.590543 10914 tablet_bootstrap.cc:654] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.607069 10646 raft_consensus.cc:2468] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:08.609006 10686 leader_election.cc:304] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.610754 10970 raft_consensus.cc:2804] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.611294 10970 raft_consensus.cc:493] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.610958 10972 consensus_queue.cc:1048] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.611187 10913 raft_consensus.cc:359] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.612227 10913 raft_consensus.cc:385] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.614161 10913 raft_consensus.cc:740] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.612289 10970 raft_consensus.cc:3060] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.618418 10982 raft_consensus.cc:493] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.619037 10982 raft_consensus.cc:515] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.613040 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.622088 10913 consensus_queue.cc:260] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.623799 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d56437e06784282b4cb42f93fa4094a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:08.624564 10913 ts_tablet_manager.cc:1434] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.033s	user 0.004s	sys 0.002s
I20260709 13:06:08.624508 10646 raft_consensus.cc:2468] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:08.625705 10982 leader_election.cc:290] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:08.627597 11011 raft_consensus.cc:515] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.622128 10795 raft_consensus.cc:2468] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:08.626447 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d56437e06784282b4cb42f93fa4094a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.627669 10760 leader_election.cc:304] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.635167 10869 raft_consensus.cc:2468] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:08.635818 11011 leader_election.cc:290] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.636606 10913 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.636611 10982 raft_consensus.cc:2804] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.637320 10982 raft_consensus.cc:493] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.637729 10982 raft_consensus.cc:3060] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.644420 10915 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.644986 10915 ts_tablet_manager.cc:1403] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.110s	user 0.019s	sys 0.028s
I20260709 13:06:08.647882 10913 tablet_bootstrap.cc:654] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.648409 10970 raft_consensus.cc:515] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.649135 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:08.649933 10916 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.650072 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:08.650545 10916 ts_tablet_manager.cc:1403] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.106s	user 0.013s	sys 0.012s
I20260709 13:06:08.650718 10794 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.651750 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:08.652403 10646 raft_consensus.cc:3060] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.653391 10970 leader_election.cc:290] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.653805 10916 raft_consensus.cc:359] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.654585 10916 raft_consensus.cc:385] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.654850 10916 raft_consensus.cc:740] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.655709 10916 consensus_queue.cc:260] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.658267 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:08.659121 10792 raft_consensus.cc:3060] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.659811 10916 ts_tablet_manager.cc:1434] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:08.661068 10916 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.649778 10720 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.647801 10915 raft_consensus.cc:359] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.662840 10915 raft_consensus.cc:385] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.663125 10915 raft_consensus.cc:740] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.668457 10982 raft_consensus.cc:515] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.671511 10982 leader_election.cc:290] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:08.673810 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d56437e06784282b4cb42f93fa4094a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:08.674603 10645 raft_consensus.cc:3060] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.677604 10915 consensus_queue.cc:260] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.683858 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d56437e06784282b4cb42f93fa4094a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:08.684864 10867 raft_consensus.cc:3060] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.687924 10720 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:08.689209 10915 ts_tablet_manager.cc:1434] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.038s	user 0.005s	sys 0.004s
I20260709 13:06:08.690032 10835 leader_election.cc:304] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:08.690668 10915 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.694015 11011 raft_consensus.cc:2804] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.694675 11011 raft_consensus.cc:697] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:08.695580 11011 consensus_queue.cc:237] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.702699 10794 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:08.710384 10867 raft_consensus.cc:2468] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:08.717901 10646 raft_consensus.cc:2468] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:08.719404 10686 leader_election.cc:304] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.720345 10970 raft_consensus.cc:2804] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.720847 10970 raft_consensus.cc:697] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:08.721809 10970 consensus_queue.cc:237] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.733172 10792 raft_consensus.cc:2468] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:08.734162 10992 raft_consensus.cc:493] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.734706 10992 raft_consensus.cc:515] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.736936 10992 leader_election.cc:290] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:08.738236 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:08.739910 10835 leader_election.cc:343] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.740708 10645 raft_consensus.cc:2468] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:08.738626 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:08.743595 10914 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:08.744416 10916 tablet_bootstrap.cc:654] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.748471 10914 ts_tablet_manager.cc:1403] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.237s	user 0.009s	sys 0.027s
I20260709 13:06:08.754166 10758 leader_election.cc:304] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:08.752761 10646 raft_consensus.cc:2468] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:08.757488 10982 raft_consensus.cc:2804] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.759572 10982 raft_consensus.cc:697] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:08.761943 10982 consensus_queue.cc:237] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.768599 10970 consensus_queue.cc:1048] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:08.770090 10913 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.770713 10913 ts_tablet_manager.cc:1403] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.137s	user 0.014s	sys 0.014s
I20260709 13:06:08.771199 10542 catalog_manager.cc:5697] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:08.774719 10834 leader_election.cc:304] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:08.776108 10992 raft_consensus.cc:2804] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.776499 10992 raft_consensus.cc:493] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.776787 10992 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.788493 10914 raft_consensus.cc:359] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.790102 10914 raft_consensus.cc:385] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.790514 10914 raft_consensus.cc:740] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.791404 10914 consensus_queue.cc:260] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.793993 10913 raft_consensus.cc:359] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.794791 11009 raft_consensus.cc:493] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.795179 10913 raft_consensus.cc:385] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.795681 10913 raft_consensus.cc:740] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.795360 11009 raft_consensus.cc:515] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.795199 10542 catalog_manager.cc:5697] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:08.796634 10913 consensus_queue.cc:260] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.777448 10915 tablet_bootstrap.cc:654] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.798311 11009 leader_election.cc:290] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:08.800348 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:08.802026 10612 leader_election.cc:343] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.802590 10913 ts_tablet_manager.cc:1434] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.032s	user 0.009s	sys 0.000s
I20260709 13:06:08.802630 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:08.803884 10913 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
W20260709 13:06:08.804692 10609 leader_election.cc:343] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.805974 10609 leader_election.cc:304] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:08.806797 11036 raft_consensus.cc:493] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.807287 11036 raft_consensus.cc:515] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.810215 10914 ts_tablet_manager.cc:1434] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.061s	user 0.010s	sys 0.000s
I20260709 13:06:08.811389 11036 leader_election.cc:290] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.812813 11036 raft_consensus.cc:2749] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.811467 10914 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:08.814738 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:08.815541 10720 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:08.817781 11051 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.818527 11051 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.820755 10612 leader_election.cc:304] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:08.822069 11036 raft_consensus.cc:2804] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:08.822352 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:08.822676 11036 raft_consensus.cc:493] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:08.822950 11051 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:08.823217 11036 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.823163 10792 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:08.824695 10913 tablet_bootstrap.cc:654] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.825650 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:08.832504 10914 tablet_bootstrap.cc:654] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:08.833176 10834 leader_election.cc:343] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.822327 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:08.835078 10832 leader_election.cc:343] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.835562 10832 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:08.838429 11051 raft_consensus.cc:2749] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:08.838764 10992 raft_consensus.cc:515] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.840917 11009 consensus_queue.cc:1048] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:06:08.842851 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:08.843992 10646 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.844463 10916 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.845054 10916 ts_tablet_manager.cc:1403] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.184s	user 0.003s	sys 0.010s
I20260709 13:06:08.848546 10916 raft_consensus.cc:359] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.849426 10916 raft_consensus.cc:385] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.849810 10916 raft_consensus.cc:740] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.850760 10916 consensus_queue.cc:260] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.853341 10916 ts_tablet_manager.cc:1434] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.005s
I20260709 13:06:08.854586 10916 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.859467 10972 consensus_queue.cc:1048] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 13:06:08.864171 10913 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.865039 10913 ts_tablet_manager.cc:1403] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.061s	user 0.012s	sys 0.003s
I20260709 13:06:08.868893 10913 raft_consensus.cc:359] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.869796 10913 raft_consensus.cc:385] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.870070 10913 raft_consensus.cc:740] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.870918 10913 consensus_queue.cc:260] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.888383 10915 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.889057 10915 ts_tablet_manager.cc:1403] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.199s	user 0.012s	sys 0.019s
I20260709 13:06:08.889228 10541 catalog_manager.cc:5697] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:08.892290 10915 raft_consensus.cc:359] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.899104 10915 raft_consensus.cc:385] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.899432 10915 raft_consensus.cc:740] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.900208 10915 consensus_queue.cc:260] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:08.894757 10913 ts_tablet_manager.cc:1434] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 13:06:08.907639 10913 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.909271 10646 raft_consensus.cc:2468] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:08.893277 10916 tablet_bootstrap.cc:654] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.907394 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:08.906225 10992 leader_election.cc:290] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:08.911568 10835 leader_election.cc:343] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:08.914606 11086 consensus_queue.cc:1048] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:08.896569 10914 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:08.915730 10914 ts_tablet_manager.cc:1403] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.105s	user 0.008s	sys 0.012s
I20260709 13:06:08.919369 10914 raft_consensus.cc:359] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.920122 10914 raft_consensus.cc:385] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.920413 10914 raft_consensus.cc:740] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.921295 10914 consensus_queue.cc:260] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.923858 10915 ts_tablet_manager.cc:1434] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.000s
I20260709 13:06:08.925904 10916 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:08.926469 10916 ts_tablet_manager.cc:1403] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.001s
I20260709 13:06:08.926956 10915 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.927973 10914 ts_tablet_manager.cc:1434] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:08.929303 10914 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:08.930429 11036 raft_consensus.cc:515] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.936004 10914 tablet_bootstrap.cc:654] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.921747 10913 tablet_bootstrap.cc:654] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.940245 10916 raft_consensus.cc:359] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.946723 10916 raft_consensus.cc:385] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.941351 10915 tablet_bootstrap.cc:654] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:08.947074 10916 raft_consensus.cc:740] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.952049 10916 consensus_queue.cc:260] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.958560 10916 ts_tablet_manager.cc:1434] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.003s
W20260709 13:06:08.966684 10612 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 68438.5us
I20260709 13:06:08.967867 10834 leader_election.cc:304] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:08.969343 10992 raft_consensus.cc:2804] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:08.969895 10992 raft_consensus.cc:697] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:08.959983 10916 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:08.971263 10992 consensus_queue.cc:237] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.976929 10915 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:08.977624 10915 ts_tablet_manager.cc:1403] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.051s	user 0.014s	sys 0.001s
I20260709 13:06:08.977768 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:08.978474 10720 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:08.978614 11036 leader_election.cc:290] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:08.980860 10915 raft_consensus.cc:359] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.990520 10915 raft_consensus.cc:385] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.991060 10915 raft_consensus.cc:740] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.991881 10913 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:08.992462 10913 ts_tablet_manager.cc:1403] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.008s
I20260709 13:06:08.992391 10915 consensus_queue.cc:260] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:08.995141 10915 ts_tablet_manager.cc:1434] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.003s
I20260709 13:06:08.996253 10915 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:08.995878 10913 raft_consensus.cc:359] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:08.996757 10913 raft_consensus.cc:385] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:08.997062 10913 raft_consensus.cc:740] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:08.997572 11011 raft_consensus.cc:493] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:08.998095 11011 raft_consensus.cc:515] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:08.997973 10913 consensus_queue.cc:260] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.000896 11011 leader_election.cc:290] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.001478 10913 ts_tablet_manager.cc:1434] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.001s
I20260709 13:06:09.002805 10913 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:08.981714 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:09.003863 10793 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.004881 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:09.006309 10835 leader_election.cc:343] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.011368 10720 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:09.013803 10913 tablet_bootstrap.cc:654] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.021863 10542 catalog_manager.cc:5697] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.022572 10916 tablet_bootstrap.cc:654] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.030912 10914 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:09.031505 10914 ts_tablet_manager.cc:1403] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.102s	user 0.011s	sys 0.010s
I20260709 13:06:09.034871 10914 raft_consensus.cc:359] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.035619 10914 raft_consensus.cc:385] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.035892 10914 raft_consensus.cc:740] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.039413 10914 consensus_queue.cc:260] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.042148 10914 ts_tablet_manager.cc:1434] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 13:06:09.044039 10914 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:09.038141 10612 leader_election.cc:304] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.048833 11009 raft_consensus.cc:2804] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:09.049371 11009 raft_consensus.cc:697] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:09.050572 11009 consensus_queue.cc:237] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.052623 10915 tablet_bootstrap.cc:654] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.055891 10916 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:09.056602 10916 ts_tablet_manager.cc:1403] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.097s	user 0.023s	sys 0.010s
I20260709 13:06:09.057816 10914 tablet_bootstrap.cc:654] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.062192 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:09.063393 10913 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:09.063241 10646 raft_consensus.cc:2468] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:09.064023 10913 ts_tablet_manager.cc:1403] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.061s	user 0.011s	sys 0.012s
I20260709 13:06:09.065171 10834 leader_election.cc:304] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:09.067090 10992 raft_consensus.cc:2804] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.067770 10992 raft_consensus.cc:493] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.067504 10913 raft_consensus.cc:359] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.068318 10992 raft_consensus.cc:3060] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.068393 10913 raft_consensus.cc:385] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.069761 10913 raft_consensus.cc:740] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.070667 10913 consensus_queue.cc:260] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.075644 10913 ts_tablet_manager.cc:1434] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.001s
I20260709 13:06:09.082685 10992 raft_consensus.cc:515] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.085026 10913 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:09.086902 11087 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.087793 11087 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.091630 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:09.092394 10792 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:09.085145 10793 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:09.086621 10915 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:09.094069 10915 ts_tablet_manager.cc:1403] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.098s	user 0.016s	sys 0.013s
I20260709 13:06:09.094376 10609 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.095685 11036 raft_consensus.cc:2804] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.095769 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:09.096176 11036 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.096542 10646 raft_consensus.cc:3060] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.096781 11036 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.097857 10915 raft_consensus.cc:359] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.098701 10915 raft_consensus.cc:385] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.102280 10915 raft_consensus.cc:740] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.103181 10915 consensus_queue.cc:260] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.105882 10915 ts_tablet_manager.cc:1434] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 13:06:09.106637 10646 raft_consensus.cc:2468] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:09.107259 10915 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:09.107967 10834 leader_election.cc:304] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.101951 10913 tablet_bootstrap.cc:654] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.108990 11011 raft_consensus.cc:2804] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:09.109546 11011 raft_consensus.cc:697] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:09.110531 11011 consensus_queue.cc:237] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.112957 10982 raft_consensus.cc:493] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.113517 10982 raft_consensus.cc:515] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.115427 10992 leader_election.cc:290] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.115716 11087 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:09.060932 10916 raft_consensus.cc:359] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.118628 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:09.120667 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a233da3c2e7d4fe795e5191bd32c8daf" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:09.118150 10916 raft_consensus.cc:385] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.121423 10916 raft_consensus.cc:740] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.122592 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:09.123275 10867 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:09.123224 11036 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.123106 10916 consensus_queue.cc:260] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.125557 11036 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:09.126212 10916 ts_tablet_manager.cc:1434] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.069s	user 0.006s	sys 0.006s
I20260709 13:06:09.126694 10915 tablet_bootstrap.cc:654] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.128980 10540 catalog_manager.cc:5697] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.119424 10982 leader_election.cc:290] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.130877 10913 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:09.130913 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a233da3c2e7d4fe795e5191bd32c8daf" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:09.131515 10835 leader_election.cc:343] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.131619 10913 ts_tablet_manager.cc:1403] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.047s	user 0.007s	sys 0.009s
I20260709 13:06:09.131808 10644 raft_consensus.cc:2468] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:09.133394 10760 leader_election.cc:304] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.134366 10982 raft_consensus.cc:2804] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.134786 10982 raft_consensus.cc:493] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.134617 10972 consensus_queue.cc:1048] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:09.135195 10982 raft_consensus.cc:3060] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.138536 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:09.139189 10867 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.139323 10913 raft_consensus.cc:359] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.140306 10913 raft_consensus.cc:385] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.121407 10721 raft_consensus.cc:2468] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:09.140681 10913 raft_consensus.cc:740] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.185911 10915 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:09.186604 10915 ts_tablet_manager.cc:1403] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.080s	user 0.015s	sys 0.013s
W20260709 13:06:09.187680 10761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 45570.3us
I20260709 13:06:09.189664 10913 consensus_queue.cc:260] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.192621 10914 tablet_bootstrap.cc:492] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:09.193220 10914 ts_tablet_manager.cc:1403] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.149s	user 0.007s	sys 0.005s
I20260709 13:06:09.194968 10916 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:09.192636 10913 ts_tablet_manager.cc:1434] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.060s	user 0.013s	sys 0.008s
I20260709 13:06:09.198415 10913 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:09.202405 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:09.203254 10793 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.206578 10913 tablet_bootstrap.cc:654] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.207356 10914 raft_consensus.cc:359] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.208137 10914 raft_consensus.cc:385] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.208418 10914 raft_consensus.cc:740] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.209944 10915 raft_consensus.cc:359] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.210747 10915 raft_consensus.cc:385] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.210021 10543 catalog_manager.cc:5697] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:09.211027 10915 raft_consensus.cc:740] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.211905 10915 consensus_queue.cc:260] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.213737 10982 raft_consensus.cc:515] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.214304 10915 ts_tablet_manager.cc:1434] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 13:06:09.215631 10915 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:09.218865 10970 consensus_queue.cc:1048] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:09.220798 10992 raft_consensus.cc:493] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.221361 10992 raft_consensus.cc:515] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.223095 10915 tablet_bootstrap.cc:654] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.224226 10992 leader_election.cc:290] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:09.225592 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:09.226420 10644 raft_consensus.cc:2468] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:09.228134 10834 leader_election.cc:304] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.229398 11009 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.230590 11009 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.233752 10916 tablet_bootstrap.cc:654] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.209306 10914 consensus_queue.cc:260] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.239213 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:09.244159 10612 leader_election.cc:343] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.229398 10992 raft_consensus.cc:2804] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.248521 10982 leader_election.cc:290] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.249109 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a233da3c2e7d4fe795e5191bd32c8daf" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:09.249869 10644 raft_consensus.cc:3060] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.237110 11009 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.251703 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
W20260709 13:06:09.253492 10609 leader_election.cc:343] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.254022 10609 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:09.259011 10644 raft_consensus.cc:2468] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:09.259428 11009 raft_consensus.cc:2749] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:09.262396 10760 leader_election.cc:304] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.266040 10982 raft_consensus.cc:2804] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:09.266930 10982 raft_consensus.cc:697] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:09.269721 10992 raft_consensus.cc:493] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.270437 11014 raft_consensus.cc:493] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.271209 11014 raft_consensus.cc:515] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.272784 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a233da3c2e7d4fe795e5191bd32c8daf" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:09.273483 10721 raft_consensus.cc:3060] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.271129 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:09.277665 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a50b607435548d3a8c9a26b49489a9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:09.278671 10869 raft_consensus.cc:2468] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:09.283715 10758 leader_election.cc:304] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:09.285247 10914 ts_tablet_manager.cc:1434] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.092s	user 0.007s	sys 0.000s
I20260709 13:06:09.268213 10982 consensus_queue.cc:237] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.270466 10992 raft_consensus.cc:3060] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.293416 10915 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:09.294032 10915 ts_tablet_manager.cc:1403] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.079s	user 0.013s	sys 0.000s
I20260709 13:06:09.285869 11014 leader_election.cc:290] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:09.297631 10915 raft_consensus.cc:359] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.298039 11014 raft_consensus.cc:2804] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.298434 10915 raft_consensus.cc:385] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.305878 10915 raft_consensus.cc:740] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.309177 10914 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:09.316576 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a50b607435548d3a8c9a26b49489a9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:09.311085 10915 consensus_queue.cc:260] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.298955 10972 raft_consensus.cc:493] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.305958 11014 raft_consensus.cc:493] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.329387 10915 ts_tablet_manager.cc:1434] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.035s	user 0.012s	sys 0.005s
I20260709 13:06:09.385679 10913 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:09.386691 10913 ts_tablet_manager.cc:1403] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.189s	user 0.032s	sys 0.039s
I20260709 13:06:09.387962 10914 tablet_bootstrap.cc:654] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.390300 10867 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:09.323764 10720 raft_consensus.cc:2468] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:09.392109 10609 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.371984 10915 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:09.334757 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 51266.2us
I20260709 13:06:09.330479 10721 raft_consensus.cc:2468] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:09.405985 10916 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:09.406890 10916 ts_tablet_manager.cc:1403] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.212s	user 0.018s	sys 0.019s
I20260709 13:06:09.332031 11014 raft_consensus.cc:3060] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.418205 10913 raft_consensus.cc:359] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.419234 10913 raft_consensus.cc:385] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.419610 10913 raft_consensus.cc:740] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.342307 11011 raft_consensus.cc:493] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.327571 10972 raft_consensus.cc:515] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:09.342010 10832 leader_election.cc:343] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.409216 11087 raft_consensus.cc:2804] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:09.421716 10913 consensus_queue.cc:260] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.336247 10793 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:09.421139 11036 raft_consensus.cc:493] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.425866 11011 raft_consensus.cc:515] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.431002 11036 raft_consensus.cc:515] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:09.432590 11011 leader_election.cc:290] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:09.433109 11036 leader_election.cc:290] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:09.372646 11051 consensus_queue.cc:1048] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260709 13:06:09.436419 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:09.438057 10609 leader_election.cc:343] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.444406 10982 consensus_queue.cc:1048] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.445967 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:09.449823 11087 raft_consensus.cc:697] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:09.450867 11087 consensus_queue.cc:237] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:09.460832 10612 leader_election.cc:343] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.461536 10612 leader_election.cc:304] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:09.462900 11086 raft_consensus.cc:2749] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:09.466233 11011 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.468112 10982 consensus_queue.cc:1048] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.471526 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:09.472783 10916 raft_consensus.cc:359] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.473657 10916 raft_consensus.cc:385] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.474242 10916 raft_consensus.cc:740] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.479225 10992 raft_consensus.cc:515] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.485713 10916 consensus_queue.cc:260] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:09.476395 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 134321us
I20260709 13:06:09.498363 10915 tablet_bootstrap.cc:654] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.471709 11011 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.510656 10913 ts_tablet_manager.cc:1434] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.124s	user 0.016s	sys 0.001s
I20260709 13:06:09.512588 10913 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:09.517108 10992 leader_election.cc:290] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:09.518644 10835 leader_election.cc:343] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.507731 10916 ts_tablet_manager.cc:1434] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.100s	user 0.001s	sys 0.006s
I20260709 13:06:09.549938 10972 leader_election.cc:290] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:09.571884 10914 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:09.573254 10914 ts_tablet_manager.cc:1403] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.264s	user 0.022s	sys 0.009s
I20260709 13:06:09.510691 10721 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.602909 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:09.538426 11086 consensus_queue.cc:1048] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.055s
I20260709 13:06:09.633756 10916 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:09.655886 10832 leader_election.cc:343] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.656836 10832 leader_election.cc:304] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:09.663105 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:09.663810 11011 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:09.635092 10914 raft_consensus.cc:359] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.666508 10914 raft_consensus.cc:385] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.666988 10914 raft_consensus.cc:740] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.672857 10914 consensus_queue.cc:260] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.676551 10914 ts_tablet_manager.cc:1434] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.003s
I20260709 13:06:09.678850 10914 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:09.694828 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:09.695715 10642 raft_consensus.cc:3060] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.698292 10914 tablet_bootstrap.cc:654] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:09.677121 11115 log.cc:927] Time spent T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 13:06:09.704717 10541 catalog_manager.cc:5697] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:09.692987 11011 raft_consensus.cc:2749] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:09.763734 11107 raft_consensus.cc:493] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.764430 11107 raft_consensus.cc:515] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.751286 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:09.767050 11107 leader_election.cc:290] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:09.769253 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:09.716845 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 54300.4us
I20260709 13:06:09.777542 10992 consensus_queue.cc:1048] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.707831 11014 raft_consensus.cc:515] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.796134 10915 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:09.796748 10915 ts_tablet_manager.cc:1403] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.457s	user 0.031s	sys 0.047s
I20260709 13:06:09.798276 10913 tablet_bootstrap.cc:654] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.777588 10643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:09.788427 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 118427us
W20260709 13:06:09.789635 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 106821us
I20260709 13:06:09.809494 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:09.799700 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:09.809270 10835 leader_election.cc:343] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:09.814949 10643 raft_consensus.cc:2468] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:09.824641 10867 raft_consensus.cc:2393] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:09.835232 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38f642f247be413085d971feb65e94be" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:09.836148 10795 raft_consensus.cc:3060] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.837464 11014 leader_election.cc:290] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:09.838584 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a50b607435548d3a8c9a26b49489a9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:09.841391 10718 raft_consensus.cc:3060] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.839146 10642 raft_consensus.cc:2468] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:09.856082 10686 leader_election.cc:304] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:09.858166 10834 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:09.860235 10916 tablet_bootstrap.cc:654] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:09.860752 11118 consensus_queue.cc:1048] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.861157 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a50b607435548d3a8c9a26b49489a9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:09.862083 10867 raft_consensus.cc:3060] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.864431 10760 leader_election.cc:304] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:09.866809 11051 consensus_queue.cc:1048] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:09.862378 11118 raft_consensus.cc:2804] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:09.871327 11118 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:09.871742 11118 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:09.873131 11087 consensus_queue.cc:1048] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.876940 10915 raft_consensus.cc:359] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.877967 10915 raft_consensus.cc:385] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.880942 10915 raft_consensus.cc:740] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.860126 10834 leader_election.cc:304] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:09.904335 10970 raft_consensus.cc:2749] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:09.907972 10795 raft_consensus.cc:2468] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:09.919147 11107 raft_consensus.cc:2749] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:09.910830 10913 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:09.920514 10913 ts_tablet_manager.cc:1403] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.408s	user 0.017s	sys 0.014s
I20260709 13:06:09.911820 10915 consensus_queue.cc:260] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.929668 10915 ts_tablet_manager.cc:1434] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.133s	user 0.012s	sys 0.001s
I20260709 13:06:09.936611 10992 raft_consensus.cc:2804] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:06:09.933267 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 73216.8us
I20260709 13:06:09.946568 10913 raft_consensus.cc:359] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:09.947769 10913 raft_consensus.cc:385] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.953749 10915 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:09.950116 10913 raft_consensus.cc:740] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.955490 10913 consensus_queue.cc:260] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:09.958366 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:09.959897 10914 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:09.960649 10914 ts_tablet_manager.cc:1403] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.282s	user 0.019s	sys 0.002s
I20260709 13:06:09.952194 10992 raft_consensus.cc:697] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:09.969542 10913 ts_tablet_manager.cc:1434] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.046s	user 0.007s	sys 0.001s
I20260709 13:06:09.975914 11086 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:09.976536 11087 consensus_queue.cc:1048] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.978053 10916 tablet_bootstrap.cc:492] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:09.978631 10916 ts_tablet_manager.cc:1403] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.352s	user 0.026s	sys 0.041s
I20260709 13:06:09.978731 10914 raft_consensus.cc:359] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.979560 10914 raft_consensus.cc:385] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:09.976580 11086 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.979856 10914 raft_consensus.cc:740] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:09.951460 11107 consensus_queue.cc:1048] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
I20260709 13:06:09.984972 10914 consensus_queue.cc:260] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:09.986653 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:09.987732 10992 consensus_queue.cc:237] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:09.999755 10913 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:09.984375 11086 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:09.986697 10718 raft_consensus.cc:2468] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:10.001332 11121 consensus_queue.cc:1048] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:09.998721 10916 raft_consensus.cc:359] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.005067 10970 raft_consensus.cc:493] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.012145 10970 raft_consensus.cc:515] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.014092 10761 leader_election.cc:304] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.014670 10970 leader_election.cc:290] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:10.015164 11107 raft_consensus.cc:2804] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:06:10.002519 10609 leader_election.cc:343] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.015708 11107 raft_consensus.cc:697] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:10.016871 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.016731 11107 consensus_queue.cc:237] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.010273 11126 consensus_queue.cc:1048] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.019595 10916 raft_consensus.cc:385] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.020164 10916 raft_consensus.cc:740] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.020654 10915 tablet_bootstrap.cc:654] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.020947 10916 consensus_queue.cc:260] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.022403 10914 ts_tablet_manager.cc:1434] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.061s	user 0.009s	sys 0.000s
I20260709 13:06:10.021428 11118 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.025226 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.054610 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:10.055445 10641 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:10.056917 10835 consensus_peers.cc:597] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:10.058825 10916 ts_tablet_manager.cc:1434] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.080s	user 0.011s	sys 0.000s
I20260709 13:06:10.065788 10916 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:10.065519 11136 consensus_queue.cc:1048] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.067183 10914 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:10.069844 11118 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.062340 10867 raft_consensus.cc:2468] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
W20260709 13:06:10.074440 10609 leader_election.cc:343] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.096113 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4cf8b7376abf4fc68a82af6386b6a6b7" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.098006 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:10.093046 10972 raft_consensus.cc:493] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.099295 10972 raft_consensus.cc:515] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.093446 10609 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:10.103442 10915 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:10.104030 10915 ts_tablet_manager.cc:1403] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.160s	user 0.016s	sys 0.009s
I20260709 13:06:10.109979 11144 raft_consensus.cc:2749] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:10.119614 10972 leader_election.cc:290] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:10.120608 10641 raft_consensus.cc:2468] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:10.121090 10913 tablet_bootstrap.cc:654] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.124054 10834 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:10.131654 10914 tablet_bootstrap.cc:654] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.135066 10992 raft_consensus.cc:2804] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.142498 10992 raft_consensus.cc:697] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:10.145327 10992 consensus_queue.cc:237] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.149953 10543 catalog_manager.cc:5697] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.185393 10684 leader_election.cc:304] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:10.193032 10972 raft_consensus.cc:2749] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:10.194948 10835 leader_election.cc:343] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.208201 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.211491 10916 tablet_bootstrap.cc:654] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.209324 10791 raft_consensus.cc:2468] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:10.225608 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.231750 10915 raft_consensus.cc:359] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.236196 10542 catalog_manager.cc:5697] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.229784 10867 raft_consensus.cc:2468] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
W20260709 13:06:10.251096 11142 log.cc:927] Time spent T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 13:06:10.266587 10913 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:10.267233 10913 ts_tablet_manager.cc:1403] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.268s	user 0.034s	sys 0.008s
I20260709 13:06:10.267642 10914 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:10.268219 10914 ts_tablet_manager.cc:1403] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.217s	user 0.015s	sys 0.002s
I20260709 13:06:10.270457 10913 raft_consensus.cc:359] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.271273 10913 raft_consensus.cc:385] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.271566 10913 raft_consensus.cc:740] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.271275 10914 raft_consensus.cc:359] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.272373 10913 consensus_queue.cc:260] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.272832 10914 raft_consensus.cc:385] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.274590 10916 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:10.275395 10916 ts_tablet_manager.cc:1403] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.210s	user 0.014s	sys 0.000s
I20260709 13:06:10.275748 10914 raft_consensus.cc:740] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.252045 10915 raft_consensus.cc:385] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.255085 10684 leader_election.cc:304] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.274622 10913 ts_tablet_manager.cc:1434] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:10.278466 10970 raft_consensus.cc:2804] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.278890 10970 raft_consensus.cc:493] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.278740 10916 raft_consensus.cc:359] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.280020 10916 raft_consensus.cc:385] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.276715 10914 consensus_queue.cc:260] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.279271 10970 raft_consensus.cc:3060] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.282241 10720 raft_consensus.cc:3060] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.284721 10914 ts_tablet_manager.cc:1434] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.001s
I20260709 13:06:10.288393 10914 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:10.288688 10970 raft_consensus.cc:515] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.290872 10970 leader_election.cc:290] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:10.279731 10913 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:10.280315 10916 raft_consensus.cc:740] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.294118 10916 consensus_queue.cc:260] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.294780 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:10.295223 10914 tablet_bootstrap.cc:654] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.295410 10795 raft_consensus.cc:3060] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.296026 10915 raft_consensus.cc:740] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.296394 10916 ts_tablet_manager.cc:1434] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.000s
I20260709 13:06:10.297626 10916 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:10.298982 11107 raft_consensus.cc:493] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.299515 11107 raft_consensus.cc:515] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.303661 11107 leader_election.cc:290] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.304047 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:10.304800 10916 tablet_bootstrap.cc:654] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.304690 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.305436 10644 raft_consensus.cc:2468] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
W20260709 13:06:10.305676 10761 leader_election.cc:343] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.306789 10760 leader_election.cc:304] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:10.307948 11107 raft_consensus.cc:2804] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.308329 11107 raft_consensus.cc:493] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.308661 11107 raft_consensus.cc:3060] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.309155 10795 raft_consensus.cc:2468] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:10.309486 10914 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:10.310461 10913 tablet_bootstrap.cc:654] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.310070 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95dffb6d91746b6bc3fd74d12cbdccc" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:10.310947 10914 ts_tablet_manager.cc:1403] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.003s
I20260709 13:06:10.311124 10867 raft_consensus.cc:3060] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.312381 10684 leader_election.cc:304] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.313575 10972 raft_consensus.cc:2804] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.298219 10915 consensus_queue.cc:260] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.314234 10972 raft_consensus.cc:697] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:10.314450 10914 raft_consensus.cc:359] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.315482 10914 raft_consensus.cc:385] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.315135 10972 consensus_queue.cc:237] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.315708 10915 ts_tablet_manager.cc:1434] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.211s	user 0.007s	sys 0.002s
I20260709 13:06:10.315871 10914 raft_consensus.cc:740] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.316919 10915 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:10.313612 11051 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.316984 10914 consensus_queue.cc:260] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.317783 11051 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.255280 10542 catalog_manager.cc:5697] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:10.319844 10914 ts_tablet_manager.cc:1434] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:10.322817 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.323366 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.324275 10644 raft_consensus.cc:2468] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:10.329735 10834 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:10.330770 10992 raft_consensus.cc:2804] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.331195 10992 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.331578 10992 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:10.333438 10832 leader_election.cc:343] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.334172 11136 raft_consensus.cc:493] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.334741 11136 raft_consensus.cc:515] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.333451 10914 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:10.339321 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.337275 11136 leader_election.cc:290] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.339473 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:10.321166 11051 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:10.342185 10612 leader_election.cc:343] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.344906 11107 raft_consensus.cc:515] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:10.347898 10609 leader_election.cc:343] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.348671 10609 leader_election.cc:304] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:10.349370 10992 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.350389 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:10.358019 10644 raft_consensus.cc:3060] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.350849 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "de828056db2a4621ba3a4e4ec0cc5d53" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:10.359320 10992 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:10.359763 10717 raft_consensus.cc:3060] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.356359 10867 raft_consensus.cc:2468] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:10.360975 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:10.361670 10646 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.366161 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:10.366737 10644 raft_consensus.cc:2468] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:10.368147 10760 leader_election.cc:304] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
W20260709 13:06:10.368479 10832 leader_election.cc:343] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.352042 11107 leader_election.cc:290] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.352977 11136 raft_consensus.cc:2749] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:10.331578 10915 tablet_bootstrap.cc:654] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.369906 11126 raft_consensus.cc:2804] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.370442 11126 raft_consensus.cc:697] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:10.371112 10717 raft_consensus.cc:2468] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:10.371465 11126 consensus_queue.cc:237] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.378652 10913 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:10.379289 10913 ts_tablet_manager.cc:1403] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.101s	user 0.034s	sys 0.021s
I20260709 13:06:10.382829 10913 raft_consensus.cc:359] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.382581 10541 catalog_manager.cc:5697] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.383541 10913 raft_consensus.cc:385] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.383898 10913 raft_consensus.cc:740] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.384512 10646 raft_consensus.cc:2468] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:10.384699 10913 consensus_queue.cc:260] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.375578 10914 tablet_bootstrap.cc:654] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.386246 11051 raft_consensus.cc:493] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.386775 11051 raft_consensus.cc:515] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.389567 11051 leader_election.cc:290] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:10.389775 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:10.390545 10915 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:10.391057 10915 ts_tablet_manager.cc:1403] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.074s	user 0.004s	sys 0.012s
W20260709 13:06:10.391351 10835 leader_election.cc:343] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.393859 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.394305 10915 raft_consensus.cc:359] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.394721 10834 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:10.395054 10915 raft_consensus.cc:385] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.395365 10915 raft_consensus.cc:740] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.396011 10992 raft_consensus.cc:2804] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.396570 10992 raft_consensus.cc:697] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:10.397639 10992 consensus_queue.cc:237] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.399089 10913 ts_tablet_manager.cc:1434] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 13:06:10.400314 10913 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:10.396310 10915 consensus_queue.cc:260] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:10.404132 10832 leader_election.cc:343] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.404652 10832 leader_election.cc:304] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:10.405292 10914 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:10.405444 11051 raft_consensus.cc:2749] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:10.405834 10914 ts_tablet_manager.cc:1403] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.073s	user 0.014s	sys 0.000s
I20260709 13:06:10.412451 10915 ts_tablet_manager.cc:1434] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.021s	user 0.009s	sys 0.000s
I20260709 13:06:10.423143 10915 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:10.438839 10914 raft_consensus.cc:359] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.441044 10916 tablet_bootstrap.cc:492] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:10.439721 10914 raft_consensus.cc:385] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.441627 10914 raft_consensus.cc:740] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.441809 10916 ts_tablet_manager.cc:1403] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.144s	user 0.006s	sys 0.019s
I20260709 13:06:10.442412 10914 consensus_queue.cc:260] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.446094 10913 tablet_bootstrap.cc:654] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.445775 10916 raft_consensus.cc:359] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.446646 10916 raft_consensus.cc:385] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.447000 10916 raft_consensus.cc:740] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.447775 10916 consensus_queue.cc:260] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.450539 10916 ts_tablet_manager.cc:1434] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:10.451674 10916 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:10.454689 10914 ts_tablet_manager.cc:1434] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.049s	user 0.007s	sys 0.000s
I20260709 13:06:10.455925 10914 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:10.460080 10916 tablet_bootstrap.cc:654] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.462914 10914 tablet_bootstrap.cc:654] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.466607 10540 catalog_manager.cc:5697] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.474357 10913 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:10.476295 10913 ts_tablet_manager.cc:1403] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.076s	user 0.010s	sys 0.009s
I20260709 13:06:10.482707 10913 raft_consensus.cc:359] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.484198 10913 raft_consensus.cc:385] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.484599 10913 raft_consensus.cc:740] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.441864 10915 tablet_bootstrap.cc:654] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.490432 10543 catalog_manager.cc:5697] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.495687 10544 catalog_manager.cc:5697] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:10.497438 11126 consensus_queue.cc:1048] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.488392 10913 consensus_queue.cc:260] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.505196 10992 consensus_queue.cc:1048] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.505244 10913 ts_tablet_manager.cc:1434] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.026s	user 0.005s	sys 0.001s
I20260709 13:06:10.532809 10914 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:10.533553 10913 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:10.533869 10914 ts_tablet_manager.cc:1403] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.078s	user 0.023s	sys 0.017s
I20260709 13:06:10.537151 10914 raft_consensus.cc:359] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.537899 10914 raft_consensus.cc:385] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.538193 10914 raft_consensus.cc:740] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.539738 10992 raft_consensus.cc:493] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.540339 10992 raft_consensus.cc:515] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.542800 10992 leader_election.cc:290] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:10.545070 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.546617 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:10.548166 10834 leader_election.cc:343] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:10.549604 10832 leader_election.cc:343] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.550083 10832 leader_election.cc:304] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:10.550340 10913 tablet_bootstrap.cc:654] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.551235 11111 raft_consensus.cc:2749] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:10.558472 10992 consensus_queue.cc:1048] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:10.560534 10916 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:10.561179 10916 ts_tablet_manager.cc:1403] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.110s	user 0.012s	sys 0.014s
I20260709 13:06:10.565418 10992 raft_consensus.cc:493] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.566352 10992 raft_consensus.cc:515] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.569270 10970 raft_consensus.cc:493] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.569818 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.569945 10970 raft_consensus.cc:515] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.565083 10916 raft_consensus.cc:359] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.571887 10916 raft_consensus.cc:385] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.572525 10992 leader_election.cc:290] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.572801 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:10.572242 10916 raft_consensus.cc:740] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.575186 10916 consensus_queue.cc:260] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.577417 10916 ts_tablet_manager.cc:1434] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:06:10.579634 10916 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:10.580385 10835 leader_election.cc:343] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.570535 10644 raft_consensus.cc:2468] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:10.581995 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.582715 10867 raft_consensus.cc:2468] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:10.581282 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.583688 10791 raft_consensus.cc:2468] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:10.584748 10834 leader_election.cc:304] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:10.585739 10684 leader_election.cc:304] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.586352 11051 raft_consensus.cc:2804] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.586614 10972 raft_consensus.cc:2804] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.586787 11051 raft_consensus.cc:493] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.587064 10972 raft_consensus.cc:493] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.587277 11051 raft_consensus.cc:3060] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.587491 10972 raft_consensus.cc:3060] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.573117 10970 leader_election.cc:290] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:10.594630 11126 consensus_queue.cc:1048] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.600081 10913 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:10.600907 10913 ts_tablet_manager.cc:1403] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.079s	user 0.021s	sys 0.021s
I20260709 13:06:10.606531 10992 consensus_queue.cc:1048] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.612545 11051 raft_consensus.cc:515] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.618333 10913 raft_consensus.cc:359] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.619457 10913 raft_consensus.cc:385] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.619827 10913 raft_consensus.cc:740] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.620922 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:10.620805 10913 consensus_queue.cc:260] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.621682 10646 raft_consensus.cc:3060] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.629334 10913 ts_tablet_manager.cc:1434] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.028s	user 0.011s	sys 0.000s
I20260709 13:06:10.630545 10913 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:10.636183 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
W20260709 13:06:10.638373 10835 leader_election.cc:343] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.660559 10717 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.697924 10914 consensus_queue.cc:260] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.700098 11051 leader_election.cc:290] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:10.719254 11121 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.719798 11121 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.721573 10915 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:10.722154 10915 ts_tablet_manager.cc:1403] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.299s	user 0.044s	sys 0.065s
I20260709 13:06:10.722508 10913 tablet_bootstrap.cc:654] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.722646 11121 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:10.727479 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.731325 10915 raft_consensus.cc:359] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.732057 10915 raft_consensus.cc:385] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.732323 10915 raft_consensus.cc:740] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.733366 10916 tablet_bootstrap.cc:654] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:10.733300 10832 leader_election.cc:343] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.738075 11022 raft_consensus.cc:493] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.746466 11022 raft_consensus.cc:515] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.749023 11022 leader_election.cc:290] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:10.733868 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:10.751302 10835 leader_election.cc:343] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:10.751821 10835 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:10.752049 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:10.752907 10795 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:10.754527 11051 raft_consensus.cc:2749] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:10.755496 10684 leader_election.cc:304] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.760161 10970 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.762369 10970 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.766836 10914 ts_tablet_manager.cc:1434] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.233s	user 0.007s	sys 0.001s
I20260709 13:06:10.765863 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.761360 11022 raft_consensus.cc:2804] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:10.771039 11022 raft_consensus.cc:493] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:10.771603 11022 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.718269 10972 raft_consensus.cc:515] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.761366 10646 raft_consensus.cc:2468] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:10.780424 10914 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:10.792776 10834 leader_election.cc:304] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:10.782204 10915 consensus_queue.cc:260] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.795253 11111 raft_consensus.cc:2804] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.795876 11111 raft_consensus.cc:697] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:10.796162 10915 ts_tablet_manager.cc:1434] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.074s	user 0.007s	sys 0.001s
I20260709 13:06:10.796785 11111 consensus_queue.cc:237] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:10.797438 10915 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:10.775287 10644 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:10.800124 10970 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:10.779978 11051 consensus_queue.cc:1048] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 13:06:10.809057 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:10.809284 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.809791 10792 raft_consensus.cc:3060] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.824841 10914 tablet_bootstrap.cc:654] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.826884 10916 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:10.827500 10916 ts_tablet_manager.cc:1403] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.248s	user 0.011s	sys 0.006s
I20260709 13:06:10.830294 10792 raft_consensus.cc:2468] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:10.831627 10972 leader_election.cc:290] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:10.832067 11014 consensus_queue.cc:1048] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.833081 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:10.833583 10684 leader_election.cc:304] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.831655 10913 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:10.834447 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0681122286764dc3b7d97a1a9962e46e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:10.844271 10867 raft_consensus.cc:3060] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.848479 11022 raft_consensus.cc:515] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.841962 10913 ts_tablet_manager.cc:1403] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.212s	user 0.035s	sys 0.037s
I20260709 13:06:10.867638 11022 leader_election.cc:290] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:10.874351 10913 raft_consensus.cc:359] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.875468 10913 raft_consensus.cc:385] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.875667 10867 raft_consensus.cc:2468] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:10.875896 10913 raft_consensus.cc:740] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.876928 10913 consensus_queue.cc:260] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.853972 10970 raft_consensus.cc:2804] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.855063 10686 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:10.832067 10915 tablet_bootstrap.cc:654] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.860018 10916 raft_consensus.cc:359] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.880525 10916 raft_consensus.cc:385] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.880991 10916 raft_consensus.cc:740] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.883059 10970 raft_consensus.cc:697] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:10.883824 10916 consensus_queue.cc:260] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.891825 10913 ts_tablet_manager.cc:1434] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.001s
I20260709 13:06:10.872650 10540 catalog_manager.cc:5697] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:10.889983 10972 raft_consensus.cc:2749] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:10.888203 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:10.903760 10792 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:10.909778 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:10.911092 10644 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:10.892081 10609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 45974.5us
W20260709 13:06:10.891021 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 92882.1us
I20260709 13:06:10.914088 10916 ts_tablet_manager.cc:1434] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.074s	user 0.011s	sys 0.000s
I20260709 13:06:10.917196 10913 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:10.923866 11014 consensus_queue.cc:1048] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:10.914601 10970 consensus_queue.cc:237] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.928560 10916 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:10.932348 10914 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:10.932945 10914 ts_tablet_manager.cc:1403] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.160s	user 0.013s	sys 0.007s
I20260709 13:06:10.940277 10792 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:10.959887 10684 leader_election.cc:304] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.962240 11184 raft_consensus.cc:2804] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:10.962832 11184 raft_consensus.cc:697] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:10.963781 11184 consensus_queue.cc:237] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:10.965278 11014 raft_consensus.cc:493] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:10.965921 11014 raft_consensus.cc:515] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:10.966454 10914 raft_consensus.cc:359] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.967492 10914 raft_consensus.cc:385] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:10.967787 10914 raft_consensus.cc:740] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:10.968688 10914 consensus_queue.cc:260] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:10.969076 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:10.969841 10866 raft_consensus.cc:2468] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:10.972755 10758 leader_election.cc:304] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:10.978660 10914 ts_tablet_manager.cc:1434] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.002s
I20260709 13:06:10.986644 10913 tablet_bootstrap.cc:654] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:10.989956 10915 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:10.990547 10915 ts_tablet_manager.cc:1403] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.193s	user 0.019s	sys 0.014s
I20260709 13:06:10.985791 10644 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:11.003043 11014 leader_election.cc:290] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:10.996091 10972 consensus_queue.cc:1048] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 13:06:11.006600 11126 raft_consensus.cc:2804] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.007354 11126 raft_consensus.cc:493] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.007827 11126 raft_consensus.cc:3060] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.044993 11178 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.045811 11178 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.048151 11178 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:11.048492 10915 raft_consensus.cc:359] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.050611 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.049487 10915 raft_consensus.cc:385] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.057859 10915 raft_consensus.cc:740] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.063258 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:11.064218 10641 raft_consensus.cc:2468] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:11.077167 10915 consensus_queue.cc:260] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.090292 10915 ts_tablet_manager.cc:1434] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.099s	user 0.019s	sys 0.003s
W20260709 13:06:11.103266 10835 leader_election.cc:343] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.104583 10913 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:11.105075 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:11.105731 10913 ts_tablet_manager.cc:1403] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.205s	user 0.011s	sys 0.011s
I20260709 13:06:11.089452 11184 consensus_queue.cc:1048] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:06:11.092141 11126 raft_consensus.cc:515] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.126677 11126 leader_election.cc:290] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:11.130775 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:11.140128 10867 raft_consensus.cc:3060] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.141883 10914 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.139585 10539 catalog_manager.cc:5697] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.124408 11178 consensus_queue.cc:1048] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 13:06:11.138018 10913 raft_consensus.cc:359] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.147207 10913 raft_consensus.cc:385] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.147545 10913 raft_consensus.cc:740] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.148275 10913 consensus_queue.cc:260] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.163597 10914 tablet_bootstrap.cc:654] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.164350 10867 raft_consensus.cc:2468] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:11.165681 10758 leader_election.cc:304] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:11.166622 11126 raft_consensus.cc:2804] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:11.167320 11126 raft_consensus.cc:697] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:11.168736 11126 consensus_queue.cc:237] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.169528 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:11.170238 10644 raft_consensus.cc:3060] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.180641 10916 tablet_bootstrap.cc:654] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:11.181958 11188 log.cc:927] Time spent T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 13:06:11.185276 10913 ts_tablet_manager.cc:1434] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.079s	user 0.007s	sys 0.000s
I20260709 13:06:11.203357 11177 raft_consensus.cc:493] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.204087 11177 raft_consensus.cc:515] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.198096 10913 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:11.208117 11177 leader_election.cc:290] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:11.222121 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:11.222951 10915 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.227278 10916 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.229684 10916 ts_tablet_manager.cc:1403] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.301s	user 0.015s	sys 0.003s
W20260709 13:06:11.223963 10834 leader_election.cc:343] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.234552 10916 raft_consensus.cc:359] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.235323 10916 raft_consensus.cc:385] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.235631 10916 raft_consensus.cc:740] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.236418 10916 consensus_queue.cc:260] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.239693 10914 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:11.240346 10914 ts_tablet_manager.cc:1403] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.099s	user 0.015s	sys 0.037s
I20260709 13:06:11.241186 10915 tablet_bootstrap.cc:654] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.243397 10914 raft_consensus.cc:359] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.244073 10914 raft_consensus.cc:385] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.244306 10914 raft_consensus.cc:740] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.244854 10914 consensus_queue.cc:260] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.245872 11184 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.246376 11184 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.253535 10916 ts_tablet_manager.cc:1434] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 13:06:11.254755 10916 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.249333 11184 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.258173 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:11.262944 10914 ts_tablet_manager.cc:1434] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 13:06:11.261437 10916 tablet_bootstrap.cc:654] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:11.268237 10684 leader_election.cc:343] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:11.273144 10832 consensus_peers.cc:597] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:11.273911 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 146682us
W20260709 13:06:11.274823 10832 leader_election.cc:343] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.275429 10832 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:11.287034 10914 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.293665 10644 raft_consensus.cc:2468] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:11.298638 10542 catalog_manager.cc:5697] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.304903 10992 raft_consensus.cc:2749] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:11.297684 11182 raft_consensus.cc:493] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.306252 11182 raft_consensus.cc:515] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.316286 11182 leader_election.cc:290] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:11.317775 10717 raft_consensus.cc:3060] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.318456 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:11.321558 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.322664 10544 catalog_manager.cc:5697] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.329660 10914 tablet_bootstrap.cc:654] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.329485 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:11.330816 10609 leader_election.cc:343] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:11.335300 10612 leader_election.cc:343] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.335817 10612 leader_election.cc:304] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:11.339862 10913 tablet_bootstrap.cc:654] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.340345 11126 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.340862 10916 tablet_bootstrap.cc:492] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.341476 10916 ts_tablet_manager.cc:1403] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.087s	user 0.016s	sys 0.038s
I20260709 13:06:11.340993 11126 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.345134 11126 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:11.345288 10915 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.346022 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:11.346489 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.347007 11177 consensus_queue.cc:1048] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:11.346065 10915 ts_tablet_manager.cc:1403] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.124s	user 0.033s	sys 0.014s
I20260709 13:06:11.348811 10758 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:11.352368 11126 raft_consensus.cc:2749] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:11.350622 11182 raft_consensus.cc:2749] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:11.363648 10915 raft_consensus.cc:359] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.367079 10915 raft_consensus.cc:385] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.367578 10915 raft_consensus.cc:740] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.369048 10915 consensus_queue.cc:260] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.386840 10915 ts_tablet_manager.cc:1434] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.036s	user 0.010s	sys 0.001s
I20260709 13:06:11.395592 10916 raft_consensus.cc:359] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.397183 10916 raft_consensus.cc:385] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.397524 10916 raft_consensus.cc:740] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.397408 11178 consensus_queue.cc:1048] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:11.398514 10916 consensus_queue.cc:260] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.403717 11126 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.404277 11126 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.410529 10916 ts_tablet_manager.cc:1434] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.061s	user 0.007s	sys 0.000s
I20260709 13:06:11.410524 10915 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.415154 11126 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.415310 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.421077 10719 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:11.422829 10761 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:11.424003 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.424867 10866 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:11.427392 10913 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:11.428026 11126 raft_consensus.cc:2804] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.428544 11126 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.428892 11126 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.430979 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.431756 10869 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:11.428185 10913 ts_tablet_manager.cc:1403] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.241s	user 0.024s	sys 0.014s
I20260709 13:06:11.436990 10916 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.442490 10913 raft_consensus.cc:359] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.443374 10913 raft_consensus.cc:385] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.444005 10913 raft_consensus.cc:740] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.445582 10913 consensus_queue.cc:260] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.449070 10970 consensus_queue.cc:1048] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:06:11.454842 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:11.455911 10914 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:11.458046 10914 ts_tablet_manager.cc:1403] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.171s	user 0.026s	sys 0.030s
I20260709 13:06:11.469026 10913 ts_tablet_manager.cc:1434] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.000s
I20260709 13:06:11.470742 10684 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:11.471915 10972 raft_consensus.cc:2804] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 13:06:11.472223 10832 leader_election.cc:343] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.472682 11120 consensus_queue.cc:1048] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:11.472868 10832 leader_election.cc:304] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:11.472326 10972 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.481618 10972 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.477298 10916 tablet_bootstrap.cc:654] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.483649 11051 raft_consensus.cc:2749] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:11.474647 10913 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:11.480763 10791 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.480549 10914 raft_consensus.cc:359] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.487834 10914 raft_consensus.cc:385] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.488186 10914 raft_consensus.cc:740] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.489423 10914 consensus_queue.cc:260] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.493760 10914 ts_tablet_manager.cc:1434] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.035s	user 0.009s	sys 0.000s
I20260709 13:06:11.495289 10914 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.501214 10972 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.511952 10916 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.512614 10916 ts_tablet_manager.cc:1403] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.089s	user 0.008s	sys 0.007s
I20260709 13:06:11.518386 11120 consensus_queue.cc:1048] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:11.528416 10913 tablet_bootstrap.cc:654] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.534164 10915 tablet_bootstrap.cc:654] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.533944 11120 consensus_queue.cc:1048] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:11.520512 11126 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.543025 10914 tablet_bootstrap.cc:654] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.550954 10972 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.551498 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:11.558100 10793 raft_consensus.cc:2393] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:11.568427 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:11.569168 10865 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.576964 11014 consensus_queue.cc:1048] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:11.581573 10916 raft_consensus.cc:359] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.582726 10916 raft_consensus.cc:385] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.583413 10916 raft_consensus.cc:740] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.588044 10916 consensus_queue.cc:260] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.591099 10916 ts_tablet_manager.cc:1434] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.078s	user 0.009s	sys 0.001s
I20260709 13:06:11.597484 11126 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.598130 10916 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.598057 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:11.598800 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:11.599494 10717 raft_consensus.cc:2393] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:11.601729 10761 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:11.603612 11177 raft_consensus.cc:493] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.604383 11177 raft_consensus.cc:515] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.607829 11014 raft_consensus.cc:2749] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:11.608273 11177 leader_election.cc:290] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:11.609364 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:11.611189 10834 leader_election.cc:343] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.603781 10865 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:11.613266 10916 tablet_bootstrap.cc:654] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.616291 10684 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:11.619235 10915 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.619817 10915 ts_tablet_manager.cc:1403] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.210s	user 0.012s	sys 0.000s
I20260709 13:06:11.620611 10913 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:11.621238 10913 ts_tablet_manager.cc:1403] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.151s	user 0.012s	sys 0.003s
I20260709 13:06:11.622853 10970 raft_consensus.cc:2804] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:11.623657 10970 raft_consensus.cc:697] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:11.624619 10970 consensus_queue.cc:237] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.643589 11107 consensus_queue.cc:1048] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:11.648005 10913 raft_consensus.cc:359] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.653024 10913 raft_consensus.cc:385] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.653393 10913 raft_consensus.cc:740] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.654719 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.655499 10717 raft_consensus.cc:2468] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:11.657824 10913 consensus_queue.cc:260] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.659292 10835 leader_election.cc:304] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:11.660478 10915 raft_consensus.cc:359] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.661257 10915 raft_consensus.cc:385] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.661542 10915 raft_consensus.cc:740] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.662356 10915 consensus_queue.cc:260] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.664912 10915 ts_tablet_manager.cc:1434] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 13:06:11.665252 10914 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:11.665856 10914 ts_tablet_manager.cc:1403] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.171s	user 0.020s	sys 0.006s
I20260709 13:06:11.666059 10915 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.668370 10914 raft_consensus.cc:359] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.669061 10914 raft_consensus.cc:385] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.677585 11177 raft_consensus.cc:2804] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.678288 11177 raft_consensus.cc:493] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.678718 11177 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.669339 10914 raft_consensus.cc:740] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.682149 10914 consensus_queue.cc:260] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.689323 10916 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.689925 10913 ts_tablet_manager.cc:1434] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.068s	user 0.007s	sys 0.003s
I20260709 13:06:11.690001 10916 ts_tablet_manager.cc:1403] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.092s	user 0.009s	sys 0.006s
I20260709 13:06:11.695287 10914 ts_tablet_manager.cc:1434] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.029s	user 0.004s	sys 0.002s
I20260709 13:06:11.699050 10913 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:11.700680 10914 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.721218 10916 raft_consensus.cc:359] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.722091 10916 raft_consensus.cc:385] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.722383 10916 raft_consensus.cc:740] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.723192 10916 consensus_queue.cc:260] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.728597 10540 catalog_manager.cc:5697] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.731016 10916 ts_tablet_manager.cc:1434] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 13:06:11.732645 10914 tablet_bootstrap.cc:654] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.724476 11177 raft_consensus.cc:515] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.737143 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:11.737378 11177 leader_election.cc:290] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:11.738749 10834 leader_election.cc:343] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.720240 10913 tablet_bootstrap.cc:654] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.743273 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:11.743978 10719 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.747910 10915 tablet_bootstrap.cc:654] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.761389 10913 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:11.764015 10913 ts_tablet_manager.cc:1403] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.066s	user 0.015s	sys 0.011s
I20260709 13:06:11.755126 10916 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.767261 10913 raft_consensus.cc:359] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.767977 10913 raft_consensus.cc:385] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.768253 10913 raft_consensus.cc:740] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.769147 10913 consensus_queue.cc:260] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.762872 10719 raft_consensus.cc:2468] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:11.771762 10914 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:11.772277 10914 ts_tablet_manager.cc:1403] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.072s	user 0.007s	sys 0.014s
I20260709 13:06:11.773597 10835 leader_election.cc:304] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:11.774935 11177 raft_consensus.cc:2804] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:11.774997 10914 raft_consensus.cc:359] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.775833 10914 raft_consensus.cc:385] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.775599 11177 raft_consensus.cc:697] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:11.776149 10914 raft_consensus.cc:740] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.776767 11177 consensus_queue.cc:237] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.777732 10913 ts_tablet_manager.cc:1434] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:11.779778 10913 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:11.782552 10915 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.783223 10915 ts_tablet_manager.cc:1403] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.117s	user 0.012s	sys 0.002s
I20260709 13:06:11.786187 10915 raft_consensus.cc:359] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.776897 10914 consensus_queue.cc:260] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.787148 10915 raft_consensus.cc:385] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.787606 10915 raft_consensus.cc:740] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.787863 10916 tablet_bootstrap.cc:654] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.788724 10915 consensus_queue.cc:260] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.790771 11182 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.791302 11182 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.791688 10915 ts_tablet_manager.cc:1434] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:11.791610 10540 catalog_manager.cc:5697] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:11.793183 10915 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.803013 10916 tablet_bootstrap.cc:492] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.804055 10916 ts_tablet_manager.cc:1403] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.049s	user 0.015s	sys 0.004s
I20260709 13:06:11.805462 11182 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.805241 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:11.811524 10915 tablet_bootstrap.cc:654] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:11.812000 10609 leader_election.cc:343] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.813668 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.814476 10865 raft_consensus.cc:2468] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:11.817247 10914 ts_tablet_manager.cc:1434] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.045s	user 0.008s	sys 0.001s
I20260709 13:06:11.817826 10609 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:11.818524 10914 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.819207 11182 raft_consensus.cc:2804] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.819658 11182 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.820340 11182 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.821827 11136 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.822595 10916 raft_consensus.cc:359] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.822692 11136 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.823534 10916 raft_consensus.cc:385] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.823925 10916 raft_consensus.cc:740] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.824682 10916 consensus_queue.cc:260] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.827183 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.827891 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:11.828572 10791 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
W20260709 13:06:11.828917 10612 leader_election.cc:343] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.825570 10915 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.829965 10915 ts_tablet_manager.cc:1403] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.037s	user 0.011s	sys 0.004s
I20260709 13:06:11.826330 11136 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:11.833412 10915 raft_consensus.cc:359] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.835703 10915 raft_consensus.cc:385] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.836054 10915 raft_consensus.cc:740] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.835830 10609 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:11.837883 11136 raft_consensus.cc:2804] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.838377 11136 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.838845 11136 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.839394 10913 tablet_bootstrap.cc:654] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.849357 11136 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.850383 11182 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.853615 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
W20260709 13:06:11.855376 10612 leader_election.cc:343] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.856168 11136 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:11.858862 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:11.859098 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:11.859839 10865 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.860467 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:11.859606 10794 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.863502 10914 tablet_bootstrap.cc:654] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.859030 10915 consensus_queue.cc:260] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.868932 10915 ts_tablet_manager.cc:1434] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.039s	user 0.005s	sys 0.000s
I20260709 13:06:11.870174 10915 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.853242 11182 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:11.880694 10916 ts_tablet_manager.cc:1434] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.076s	user 0.005s	sys 0.004s
I20260709 13:06:11.884130 10916 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.884435 10915 tablet_bootstrap.cc:654] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.885262 10794 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:11.889819 10913 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:11.890507 10913 ts_tablet_manager.cc:1403] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.111s	user 0.013s	sys 0.004s
I20260709 13:06:11.891229 10916 tablet_bootstrap.cc:654] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.894366 10913 raft_consensus.cc:359] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.895644 10913 raft_consensus.cc:385] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.896076 10913 raft_consensus.cc:740] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
W20260709 13:06:11.883358 10609 leader_election.cc:343] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.897370 10913 consensus_queue.cc:260] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:11.910157 10913 ts_tablet_manager.cc:1434] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.002s
I20260709 13:06:11.906075 10865 raft_consensus.cc:2468] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:11.913841 10913 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:11.911902 10609 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:11.912279 10915 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.915526 10915 ts_tablet_manager.cc:1403] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.046s	user 0.010s	sys 0.010s
I20260709 13:06:11.916857 11222 raft_consensus.cc:2804] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:11.917462 11222 raft_consensus.cc:697] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:11.919162 11222 consensus_queue.cc:237] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.923334 10916 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:11.925318 10916 ts_tablet_manager.cc:1403] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.005s
I20260709 13:06:11.925104 10915 raft_consensus.cc:359] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.924649 11223 raft_consensus.cc:493] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.927291 10915 raft_consensus.cc:385] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.927636 11223 raft_consensus.cc:515] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:11.928586 10915 raft_consensus.cc:740] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.928179 10914 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:11.929765 10914 ts_tablet_manager.cc:1403] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.112s	user 0.023s	sys 0.011s
I20260709 13:06:11.930219 11223 leader_election.cc:290] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:11.929616 10915 consensus_queue.cc:260] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.930850 11126 raft_consensus.cc:493] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:11.931423 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.931380 11126 raft_consensus.cc:515] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.932353 10719 raft_consensus.cc:2393] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:11.932737 10915 ts_tablet_manager.cc:1434] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 13:06:11.933045 10913 tablet_bootstrap.cc:654] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.933911 10915 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:11.935030 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:11.936692 10760 leader_election.cc:343] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:11.937023 10914 raft_consensus.cc:359] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.938202 10914 raft_consensus.cc:385] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.938476 10914 raft_consensus.cc:740] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.939319 10914 consensus_queue.cc:260] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:11.941239 10915 tablet_bootstrap.cc:654] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:11.942088 11126 leader_election.cc:290] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:11.943862 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:11.923779 10609 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:11.944608 10719 raft_consensus.cc:2468] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:11.945389 11223 raft_consensus.cc:2804] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:11.946635 11223 raft_consensus.cc:697] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:11.947993 11223 consensus_queue.cc:237] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.953650 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:11.954164 10914 ts_tablet_manager.cc:1434] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 13:06:11.955374 10761 leader_election.cc:304] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:11.955643 10609 leader_election.cc:304] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:11.956696 11182 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.956772 11126 raft_consensus.cc:2804] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:11.964449 10914 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:11.960180 11126 raft_consensus.cc:493] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:11.965969 11126 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:11.966461 10916 raft_consensus.cc:359] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.967299 10916 raft_consensus.cc:385] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:11.967628 10916 raft_consensus.cc:740] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:11.968557 10916 consensus_queue.cc:260] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:11.979550 10916 ts_tablet_manager.cc:1434] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 13:06:11.983880 10916 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:11.988409 10915 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:11.989238 10915 ts_tablet_manager.cc:1403] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.056s	user 0.014s	sys 0.006s
I20260709 13:06:11.999621 10970 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.000278 10970 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.002718 10916 tablet_bootstrap.cc:654] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.039839 11014 raft_consensus.cc:493] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.040721 11014 raft_consensus.cc:515] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.012427 11182 raft_consensus.cc:2749] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.047891 10970 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:12.044440 11014 leader_election.cc:290] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:12.054100 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:12.045383 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.056862 10791 raft_consensus.cc:2393] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate bf9fc3264f844470bc0ea04312ff2dd6 in this term.
I20260709 13:06:12.048137 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.003823 10915 raft_consensus.cc:359] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.057463 10865 raft_consensus.cc:2468] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.046237 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.059216 10915 raft_consensus.cc:385] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.060981 10915 raft_consensus.cc:740] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.066084 10684 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:12.066825 10915 consensus_queue.cc:260] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.068471 11022 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.061211 10642 raft_consensus.cc:2468] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.075176 10760 leader_election.cc:304] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.077023 10970 consensus_queue.cc:1048] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.077867 11107 raft_consensus.cc:2804] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.078255 11107 raft_consensus.cc:493] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.078573 11107 raft_consensus.cc:3060] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.084738 10913 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:12.085470 10913 ts_tablet_manager.cc:1403] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.172s	user 0.005s	sys 0.010s
I20260709 13:06:12.031819 11126 raft_consensus.cc:515] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.088201 11014 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.090724 11014 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.092398 11126 leader_election.cc:290] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.095690 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:12.097676 10913 raft_consensus.cc:359] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.098462 10913 raft_consensus.cc:385] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.098728 10913 raft_consensus.cc:740] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.098912 10915 ts_tablet_manager.cc:1434] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.109s	user 0.005s	sys 0.007s
I20260709 13:06:12.099514 10913 consensus_queue.cc:260] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.100160 10915 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:12.096892 10642 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.102695 11014 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.089706 10914 tablet_bootstrap.cc:654] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.104683 10542 catalog_manager.cc:5697] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:12.116827 10916 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:12.117427 10916 ts_tablet_manager.cc:1403] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.134s	user 0.016s	sys 0.014s
I20260709 13:06:12.114007 11022 raft_consensus.cc:2749] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.134913 10916 raft_consensus.cc:359] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.135538 10916 raft_consensus.cc:385] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.135926 10916 raft_consensus.cc:740] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.137241 10913 ts_tablet_manager.cc:1434] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.047s	user 0.003s	sys 0.005s
I20260709 13:06:12.136503 10916 consensus_queue.cc:260] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.142987 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.143741 10867 raft_consensus.cc:2468] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.148343 11178 raft_consensus.cc:493] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.148938 11178 raft_consensus.cc:515] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.151430 11178 leader_election.cc:290] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.152204 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.152936 10644 raft_consensus.cc:2468] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:12.153470 10758 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:12.153872 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.154664 10970 consensus_queue.cc:1048] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:12.155539 10834 leader_election.cc:304] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.156924 11014 raft_consensus.cc:2804] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.157364 11014 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.157739 11014 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.161955 10915 tablet_bootstrap.cc:654] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.162396 11051 raft_consensus.cc:2804] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.172147 11051 raft_consensus.cc:493] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.172837 11051 raft_consensus.cc:3060] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:12.173195 10835 leader_election.cc:343] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.165702 11178 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.175050 11178 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.182135 11182 raft_consensus.cc:493] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.182869 11182 raft_consensus.cc:515] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.195175 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.195968 11178 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:12.197707 10609 leader_election.cc:343] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.198827 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:12.200505 10834 leader_election.cc:343] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.201359 10916 ts_tablet_manager.cc:1434] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.084s	user 0.007s	sys 0.000s
I20260709 13:06:12.185266 11182 leader_election.cc:290] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.202808 10916 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:12.149274 11107 raft_consensus.cc:515] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.209227 10913 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:12.199142 11014 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.228143 10914 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:12.229270 10914 ts_tablet_manager.cc:1403] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.267s	user 0.018s	sys 0.017s
I20260709 13:06:12.231771 10913 tablet_bootstrap.cc:654] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.233357 10914 raft_consensus.cc:359] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.239423 10914 raft_consensus.cc:385] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.240274 10914 raft_consensus.cc:740] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.246177 10970 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.246825 10970 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.247548 11178 consensus_queue.cc:1048] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:12.248857 10970 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.251665 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.252758 10867 raft_consensus.cc:2468] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:12.254196 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:12.254949 10717 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.254766 10684 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:12.257457 11022 raft_consensus.cc:2804] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.257908 11022 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.258279 11022 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.266628 11177 consensus_queue.cc:1048] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.271462 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:12.272414 10793 raft_consensus.cc:2393] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:12.276727 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:12.278789 10832 leader_election.cc:343] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.279469 10832 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:12.252722 10914 consensus_queue.cc:260] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.284369 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.284623 11177 raft_consensus.cc:2749] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.283166 10642 raft_consensus.cc:2468] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.311558 10916 tablet_bootstrap.cc:654] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.314832 10760 leader_election.cc:304] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.318186 10914 ts_tablet_manager.cc:1434] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.088s	user 0.009s	sys 0.001s
I20260709 13:06:12.321921 11014 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.324157 11126 raft_consensus.cc:2804] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:12.324801 11126 raft_consensus.cc:697] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:12.325719 11126 consensus_queue.cc:237] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.312359 10717 raft_consensus.cc:2468] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.346354 10914 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:12.307403 11051 raft_consensus.cc:515] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:12.338380 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 97305.1us
I20260709 13:06:12.362078 10913 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:12.367038 10913 ts_tablet_manager.cc:1403] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.160s	user 0.023s	sys 0.031s
I20260709 13:06:12.363116 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:12.364420 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.370257 11107 leader_election.cc:290] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:12.374979 10914 tablet_bootstrap.cc:654] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.371618 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:12.383074 11246 raft_consensus.cc:493] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.383859 11246 raft_consensus.cc:515] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.386317 10915 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:12.386962 10915 ts_tablet_manager.cc:1403] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.287s	user 0.012s	sys 0.028s
I20260709 13:06:12.386763 11223 consensus_queue.cc:1048] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:12.388372 11246 leader_election.cc:290] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:12.381629 10642 raft_consensus.cc:3060] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.390306 10915 raft_consensus.cc:359] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.391099 10915 raft_consensus.cc:385] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.391377 10915 raft_consensus.cc:740] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.392154 10915 consensus_queue.cc:260] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.398488 10915 ts_tablet_manager.cc:1434] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:06:12.399483 10913 raft_consensus.cc:359] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.400118 10915 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:12.400102 10913 raft_consensus.cc:385] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.401692 10913 raft_consensus.cc:740] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.402900 10913 consensus_queue.cc:260] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.404719 10913 ts_tablet_manager.cc:1434] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.037s	user 0.005s	sys 0.000s
I20260709 13:06:12.411556 10915 tablet_bootstrap.cc:654] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.414064 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a0288d9025e4e1fa0e2269184186f0c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:12.419044 11223 consensus_queue.cc:1048] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.419492 10869 raft_consensus.cc:3060] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.415383 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:12.418383 11051 leader_election.cc:290] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.394579 11239 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.423326 10913 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:12.424803 10540 catalog_manager.cc:5697] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:12.421862 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 143136us
I20260709 13:06:12.426887 10865 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.408470 11022 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.456194 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:12.456928 10721 raft_consensus.cc:3060] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.458252 10716 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:12.454747 10612 leader_election.cc:343] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.463734 10612 leader_election.cc:304] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:12.465164 10719 raft_consensus.cc:2393] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:12.468003 11182 raft_consensus.cc:2749] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.469554 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "58ce1e65607f4e02b082593d502be0e9" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:12.470242 10644 raft_consensus.cc:3060] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.476413 10869 raft_consensus.cc:2468] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.430545 11239 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.511821 10794 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.520285 10914 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:12.521023 10914 ts_tablet_manager.cc:1403] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.177s	user 0.034s	sys 0.019s
I20260709 13:06:12.495815 10721 raft_consensus.cc:2468] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
W20260709 13:06:12.507453 10835 leader_election.cc:343] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.522627 10642 raft_consensus.cc:2468] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.553725 10914 raft_consensus.cc:359] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.554628 10914 raft_consensus.cc:385] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.547431 10865 raft_consensus.cc:2468] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.554944 10914 raft_consensus.cc:740] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.555840 10914 consensus_queue.cc:260] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.558524 10914 ts_tablet_manager.cc:1434] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.000s
I20260709 13:06:12.559626 10914 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:12.560179 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:12.572394 10914 tablet_bootstrap.cc:654] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.576575 11022 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.584101 10916 tablet_bootstrap.cc:492] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:12.587844 10644 raft_consensus.cc:2468] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:12.578570 10760 leader_election.cc:304] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.589481 11236 raft_consensus.cc:2804] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:06:12.589789 10832 leader_election.cc:343] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.590123 11236 raft_consensus.cc:697] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:12.590423 10832 leader_election.cc:304] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:12.591053 11236 consensus_queue.cc:237] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.591401 11051 raft_consensus.cc:2749] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:12.594241 11249 log.cc:927] Time spent T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 13:06:12.594153 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:12.595180 10869 raft_consensus.cc:2393] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:12.596886 11239 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:12.584929 10916 ts_tablet_manager.cc:1403] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.382s	user 0.020s	sys 0.008s
I20260709 13:06:12.602444 11182 consensus_queue.cc:1048] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.604697 10916 raft_consensus.cc:359] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.604702 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.579916 10835 leader_election.cc:304] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:12.609328 10916 raft_consensus.cc:385] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.613188 10542 catalog_manager.cc:5697] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:12.615808 10916 raft_consensus.cc:740] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.626530 11223 consensus_queue.cc:1048] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.616819 11182 raft_consensus.cc:493] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.631412 10916 consensus_queue.cc:260] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:12.641770 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 52271.4us
I20260709 13:06:12.621295 11051 raft_consensus.cc:2804] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:12.643389 11051 raft_consensus.cc:697] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:12.644387 11051 consensus_queue.cc:237] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.646056 11182 raft_consensus.cc:515] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.647285 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:12.648120 10793 raft_consensus.cc:2393] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:12.651026 11182 leader_election.cc:290] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
W20260709 13:06:12.620368 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 46307.9us
I20260709 13:06:12.658900 10684 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:12.661929 11022 raft_consensus.cc:2749] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.668031 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:12.670888 10913 tablet_bootstrap.cc:654] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.672026 10916 ts_tablet_manager.cc:1434] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.002s
I20260709 13:06:12.685654 10916 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:12.695847 10609 leader_election.cc:343] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:12.677582 10612 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 49375.9us
I20260709 13:06:12.703692 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.708061 10915 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:12.708693 10915 ts_tablet_manager.cc:1403] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.309s	user 0.016s	sys 0.018s
I20260709 13:06:12.711779 10915 raft_consensus.cc:359] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.712489 10915 raft_consensus.cc:385] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.712769 10915 raft_consensus.cc:740] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.701153 10758 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:12.716130 11238 raft_consensus.cc:2804] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:12.716627 11238 raft_consensus.cc:697] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:12.717535 11238 consensus_queue.cc:237] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.713456 10915 consensus_queue.cc:260] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.726152 10915 ts_tablet_manager.cc:1434] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 13:06:12.732195 10914 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:12.732877 10914 ts_tablet_manager.cc:1403] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.173s	user 0.013s	sys 0.007s
I20260709 13:06:12.732245 10915 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:12.742830 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.743438 11051 raft_consensus.cc:493] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.744076 11051 raft_consensus.cc:515] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.746760 11051 leader_election.cc:290] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:12.747560 11126 consensus_queue.cc:1048] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:12.753882 10758 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:12.755664 10914 raft_consensus.cc:359] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.756074 11107 raft_consensus.cc:2749] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.756429 10914 raft_consensus.cc:385] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.756745 10914 raft_consensus.cc:740] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.753857 10915 tablet_bootstrap.cc:654] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.757433 10914 consensus_queue.cc:260] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.765427 10914 ts_tablet_manager.cc:1434] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.032s	user 0.007s	sys 0.000s
I20260709 13:06:12.766631 10914 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:12.769410 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:12.771118 10832 leader_election.cc:343] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:12.773455 10612 leader_election.cc:343] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.774029 10612 leader_election.cc:304] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:12.774835 11265 raft_consensus.cc:2749] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.767321 10542 catalog_manager.cc:5697] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:12.792929 11265 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.793568 11265 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.799983 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.802110 10913 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:12.802680 10913 ts_tablet_manager.cc:1403] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.394s	user 0.011s	sys 0.007s
I20260709 13:06:12.806282 11265 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.806301 10914 tablet_bootstrap.cc:654] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.808416 10540 catalog_manager.cc:5697] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:12.815939 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:12.817860 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.818612 10869 raft_consensus.cc:2468] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:12.820309 10609 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
W20260709 13:06:12.828534 10609 leader_election.cc:343] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.828534 11260 raft_consensus.cc:2804] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.831600 11260 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.831955 11260 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:12.833046 10835 leader_election.cc:343] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.833617 10835 leader_election.cc:304] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:12.845965 10916 tablet_bootstrap.cc:654] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.853801 10913 raft_consensus.cc:359] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.854911 10913 raft_consensus.cc:385] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.855288 10913 raft_consensus.cc:740] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.855370 11260 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.856305 10913 consensus_queue.cc:260] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.861271 11126 consensus_queue.cc:1048] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:06:12.860217 10914 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:12.866156 10914 ts_tablet_manager.cc:1403] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.100s	user 0.016s	sys 0.021s
I20260709 13:06:12.869354 10914 raft_consensus.cc:359] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.870098 10914 raft_consensus.cc:385] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.870383 10914 raft_consensus.cc:740] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.863266 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:12.871573 10914 consensus_queue.cc:260] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.872080 10869 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.875355 11051 raft_consensus.cc:2749] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:12.863849 10916 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:12.886353 10916 ts_tablet_manager.cc:1403] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.207s	user 0.014s	sys 0.000s
I20260709 13:06:12.863440 11260 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.886310 10914 ts_tablet_manager.cc:1434] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.008s
I20260709 13:06:12.889628 10916 raft_consensus.cc:359] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.890439 10916 raft_consensus.cc:385] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.890763 10916 raft_consensus.cc:740] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.891114 10914 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:12.891664 10916 consensus_queue.cc:260] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.891996 10869 raft_consensus.cc:2468] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:12.861888 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:12.894088 10916 ts_tablet_manager.cc:1434] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:12.904778 10609 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.905581 10916 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:12.905915 10609 leader_election.cc:343] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:12.906200 11260 raft_consensus.cc:2804] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:12.907116 11260 raft_consensus.cc:697] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:12.893604 10542 catalog_manager.cc:5697] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:12.896478 10913 ts_tablet_manager.cc:1434] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.093s	user 0.009s	sys 0.000s
I20260709 13:06:12.907972 11260 consensus_queue.cc:237] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.898603 10915 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:12.898854 10914 tablet_bootstrap.cc:654] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:12.897725 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 64723.7us
I20260709 13:06:12.909864 10913 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:12.915555 11238 raft_consensus.cc:493] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.916311 11238 raft_consensus.cc:515] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.918243 11126 raft_consensus.cc:493] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.918848 11126 raft_consensus.cc:515] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:12.920099 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.920825 10645 raft_consensus.cc:2468] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.921049 11126 leader_election.cc:290] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:12.922334 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.922477 10760 leader_election.cc:304] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.923051 10718 raft_consensus.cc:2468] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.923858 10915 ts_tablet_manager.cc:1403] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.192s	user 0.015s	sys 0.027s
I20260709 13:06:12.919304 11238 leader_election.cc:290] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.925449 11126 raft_consensus.cc:2804] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.925926 11126 raft_consensus.cc:493] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.926365 11126 raft_consensus.cc:3060] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.926858 10915 raft_consensus.cc:359] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.927825 10915 raft_consensus.cc:385] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.928126 10915 raft_consensus.cc:740] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.928916 10915 consensus_queue.cc:260] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.925817 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.936939 10867 raft_consensus.cc:2468] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.938616 10758 leader_election.cc:304] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:12.933898 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:12.939872 10718 raft_consensus.cc:2468] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:12.940414 10970 raft_consensus.cc:493] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:12.941059 10970 raft_consensus.cc:515] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.943022 10914 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:12.943251 10970 leader_election.cc:290] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:12.943624 10914 ts_tablet_manager.cc:1403] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.053s	user 0.009s	sys 0.006s
I20260709 13:06:12.944756 11107 raft_consensus.cc:2804] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.945174 11107 raft_consensus.cc:493] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.945482 11107 raft_consensus.cc:3060] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:12.934659 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:12.946596 10914 raft_consensus.cc:359] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.950600 11126 raft_consensus.cc:515] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:12.950959 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:12.951727 10869 raft_consensus.cc:2468] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:12.952148 10915 ts_tablet_manager.cc:1434] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.028s	user 0.004s	sys 0.003s
I20260709 13:06:12.953416 10684 leader_election.cc:304] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:12.953894 10915 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:12.954581 11022 raft_consensus.cc:2804] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:12.955000 11022 raft_consensus.cc:493] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:12.955340 11022 raft_consensus.cc:3060] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.949859 10916 tablet_bootstrap.cc:654] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.965931 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:12.966418 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "588969ac5b174336819392cffb0cd9e7" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:12.965662 11126 leader_election.cc:290] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:12.966704 10720 raft_consensus.cc:3060] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.967111 10642 raft_consensus.cc:3060] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:12.965955 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:12.971514 10645 raft_consensus.cc:2468] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:12.974725 10916 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:12.975317 10916 ts_tablet_manager.cc:1403] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.070s	user 0.014s	sys 0.000s
I20260709 13:06:12.977564 11126 consensus_queue.cc:1048] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:12.987077 10916 raft_consensus.cc:359] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.987898 10916 raft_consensus.cc:385] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.988184 10916 raft_consensus.cc:740] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.951161 10914 raft_consensus.cc:385] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:12.989768 10914 raft_consensus.cc:740] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:12.990685 10914 consensus_queue.cc:260] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:12.992184 10916 consensus_queue.cc:260] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:12.994674 10916 ts_tablet_manager.cc:1434] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:06:12.988036 10642 raft_consensus.cc:2468] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:12.996049 10760 leader_election.cc:304] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:12.997280 11126 raft_consensus.cc:2804] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:12.997807 11126 raft_consensus.cc:697] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:12.999003 11126 consensus_queue.cc:237] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.002650 11236 consensus_queue.cc:1048] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:13.005270 10914 ts_tablet_manager.cc:1434] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.061s	user 0.007s	sys 0.002s
I20260709 13:06:13.011024 10916 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:13.013046 10914 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.019315 10720 raft_consensus.cc:2468] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.025738 11107 raft_consensus.cc:515] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.040571 10914 tablet_bootstrap.cc:654] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.029861 10916 tablet_bootstrap.cc:654] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:12.917687 10913 tablet_bootstrap.cc:654] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.031225 11022 raft_consensus.cc:515] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.036713 10915 tablet_bootstrap.cc:654] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.035987 10544 catalog_manager.cc:5697] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:13.055863 10914 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:13.056656 10914 ts_tablet_manager.cc:1403] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.050s	user 0.012s	sys 0.007s
I20260709 13:06:13.060164 11022 leader_election.cc:290] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:13.066597 11243 consensus_queue.cc:1048] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:13.068681 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.069381 10867 raft_consensus.cc:3060] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.075421 10915 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:13.076195 10915 ts_tablet_manager.cc:1403] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.123s	user 0.012s	sys 0.008s
I20260709 13:06:13.080524 10915 raft_consensus.cc:359] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.082901 10915 raft_consensus.cc:385] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.083660 10915 raft_consensus.cc:740] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.086560 10915 consensus_queue.cc:260] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.090551 10915 ts_tablet_manager.cc:1434] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 13:06:13.062340 10913 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:13.062381 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:13.063270 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.063443 11107 leader_election.cc:290] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:13.093647 10644 raft_consensus.cc:3060] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.093952 10869 raft_consensus.cc:3060] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.094771 10915 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:13.093647 10913 ts_tablet_manager.cc:1403] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.184s	user 0.007s	sys 0.006s
I20260709 13:06:13.065775 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:13.103638 10720 raft_consensus.cc:3060] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.103890 10913 raft_consensus.cc:359] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.107551 10913 raft_consensus.cc:385] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.107925 10913 raft_consensus.cc:740] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.108783 10913 consensus_queue.cc:260] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.060290 10914 raft_consensus.cc:359] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.100000 10540 catalog_manager.cc:5697] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:13.114890 10913 ts_tablet_manager.cc:1434] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:06:13.116786 10913 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:13.118072 10914 raft_consensus.cc:385] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.118458 10914 raft_consensus.cc:740] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.119933 10914 consensus_queue.cc:260] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.122363 10916 tablet_bootstrap.cc:492] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:13.123195 10869 raft_consensus.cc:2468] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:13.123462 10913 tablet_bootstrap.cc:654] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.123705 10916 ts_tablet_manager.cc:1403] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.113s	user 0.017s	sys 0.002s
I20260709 13:06:13.091146 10867 raft_consensus.cc:2468] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.125278 10684 leader_election.cc:304] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:13.125293 10758 leader_election.cc:304] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:13.126425 11233 raft_consensus.cc:2804] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.126484 11107 raft_consensus.cc:2804] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.126938 10916 raft_consensus.cc:359] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.127324 11107 raft_consensus.cc:697] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:13.127292 11233 raft_consensus.cc:697] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:13.128650 11107 consensus_queue.cc:237] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.130079 11233 consensus_queue.cc:237] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.133899 10916 raft_consensus.cc:385] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.134370 10916 raft_consensus.cc:740] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.135283 10916 consensus_queue.cc:260] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.137575 10916 ts_tablet_manager.cc:1434] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:13.139324 10916 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:13.137909 10720 raft_consensus.cc:2468] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.112151 10644 raft_consensus.cc:2468] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:13.140372 10914 ts_tablet_manager.cc:1434] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.083s	user 0.008s	sys 0.002s
I20260709 13:06:13.152786 10914 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.157096 11233 raft_consensus.cc:493] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.157670 11233 raft_consensus.cc:515] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.159924 11233 leader_election.cc:290] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:13.165031 10914 tablet_bootstrap.cc:654] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.171417 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:13.172502 10794 raft_consensus.cc:2468] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:13.177546 11178 consensus_queue.cc:1048] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:13.179284 10916 tablet_bootstrap.cc:654] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.180716 10915 tablet_bootstrap.cc:654] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.181772 10684 leader_election.cc:304] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:13.180848 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:13.183354 10869 raft_consensus.cc:2468] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:13.183857 11233 raft_consensus.cc:2804] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:13.184430 11233 raft_consensus.cc:493] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.193351 11233 raft_consensus.cc:3060] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.195096 10913 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:13.202667 10913 ts_tablet_manager.cc:1403] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.086s	user 0.022s	sys 0.017s
I20260709 13:06:13.206997 10913 raft_consensus.cc:359] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.207744 10913 raft_consensus.cc:385] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.208010 10913 raft_consensus.cc:740] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.208364 10915 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:13.208964 10914 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:13.208860 10913 consensus_queue.cc:260] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.209551 10914 ts_tablet_manager.cc:1403] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.057s	user 0.021s	sys 0.028s
I20260709 13:06:13.208964 10915 ts_tablet_manager.cc:1403] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.114s	user 0.003s	sys 0.009s
I20260709 13:06:13.211812 10913 ts_tablet_manager.cc:1434] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:06:13.212778 10914 raft_consensus.cc:359] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.212986 10915 raft_consensus.cc:359] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.213547 10914 raft_consensus.cc:385] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.213909 10915 raft_consensus.cc:385] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.213876 10914 raft_consensus.cc:740] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.214329 10915 raft_consensus.cc:740] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.214975 10914 consensus_queue.cc:260] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.213020 10913 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:13.217451 10914 ts_tablet_manager.cc:1434] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:06:13.215224 10915 consensus_queue.cc:260] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.218554 10914 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.219726 10915 ts_tablet_manager.cc:1434] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 13:06:13.220793 10915 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:13.225904 10914 tablet_bootstrap.cc:654] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.233141 11233 raft_consensus.cc:515] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.236285 10915 tablet_bootstrap.cc:654] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.237054 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:13.237861 10794 raft_consensus.cc:3060] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.244571 10914 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:13.245198 10914 ts_tablet_manager.cc:1403] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.009s
I20260709 13:06:13.227432 10544 catalog_manager.cc:5697] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:13.248096 10914 raft_consensus.cc:359] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.248797 10914 raft_consensus.cc:385] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.249060 10914 raft_consensus.cc:740] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.244439 11262 raft_consensus.cc:493] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.249920 10914 consensus_queue.cc:260] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.250064 11262 raft_consensus.cc:515] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.252044 10914 ts_tablet_manager.cc:1434] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 13:06:13.253717 11262 leader_election.cc:290] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.271050 10915 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:13.271875 10915 ts_tablet_manager.cc:1403] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.051s	user 0.007s	sys 0.007s
I20260709 13:06:13.227305 10913 tablet_bootstrap.cc:654] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.257408 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:13.258064 10794 raft_consensus.cc:2468] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:13.275292 10612 leader_election.cc:343] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.259167 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:13.264493 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6aa9d4273100419bb29366421f9a294c" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.265391 10916 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:13.265097 11107 consensus_queue.cc:1048] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:13.276150 10869 raft_consensus.cc:3060] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.276402 10916 ts_tablet_manager.cc:1403] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.137s	user 0.015s	sys 0.000s
W20260709 13:06:13.276969 10609 leader_election.cc:343] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.277491 10609 leader_election.cc:304] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.278834 11262 raft_consensus.cc:2749] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:13.280020 10916 raft_consensus.cc:359] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.280833 10916 raft_consensus.cc:385] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.281107 10916 raft_consensus.cc:740] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.282280 10916 consensus_queue.cc:260] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.267335 10914 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.254312 11178 raft_consensus.cc:493] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.283854 11178 raft_consensus.cc:515] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.284859 10869 raft_consensus.cc:2468] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:13.286697 11262 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.287225 11262 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.290843 11178 leader_election.cc:290] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.292776 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:13.293488 10720 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:13.293851 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:13.295363 10832 leader_election.cc:343] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.296093 10914 tablet_bootstrap.cc:654] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.297144 10612 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:13.298444 11260 raft_consensus.cc:2804] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:13.299006 11260 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.299482 11260 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.301434 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:13.303117 10869 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:13.265760 10540 catalog_manager.cc:5697] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:13.261461 11233 leader_election.cc:290] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:13.281033 10684 leader_election.cc:304] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:13.313881 11022 raft_consensus.cc:2804] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.314684 11022 raft_consensus.cc:697] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:13.315003 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:13.315819 10720 raft_consensus.cc:2468] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:13.315955 11022 consensus_queue.cc:237] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.318630 10835 leader_election.cc:304] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.320569 11051 raft_consensus.cc:2804] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:13.320964 11051 raft_consensus.cc:493] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.321322 11051 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.289377 11262 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:13.331116 11236 consensus_queue.cc:1048] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:13.334265 10916 ts_tablet_manager.cc:1434] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.057s	user 0.008s	sys 0.000s
I20260709 13:06:13.275048 10915 raft_consensus.cc:359] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.338575 10915 raft_consensus.cc:385] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.347370 10915 raft_consensus.cc:740] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.375424 11239 raft_consensus.cc:493] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.376303 11239 raft_consensus.cc:515] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.386999 10915 consensus_queue.cc:260] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.401086 10916 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:13.357180 11051 raft_consensus.cc:515] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.411856 11239 leader_election.cc:290] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:13.416509 10915 ts_tablet_manager.cc:1434] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.144s	user 0.011s	sys 0.000s
I20260709 13:06:13.417725 10915 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:13.430295 11051 leader_election.cc:290] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.437958 10792 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.443048 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
W20260709 13:06:13.444787 10832 leader_election.cc:343] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:13.460899 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 96704.9us
I20260709 13:06:13.465713 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:13.467039 10720 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.467944 10542 catalog_manager.cc:5697] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:13.475517 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d61cf6b014a34a9f8758702b48e6f5e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:13.476208 10914 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:13.436569 11260 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.495204 10914 ts_tablet_manager.cc:1403] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.228s	user 0.059s	sys 0.072s
I20260709 13:06:13.496733 11260 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:13.497682 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.498389 10866 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.498562 10914 raft_consensus.cc:359] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.499265 10914 raft_consensus.cc:385] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.499536 10914 raft_consensus.cc:740] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.500217 10914 consensus_queue.cc:260] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.501811 10644 raft_consensus.cc:2468] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:13.503006 11233 raft_consensus.cc:493] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.503563 11233 raft_consensus.cc:515] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.504609 10914 ts_tablet_manager.cc:1434] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:13.505887 10914 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.505972 11233 leader_election.cc:290] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.481056 10915 tablet_bootstrap.cc:654] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.482371 10916 tablet_bootstrap.cc:654] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.507092 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:13.509181 10684 leader_election.cc:343] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.510929 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:13.511540 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:13.511650 10718 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.512209 10645 raft_consensus.cc:2468] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:13.514356 10686 leader_election.cc:304] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.515672 11238 consensus_queue.cc:1048] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:13.519480 11022 raft_consensus.cc:2804] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:13.520016 11022 raft_consensus.cc:493] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.520354 11022 raft_consensus.cc:3060] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.469944 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d61cf6b014a34a9f8758702b48e6f5e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:13.411917 10913 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:13.519402 11236 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.522761 10913 ts_tablet_manager.cc:1403] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.310s	user 0.019s	sys 0.013s
I20260709 13:06:13.522688 11236 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.525916 10913 raft_consensus.cc:359] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.526716 10913 raft_consensus.cc:385] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.527048 10913 raft_consensus.cc:740] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.526736 11236 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:13.516434 11300 raft_consensus.cc:493] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.530884 10914 tablet_bootstrap.cc:654] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.532603 11260 consensus_queue.cc:1048] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:06:13.533327 10758 leader_election.cc:343] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.527733 10913 consensus_queue.cc:260] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.534968 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:13.538533 10913 ts_tablet_manager.cc:1434] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 13:06:13.482966 10720 raft_consensus.cc:2468] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:13.549018 10866 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:13.549610 10913 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:13.551304 10718 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:13.550984 11022 raft_consensus.cc:515] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.554457 10609 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:13.555756 11262 raft_consensus.cc:2804] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.570173 11262 raft_consensus.cc:697] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:13.571645 11262 consensus_queue.cc:237] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.579602 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:13.528319 11300 raft_consensus.cc:515] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.580433 10869 raft_consensus.cc:2468] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.585867 11260 consensus_queue.cc:1048] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 13:06:13.580437 10835 leader_election.cc:304] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.590003 11300 leader_election.cc:290] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.590461 10758 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:13.591504 11246 raft_consensus.cc:2804] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.592077 11246 raft_consensus.cc:697] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:13.593017 11246 consensus_queue.cc:237] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.596343 11126 raft_consensus.cc:2804] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:13.596756 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:13.598222 10718 raft_consensus.cc:2393] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:13.600572 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:13.597174 11022 leader_election.cc:290] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:13.601346 10913 tablet_bootstrap.cc:654] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.604763 10915 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
W20260709 13:06:13.604669 10684 leader_election.cc:343] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.596807 11126 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.605424 10915 ts_tablet_manager.cc:1403] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.188s	user 0.009s	sys 0.006s
I20260709 13:06:13.605614 11126 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:13.534632 10760 leader_election.cc:304] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:13.612648 10914 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:13.613288 10914 ts_tablet_manager.cc:1403] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.108s	user 0.027s	sys 0.034s
I20260709 13:06:13.613739 10915 raft_consensus.cc:359] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.614503 10915 raft_consensus.cc:385] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.614944 10915 raft_consensus.cc:740] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.615887 10915 consensus_queue.cc:260] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.613206 11236 raft_consensus.cc:2804] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:13.618775 10915 ts_tablet_manager.cc:1434] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.002s
I20260709 13:06:13.619047 11236 raft_consensus.cc:493] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:13.619429 11236 raft_consensus.cc:3060] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.619838 10915 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:13.620374 11126 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.624262 10916 tablet_bootstrap.cc:492] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:13.624830 10916 ts_tablet_manager.cc:1403] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.224s	user 0.014s	sys 0.026s
I20260709 13:06:13.634711 11233 consensus_queue.cc:1048] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:13.635538 10970 consensus_queue.cc:1048] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:13.635859 10915 tablet_bootstrap.cc:654] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.636229 10916 raft_consensus.cc:359] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.636933 10916 raft_consensus.cc:385] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.637228 10916 raft_consensus.cc:740] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.634763 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:13.638190 10641 raft_consensus.cc:3060] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.643735 10916 consensus_queue.cc:260] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:13.646137 10916 ts_tablet_manager.cc:1434] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 13:06:13.649775 10916 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:13.651934 10641 raft_consensus.cc:2468] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:13.623736 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 89194.3us
I20260709 13:06:13.652295 11236 raft_consensus.cc:515] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:13.656239 11236 leader_election.cc:290] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:13.683888 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d61cf6b014a34a9f8758702b48e6f5e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.685107 10867 raft_consensus.cc:3060] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.694566 10540 catalog_manager.cc:5697] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:13.680163 10686 leader_election.cc:304] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.703840 10970 raft_consensus.cc:2804] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.704495 10970 raft_consensus.cc:697] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:13.704183 10914 raft_consensus.cc:359] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.705371 10970 consensus_queue.cc:237] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.704999 10914 raft_consensus.cc:385] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.708856 10914 raft_consensus.cc:740] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.690721 11126 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:13.712792 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d61cf6b014a34a9f8758702b48e6f5e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:13.710909 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:13.709596 10914 consensus_queue.cc:260] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.716112 10867 raft_consensus.cc:2468] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.719825 10641 raft_consensus.cc:3060] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:13.740196 10913 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:13.741204 10913 ts_tablet_manager.cc:1403] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.192s	user 0.023s	sys 0.009s
I20260709 13:06:13.747334 10914 ts_tablet_manager.cc:1434] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.134s	user 0.007s	sys 0.003s
I20260709 13:06:13.764298 10916 tablet_bootstrap.cc:654] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:13.767648 10913 raft_consensus.cc:359] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.768469 10913 raft_consensus.cc:385] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.768770 10913 raft_consensus.cc:740] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.769606 10913 consensus_queue.cc:260] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:13.777487 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:13.783028 10913 ts_tablet_manager.cc:1434] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.000s
I20260709 13:06:13.784796 10913 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:13.740150 11126 consensus_queue.cc:1048] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 13:06:13.794806 10913 tablet_bootstrap.cc:654] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:13.795394 10609 leader_election.cc:343] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:13.800369 10609 leader_election.cc:304] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:13.801434 10914 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:13.806128 10792 raft_consensus.cc:1240] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Rejecting Update request from peer bf9fc3264f844470bc0ea04312ff2dd6 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:06:13.809201 11260 raft_consensus.cc:2749] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:13.810439 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:13.799033 10758 leader_election.cc:304] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:13.855501 10915 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:13.856165 10915 ts_tablet_manager.cc:1403] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.237s	user 0.024s	sys 0.017s
I20260709 13:06:13.858917 10641 raft_consensus.cc:2468] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:13.863066 10915 raft_consensus.cc:359] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.863932 10915 raft_consensus.cc:385] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:13.864326 10915 raft_consensus.cc:740] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:13.865202 10915 consensus_queue.cc:260] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.867895 10915 ts_tablet_manager.cc:1434] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:06:13.871335 11259 raft_consensus.cc:493] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:13.871943 11259 raft_consensus.cc:515] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:13.881057 10915 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:13.886701 10544 catalog_manager.cc:5697] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:13.924352 10609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 120782us
I20260709 13:06:13.920149 11236 raft_consensus.cc:2804] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:13.938529 10914 tablet_bootstrap.cc:654] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:13.909574 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 136966us
W20260709 13:06:13.898839 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 86382.5us
I20260709 13:06:13.947636 11236 raft_consensus.cc:697] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:13.987918 11320 raft_consensus.cc:493] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.018775 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.023936 10913 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:14.024752 10913 ts_tablet_manager.cc:1403] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.240s	user 0.013s	sys 0.027s
I20260709 13:06:14.025632 10916 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:14.026324 10916 ts_tablet_manager.cc:1403] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.377s	user 0.031s	sys 0.034s
I20260709 13:06:13.995716 11300 consensus_queue.cc:1059] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260709 13:06:14.029973 11300 raft_consensus.cc:3055] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:14.030411 11300 raft_consensus.cc:740] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:14.031401 11300 consensus_queue.cc:260] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:14.019161 11320 raft_consensus.cc:515] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.033046 11300 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:14.046033 11259 leader_election.cc:290] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:14.039027 11315 raft_consensus.cc:493] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.050405 11236 consensus_queue.cc:237] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:14.055047 10612 leader_election.cc:343] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.065896 10915 tablet_bootstrap.cc:654] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:14.048583 11315 raft_consensus.cc:515] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.076367 11243 raft_consensus.cc:493] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.040273 11238 consensus_queue.cc:1048] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
W20260709 13:06:14.126899 10761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 47191.4us
I20260709 13:06:14.087751 10760 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:14.136109 11243 raft_consensus.cc:515] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.145399 10794 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.160629 10916 raft_consensus.cc:359] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.180747 10916 raft_consensus.cc:385] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.181315 10916 raft_consensus.cc:740] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.185686 10916 consensus_queue.cc:260] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:14.192214 10761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 59888.8us
I20260709 13:06:14.184378 11318 raft_consensus.cc:493] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.188643 11320 leader_election.cc:290] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:14.194397 11233 raft_consensus.cc:493] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.202682 10916 ts_tablet_manager.cc:1434] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.176s	user 0.015s	sys 0.000s
I20260709 13:06:14.218160 10913 raft_consensus.cc:359] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.219009 10913 raft_consensus.cc:385] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.219281 10913 raft_consensus.cc:740] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.219926 10913 consensus_queue.cc:260] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.207095 11313 raft_consensus.cc:2749] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 13:06:14.200764 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 124377us
I20260709 13:06:14.228354 10914 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:14.230831 10914 ts_tablet_manager.cc:1403] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.432s	user 0.025s	sys 0.028s
I20260709 13:06:14.235209 10913 ts_tablet_manager.cc:1434] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.210s	user 0.006s	sys 0.003s
I20260709 13:06:14.229017 10540 catalog_manager.cc:5697] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:14.206148 11318 raft_consensus.cc:515] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:14.217998 11233 raft_consensus.cc:515] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:14.250085 11326 log.cc:927] Time spent T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: Append to log took a long time: real 0.120s	user 0.001s	sys 0.001s
I20260709 13:06:14.215997 11022 consensus_queue.cc:1048] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 13:06:14.257146 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.277434 10717 raft_consensus.cc:2393] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:14.258072 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:14.259346 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:14.279089 10867 raft_consensus.cc:2468] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:14.258857 11243 leader_election.cc:290] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:14.282343 10914 raft_consensus.cc:359] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.283222 10914 raft_consensus.cc:385] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.283483 10914 raft_consensus.cc:740] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.252650 11315 leader_election.cc:290] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:14.262775 11328 consensus_queue.cc:1048] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:14.267479 11318 leader_election.cc:290] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:14.293284 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:14.298178 10867 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
W20260709 13:06:14.303550 10686 leader_election.cc:343] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.304884 11233 leader_election.cc:290] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:14.308432 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
W20260709 13:06:14.309024 10609 leader_election.cc:343] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.284145 10914 consensus_queue.cc:260] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.322611 10913 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:14.309305 10640 raft_consensus.cc:2468] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
W20260709 13:06:14.331462 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:14.334018 10916 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:14.346124 10914 ts_tablet_manager.cc:1434] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.115s	user 0.008s	sys 0.001s
I20260709 13:06:14.352919 10914 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
W20260709 13:06:14.354575 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 80342.8us
I20260709 13:06:14.338932 11246 raft_consensus.cc:493] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.364033 10684 leader_election.cc:304] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:14.373127 10970 consensus_queue.cc:1048] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:14.345358 11259 consensus_queue.cc:1048] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260709 13:06:14.385010 10915 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:14.385702 10915 ts_tablet_manager.cc:1403] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.511s	user 0.014s	sys 0.023s
I20260709 13:06:14.387074 10789 raft_consensus.cc:3060] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.388731 10915 raft_consensus.cc:359] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.389493 10915 raft_consensus.cc:385] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.391266 10915 raft_consensus.cc:740] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.406931 11331 raft_consensus.cc:2804] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:14.407745 11331 raft_consensus.cc:493] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:14.408284 11331 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.423722 10915 consensus_queue.cc:260] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.365643 11246 raft_consensus.cc:515] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.430019 10916 tablet_bootstrap.cc:654] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:14.430944 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 75445.2us
I20260709 13:06:14.393532 10686 leader_election.cc:304] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
W20260709 13:06:14.436947 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 60705.5us
I20260709 13:06:14.437263 11325 raft_consensus.cc:2804] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:14.437726 11325 raft_consensus.cc:493] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:14.438067 11325 raft_consensus.cc:3060] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.441375 11331 raft_consensus.cc:515] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.443426 11246 leader_election.cc:290] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:14.446898 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:14.447299 11320 raft_consensus.cc:493] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.447665 10869 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.447862 11320 raft_consensus.cc:515] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.450613 10914 tablet_bootstrap.cc:654] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:14.444144 10913 tablet_bootstrap.cc:654] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:14.468880 11331 leader_election.cc:290] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:14.469401 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.452768 10790 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.476096 11320 leader_election.cc:290] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:14.477806 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:14.478523 10866 raft_consensus.cc:2468] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:14.479692 10684 leader_election.cc:304] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:14.483062 10915 ts_tablet_manager.cc:1434] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.097s	user 0.007s	sys 0.001s
I20260709 13:06:14.484895 10915 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:14.490723 11320 raft_consensus.cc:2804] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:14.501056 11320 raft_consensus.cc:493] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:14.515156 10793 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.516789 10787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.520893 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:14.521206 11022 consensus_queue.cc:1048] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:14.521747 10640 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.501472 11320 raft_consensus.cc:3060] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.525856 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
W20260709 13:06:14.526994 10609 leader_election.cc:343] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.527529 10609 leader_election.cc:304] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
W20260709 13:06:14.528712 10609 leader_election.cc:343] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.529134 10609 leader_election.cc:304] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:14.529973 11300 raft_consensus.cc:2749] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:14.531790 11328 raft_consensus.cc:2749] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:14.531810 10758 leader_election.cc:304] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:14.535295 11238 raft_consensus.cc:2749] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:06:14.536727 10832 leader_election.cc:343] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:14.540805 10684 leader_election.cc:343] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:14.502445 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 133523us
I20260709 13:06:14.550572 11022 consensus_queue.cc:1048] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:14.555233 10721 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.557443 10721 raft_consensus.cc:2393] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:14.548362 10869 raft_consensus.cc:2468] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:14.574746 11311 raft_consensus.cc:493] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.575575 11311 raft_consensus.cc:515] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.584133 11311 leader_election.cc:290] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:14.593134 10914 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:14.594069 10914 ts_tablet_manager.cc:1403] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.241s	user 0.011s	sys 0.008s
W20260709 13:06:14.572791 10834 leader_election.cc:343] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.614838 11337 consensus_queue.cc:1048] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 13:06:14.617143 10916 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:14.617803 10916 ts_tablet_manager.cc:1403] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.284s	user 0.022s	sys 0.021s
I20260709 13:06:14.618222 11325 raft_consensus.cc:515] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.621737 10916 raft_consensus.cc:359] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.622128 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:14.622618 10916 raft_consensus.cc:385] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.622953 10645 raft_consensus.cc:3060] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.623082 10916 raft_consensus.cc:740] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.624152 10916 consensus_queue.cc:260] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.625200 10915 tablet_bootstrap.cc:654] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:14.597684 10716 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.640121 10835 leader_election.cc:304] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:14.641587 11177 raft_consensus.cc:2749] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:14.631902 11325 leader_election.cc:290] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
W20260709 13:06:14.635682 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 76037.4us
I20260709 13:06:14.644966 10643 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:14.646994 11313 consensus_queue.cc:1048] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:14.656854 11320 raft_consensus.cc:515] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.637984 10914 raft_consensus.cc:359] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.658905 10716 raft_consensus.cc:2393] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:14.638721 10640 raft_consensus.cc:2468] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:14.666429 10788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.667315 10788 raft_consensus.cc:2468] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:14.670439 10914 raft_consensus.cc:385] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.670924 10914 raft_consensus.cc:740] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.671736 10914 consensus_queue.cc:260] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.679400 11320 leader_election.cc:290] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:14.677285 10684 leader_election.cc:304] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:14.681124 11327 raft_consensus.cc:2804] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:14.682478 10792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ebe7292f3cf4695813d0c4e643fb632" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:14.683328 10792 raft_consensus.cc:3060] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.684267 11337 consensus_queue.cc:1048] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:14.689759 10645 raft_consensus.cc:2468] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:14.694305 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:14.700376 10869 raft_consensus.cc:3060] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:14.702185 11327 raft_consensus.cc:697] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:14.703517 11327 consensus_queue.cc:237] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.697746 10686 leader_election.cc:304] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:14.706503 11022 raft_consensus.cc:2804] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:14.707033 11022 raft_consensus.cc:697] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:14.707929 11022 consensus_queue.cc:237] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:14.712922 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd614e94b3a24216a9ae872cda1709c0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.699393 11341 raft_consensus.cc:493] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.727768 10914 ts_tablet_manager.cc:1434] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.133s	user 0.009s	sys 0.002s
I20260709 13:06:14.731914 10914 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:14.737114 10916 ts_tablet_manager.cc:1434] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.119s	user 0.007s	sys 0.001s
I20260709 13:06:14.739269 11332 consensus_queue.cc:1048] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
W20260709 13:06:14.731503 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 49166us
I20260709 13:06:14.743364 10761 leader_election.cc:304] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:14.755566 10916 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:14.725143 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 44933.2us
I20260709 13:06:14.776455 11313 raft_consensus.cc:2749] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:14.724402 11341 raft_consensus.cc:515] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:14.801528 11319 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:14.820437 10913 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:14.821206 10913 ts_tablet_manager.cc:1403] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.509s	user 0.031s	sys 0.051s
I20260709 13:06:14.810962 11319 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:14.831655 11319 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:14.836453 10915 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:14.837164 10915 ts_tablet_manager.cc:1403] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.353s	user 0.024s	sys 0.033s
W20260709 13:06:14.838608 11352 log.cc:927] Time spent T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 13:06:14.840297 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:14.842474 11334 consensus_queue.cc:1048] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:14.842973 11341 leader_election.cc:290] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:14.847105 10869 raft_consensus.cc:2468] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:14.847990 10916 tablet_bootstrap.cc:654] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:14.857868 10684 leader_election.cc:343] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.859040 10684 leader_election.cc:304] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:14.860410 11325 raft_consensus.cc:2804] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:14.860967 11325 raft_consensus.cc:697] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:14.799036 10786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
W20260709 13:06:14.865788 10684 leader_election.cc:343] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:14.861976 11325 consensus_queue.cc:237] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:14.884970 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 46285.9us
I20260709 13:06:14.891724 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:14.916918 10913 raft_consensus.cc:359] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:14.929354 10913 raft_consensus.cc:385] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:14.930039 10913 raft_consensus.cc:740] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:14.958006 10913 consensus_queue.cc:260] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:14.962852 10914 tablet_bootstrap.cc:654] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:14.904600 10786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37f22f2791e488bb016469737ecef9d" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:14.980005 10835 leader_election.cc:304] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:14.972494 10792 raft_consensus.cc:2468] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:14.974536 11360 log.cc:927] Time spent T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
W20260709 13:06:14.975658 11357 log.cc:927] Time spent T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 13:06:14.998248 11178 raft_consensus.cc:2749] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.023828 10916 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:15.024408 10916 ts_tablet_manager.cc:1403] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.269s	user 0.019s	sys 0.015s
I20260709 13:06:15.020630 10913 ts_tablet_manager.cc:1434] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.199s	user 0.011s	sys 0.001s
I20260709 13:06:15.025823 10913 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:15.060374 10915 raft_consensus.cc:359] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.064003 10915 raft_consensus.cc:385] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.064399 10915 raft_consensus.cc:740] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.092238 10832 leader_election.cc:304] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:15.092959 10916 raft_consensus.cc:359] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.098692 10916 raft_consensus.cc:385] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.098989 10916 raft_consensus.cc:740] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.100384 10915 consensus_queue.cc:260] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.099725 10916 consensus_queue.cc:260] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.116285 10916 ts_tablet_manager.cc:1434] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.092s	user 0.007s	sys 0.002s
I20260709 13:06:15.117197 10916 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:15.113420 11365 raft_consensus.cc:2749] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:15.146441 10915 ts_tablet_manager.cc:1434] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.309s	user 0.008s	sys 0.003s
I20260709 13:06:15.150309 10915 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:15.227362 10914 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:15.228071 10914 ts_tablet_manager.cc:1403] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.499s	user 0.016s	sys 0.007s
I20260709 13:06:15.230569 11368 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:15.231283 11368 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.233292 10913 tablet_bootstrap.cc:654] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.259346 11368 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:15.261181 10915 tablet_bootstrap.cc:654] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.268671 10916 tablet_bootstrap.cc:654] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.269881 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:15.270772 10791 raft_consensus.cc:2393] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 2: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:15.271421 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:15.271795 10914 raft_consensus.cc:359] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.272516 10914 raft_consensus.cc:385] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.272794 10914 raft_consensus.cc:740] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.273484 10914 consensus_queue.cc:260] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.295095 11177 raft_consensus.cc:493] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.299544 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:15.301231 10914 ts_tablet_manager.cc:1434] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.073s	user 0.008s	sys 0.000s
I20260709 13:06:15.336419 11332 raft_consensus.cc:493] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.337543 11332 raft_consensus.cc:515] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.304312 11177 raft_consensus.cc:515] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.340307 11332 leader_election.cc:290] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:15.342881 11177 leader_election.cc:290] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:15.347321 10645 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:15.348122 10645 raft_consensus.cc:2468] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:15.307626 10642 raft_consensus.cc:2468] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:15.357108 10914 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:15.372475 10834 leader_election.cc:304] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:15.373750 11323 raft_consensus.cc:2804] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:15.374181 11323 raft_consensus.cc:493] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:15.311470 11233 consensus_queue.cc:1048] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260709 13:06:15.374534 11323 raft_consensus.cc:3060] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.374439 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:15.381186 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.381872 10794 raft_consensus.cc:2468] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:15.382959 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:15.383916 10640 raft_consensus.cc:2468] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:15.392309 10834 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:15.393555 11337 raft_consensus.cc:2804] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:15.390931 11022 consensus_queue.cc:1048] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:06:15.394002 11337 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:15.394402 11337 raft_consensus.cc:3060] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:15.398527 10760 leader_election.cc:304] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:15.399729 11315 raft_consensus.cc:2804] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:15.400126 11315 raft_consensus.cc:493] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:15.400460 11315 raft_consensus.cc:3060] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.395525 10539 catalog_manager.cc:5697] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:15.410971 10684 consensus_peers.cc:597] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:15.413550 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:15.415457 10916 tablet_bootstrap.cc:492] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:15.423586 10916 ts_tablet_manager.cc:1403] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.307s	user 0.037s	sys 0.027s
I20260709 13:06:15.448529 10916 raft_consensus.cc:359] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.449397 10916 raft_consensus.cc:385] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.449841 10916 raft_consensus.cc:740] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.450800 10916 consensus_queue.cc:260] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.462790 10916 ts_tablet_manager.cc:1434] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 13:06:15.464087 10916 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
W20260709 13:06:15.464192 10686 leader_election.cc:343] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:15.464823 10686 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:15.466706 10914 tablet_bootstrap.cc:654] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.471483 10913 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:15.472313 10913 ts_tablet_manager.cc:1403] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.447s	user 0.015s	sys 0.013s
I20260709 13:06:15.476030 11324 raft_consensus.cc:493] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.477185 11233 raft_consensus.cc:2749] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.495314 10916 tablet_bootstrap.cc:654] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.502507 10913 raft_consensus.cc:359] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.503504 10913 raft_consensus.cc:385] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.503868 10913 raft_consensus.cc:740] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.504659 11337 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.507934 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:15.508824 10794 raft_consensus.cc:2393] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 2: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:15.504778 10913 consensus_queue.cc:260] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.515297 11337 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:15.512089 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:15.584849 10913 ts_tablet_manager.cc:1434] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.112s	user 0.008s	sys 0.001s
I20260709 13:06:15.586119 10913 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:15.527665 11327 consensus_queue.cc:1048] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:15.511910 10915 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:15.590472 10915 ts_tablet_manager.cc:1403] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.443s	user 0.013s	sys 0.022s
I20260709 13:06:15.592828 10913 tablet_bootstrap.cc:654] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.529719 11379 consensus_queue.cc:1048] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:15.519686 11324 raft_consensus.cc:515] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.538838 11315 raft_consensus.cc:515] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.596627 11324 leader_election.cc:290] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:15.598049 11315 leader_election.cc:290] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:15.598914 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:15.482025 11323 raft_consensus.cc:515] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:15.578670 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 52880.3us
I20260709 13:06:15.601816 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.602087 10865 raft_consensus.cc:2393] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7133813a08fe45908399f19d1e019ca6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:15.582726 10914 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:15.603348 10914 ts_tablet_manager.cc:1403] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.246s	user 0.023s	sys 0.008s
I20260709 13:06:15.604910 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:15.607146 10914 raft_consensus.cc:359] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.608099 10914 raft_consensus.cc:385] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.608433 10914 raft_consensus.cc:740] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.611001 11323 leader_election.cc:290] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:15.612084 10684 leader_election.cc:343] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:15.613844 10684 leader_election.cc:343] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:15.615873 10684 leader_election.cc:304] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:15.616192 11262 raft_consensus.cc:493] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.612609 10914 consensus_queue.cc:260] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.616896 11262 raft_consensus.cc:515] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.617305 11319 raft_consensus.cc:2749] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.614883 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:15.618767 10644 raft_consensus.cc:3060] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.619186 11262 leader_election.cc:290] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:15.621417 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.622434 10914 ts_tablet_manager.cc:1434] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.002s
I20260709 13:06:15.622848 10916 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
W20260709 13:06:15.623148 10609 leader_election.cc:343] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:15.623528 10914 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:15.623335 10916 ts_tablet_manager.cc:1403] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.160s	user 0.012s	sys 0.000s
I20260709 13:06:15.624962 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:15.629118 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:15.630259 10794 raft_consensus.cc:2393] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 1: Already voted for candidate 7133813a08fe45908399f19d1e019ca6 in this term.
I20260709 13:06:15.633538 10646 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95a19fc069014b2db397a4c45961acda" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:15.638336 10916 raft_consensus.cc:359] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.639199 10916 raft_consensus.cc:385] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.639509 10916 raft_consensus.cc:740] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.645264 10834 leader_election.cc:304] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
W20260709 13:06:15.646664 10609 leader_election.cc:343] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:15.647090 11341 raft_consensus.cc:2749] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.647398 10609 leader_election.cc:304] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:15.648343 11379 consensus_queue.cc:1048] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:15.614712 10915 raft_consensus.cc:359] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.649580 10915 raft_consensus.cc:385] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.648954 10916 consensus_queue.cc:260] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.649981 10915 raft_consensus.cc:740] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.651337 10915 consensus_queue.cc:260] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.651940 11262 raft_consensus.cc:2749] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.652248 10916 ts_tablet_manager.cc:1434] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 13:06:15.655934 10915 ts_tablet_manager.cc:1434] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.000s
I20260709 13:06:15.657512 10915 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:15.662006 10641 raft_consensus.cc:2468] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:15.664816 10834 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:15.666399 11341 raft_consensus.cc:2804] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:15.666998 11341 raft_consensus.cc:697] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:15.668582 11341 consensus_queue.cc:237] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.669979 10913 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:15.670816 10913 ts_tablet_manager.cc:1403] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.085s	user 0.006s	sys 0.007s
I20260709 13:06:15.674556 10916 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:15.686746 10913 raft_consensus.cc:359] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.687841 10913 raft_consensus.cc:385] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.688153 10913 raft_consensus.cc:740] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.659195 10644 raft_consensus.cc:2468] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:15.692585 10760 leader_election.cc:304] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:15.699565 10916 tablet_bootstrap.cc:654] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.700996 11315 raft_consensus.cc:2804] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:15.701737 11315 raft_consensus.cc:697] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:15.689131 10913 consensus_queue.cc:260] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.702735 11315 consensus_queue.cc:237] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.704450 10913 ts_tablet_manager.cc:1434] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.003s
I20260709 13:06:15.705938 10913 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:15.714666 10914 tablet_bootstrap.cc:654] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.722038 10913 tablet_bootstrap.cc:654] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.732066 10915 tablet_bootstrap.cc:654] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.743561 11236 raft_consensus.cc:493] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.744217 11236 raft_consensus.cc:515] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.747318 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:15.754040 11236 leader_election.cc:290] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:15.759794 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37b295fd9eb746c78bae881d54b57351" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.760180 10795 raft_consensus.cc:3060] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.763008 10758 leader_election.cc:304] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:15.775364 10916 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:15.775939 10916 ts_tablet_manager.cc:1403] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.102s	user 0.028s	sys 0.030s
I20260709 13:06:15.777878 10542 catalog_manager.cc:5697] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:15.790642 10542 catalog_manager.cc:5697] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:15.779284 10916 raft_consensus.cc:359] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.782392 10913 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:15.788183 11259 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.793203 10913 ts_tablet_manager.cc:1403] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.088s	user 0.017s	sys 0.002s
I20260709 13:06:15.793286 11259 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.796424 10913 raft_consensus.cc:359] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.797098 10913 raft_consensus.cc:385] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.797370 10913 raft_consensus.cc:740] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.797659 11259 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:15.798135 10913 consensus_queue.cc:260] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.798347 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:15.799437 10720 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:15.800515 10913 ts_tablet_manager.cc:1434] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:06:15.800453 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.801102 10916 raft_consensus.cc:385] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.801441 10916 raft_consensus.cc:740] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.801308 10866 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:15.802399 10916 consensus_queue.cc:260] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.803068 10609 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:15.803879 11259 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:15.804260 11259 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:15.804605 11259 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.804679 10916 ts_tablet_manager.cc:1434] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 13:06:15.805815 10916 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:15.792394 10542 catalog_manager.cc:5697] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:15.812148 10916 tablet_bootstrap.cc:654] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.824137 10913 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:15.824697 11259 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.830055 11315 raft_consensus.cc:2749] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:15.800808 10914 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:15.831190 10913 tablet_bootstrap.cc:654] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.850886 10913 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:15.851557 10913 ts_tablet_manager.cc:1403] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.028s	user 0.013s	sys 0.003s
I20260709 13:06:15.854687 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:15.856935 10865 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.858809 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:15.860284 10720 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:15.864529 10915 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:15.865450 11259 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:15.866025 10915 ts_tablet_manager.cc:1403] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.209s	user 0.015s	sys 0.011s
I20260709 13:06:15.869421 10915 raft_consensus.cc:359] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.870131 10915 raft_consensus.cc:385] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.870373 10915 raft_consensus.cc:740] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.871166 10915 consensus_queue.cc:260] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.831480 10914 ts_tablet_manager.cc:1403] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.208s	user 0.017s	sys 0.006s
I20260709 13:06:15.873360 10915 ts_tablet_manager.cc:1434] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:06:15.874428 10915 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:15.875243 10916 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:15.875800 10916 ts_tablet_manager.cc:1403] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.070s	user 0.011s	sys 0.022s
I20260709 13:06:15.877552 10720 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:15.879136 10612 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:15.879997 11259 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:15.879961 10865 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:15.880776 11259 raft_consensus.cc:697] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:15.881757 11259 consensus_queue.cc:237] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.878896 10916 raft_consensus.cc:359] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.886021 10916 raft_consensus.cc:385] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.886354 10916 raft_consensus.cc:740] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.888837 10915 tablet_bootstrap.cc:654] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.889823 10914 raft_consensus.cc:359] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.858657 10913 raft_consensus.cc:359] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.891004 10913 raft_consensus.cc:385] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.893896 10913 raft_consensus.cc:740] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.894690 10916 consensus_queue.cc:260] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.892827 10540 catalog_manager.cc:5697] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:15.890578 10914 raft_consensus.cc:385] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.897727 10914 raft_consensus.cc:740] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.894955 10913 consensus_queue.cc:260] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.901002 10914 consensus_queue.cc:260] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.901197 10913 ts_tablet_manager.cc:1434] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.049s	user 0.005s	sys 0.002s
I20260709 13:06:15.903080 10916 ts_tablet_manager.cc:1434] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.002s
I20260709 13:06:15.904270 10916 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:15.907285 10914 ts_tablet_manager.cc:1434] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.000s
I20260709 13:06:15.908289 10914 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: Bootstrap starting.
I20260709 13:06:15.909945 10913 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: Bootstrap starting.
I20260709 13:06:15.916018 10914 tablet_bootstrap.cc:654] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.920106 10916 tablet_bootstrap.cc:654] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.927812 10539 catalog_manager.cc:5697] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:15.929950 10914 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: No bootstrap required, opened a new log
I20260709 13:06:15.930594 10914 ts_tablet_manager.cc:1403] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.005s
I20260709 13:06:15.933776 10914 raft_consensus.cc:359] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.934508 10914 raft_consensus.cc:385] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.934788 10914 raft_consensus.cc:740] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.936841 11262 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.937398 11262 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.939706 11262 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:15.940548 10916 tablet_bootstrap.cc:492] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:15.941151 10916 ts_tablet_manager.cc:1403] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.037s	user 0.008s	sys 0.002s
I20260709 13:06:15.940960 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:15.945443 10915 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:15.946058 10915 ts_tablet_manager.cc:1403] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.072s	user 0.010s	sys 0.001s
I20260709 13:06:15.953557 10916 raft_consensus.cc:359] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.954458 10916 raft_consensus.cc:385] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.954756 10916 raft_consensus.cc:740] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.955456 10916 consensus_queue.cc:260] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:15.956553 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:15.949090 10915 raft_consensus.cc:359] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.959758 10915 raft_consensus.cc:385] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:15.960083 10915 raft_consensus.cc:740] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:15.960835 10915 consensus_queue.cc:260] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:15.962500 10915 ts_tablet_manager.cc:1434] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 13:06:15.963641 10915 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:15.970249 10916 ts_tablet_manager.cc:1434] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.029s	user 0.009s	sys 0.000s
I20260709 13:06:15.971422 10916 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: Bootstrap starting.
I20260709 13:06:15.972349 10913 tablet_bootstrap.cc:654] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.972570 10915 tablet_bootstrap.cc:654] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.970147 10540 catalog_manager.cc:5697] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:15.936549 10914 consensus_queue.cc:260] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:15.988092 10720 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:15.989391 10914 ts_tablet_manager.cc:1434] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: Time spent starting tablet: real 0.058s	user 0.007s	sys 0.000s
I20260709 13:06:15.993860 10612 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:15.977890 10916 tablet_bootstrap.cc:654] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:15.979975 11246 raft_consensus.cc:493] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:15.980501 11320 raft_consensus.cc:493] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:06:15.995388 10609 leader_election.cc:343] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:15.995766 11320 raft_consensus.cc:515] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:15.998052 11320 leader_election.cc:290] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:15.999589 11262 raft_consensus.cc:2804] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:15.999996 11262 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.000415 11262 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.000504 11246 raft_consensus.cc:515] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.003778 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1260e07a0f924664b09fbb38ee72b0a1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:16.005136 11126 raft_consensus.cc:493] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.005682 11126 raft_consensus.cc:515] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:16.007740 11126 leader_election.cc:290] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:16.008996 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:16.009807 10866 raft_consensus.cc:2468] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:16.011083 10758 leader_election.cc:304] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.011756 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.012094 11236 raft_consensus.cc:2804] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.012457 10795 raft_consensus.cc:2468] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:16.016227 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:16.016933 10720 raft_consensus.cc:2468] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 0.
I20260709 13:06:16.024922 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1260e07a0f924664b09fbb38ee72b0a1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.025776 10795 raft_consensus.cc:2468] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:16.020959 11262 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:16.034125 10832 leader_election.cc:304] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.012512 11236 raft_consensus.cc:493] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.039907 11236 raft_consensus.cc:3060] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.039992 10684 leader_election.cc:304] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.040362 11262 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:16.041018 11337 raft_consensus.cc:2804] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.041428 11337 raft_consensus.cc:493] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.041791 11337 raft_consensus.cc:3060] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.053449 10869 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:16.054615 11246 leader_election.cc:290] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:16.055603 11233 raft_consensus.cc:2804] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.056051 11233 raft_consensus.cc:493] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.056372 11233 raft_consensus.cc:3060] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:16.065150 10686 leader_election.cc:343] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.067749 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:16.070410 11337 raft_consensus.cc:515] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:16.072703 10609 leader_election.cc:343] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:16.068627 10640 raft_consensus.cc:2468] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:16.076205 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:16.076957 10719 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.079066 11379 raft_consensus.cc:493] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.079651 11379 raft_consensus.cc:515] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.082237 11379 leader_election.cc:290] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:16.090817 11337 leader_election.cc:290] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:16.086871 11233 raft_consensus.cc:515] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.098196 11246 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.098829 11246 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.101384 11246 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:16.101868 10644 raft_consensus.cc:1275] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:16.106004 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:16.107247 10795 raft_consensus.cc:3060] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.112936 11233 leader_election.cc:290] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:16.114503 10788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.115430 10788 raft_consensus.cc:2468] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:16.118623 10788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1260e07a0f924664b09fbb38ee72b0a1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:16.119488 10788 raft_consensus.cc:3060] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.127534 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ec1ed15631604b35b414fe86a4c3083c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:16.128253 10642 raft_consensus.cc:3060] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.134708 10916 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: No bootstrap required, opened a new log
I20260709 13:06:16.135329 10916 ts_tablet_manager.cc:1403] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: Time spent bootstrapping tablet: real 0.164s	user 0.020s	sys 0.013s
I20260709 13:06:16.137423 10642 raft_consensus.cc:2468] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:16.145927 10684 leader_election.cc:304] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.162817 10788 raft_consensus.cc:2468] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:16.148010 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.157861 10795 raft_consensus.cc:2468] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:16.178691 11236 raft_consensus.cc:515] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:16.182880 10913 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: No bootstrap required, opened a new log
I20260709 13:06:16.533987 10913 ts_tablet_manager.cc:1403] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent bootstrapping tablet: real 0.631s	user 0.023s	sys 0.056s
I20260709 13:06:16.150790 10916 raft_consensus.cc:359] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.538309 10916 raft_consensus.cc:385] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.538599 10916 raft_consensus.cc:740] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.538630 11126 consensus_queue.cc:1048] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:16.539328 10916 consensus_queue.cc:260] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.541010 10916 ts_tablet_manager.cc:1434] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: Time spent starting tablet: real 0.405s	user 0.008s	sys 0.000s
I20260709 13:06:16.195537 10915 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: No bootstrap required, opened a new log
I20260709 13:06:16.542472 10915 ts_tablet_manager.cc:1403] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.579s	user 0.012s	sys 0.028s
I20260709 13:06:16.307236 10719 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:16.547346 10794 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:16.531142 11236 leader_election.cc:290] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:16.552752 10612 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:16.553687 11262 raft_consensus.cc:2804] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.415664 11414 raft_consensus.cc:493] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.430297 11313 raft_consensus.cc:493] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.554661 11414 raft_consensus.cc:515] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.555295 11313 raft_consensus.cc:515] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.556919 11414 leader_election.cc:290] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
W20260709 13:06:16.457963 10609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 383527us
I20260709 13:06:16.468981 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:16.475651 10644 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:16.562192 10644 raft_consensus.cc:2468] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 0.
I20260709 13:06:16.481890 10640 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1260e07a0f924664b09fbb38ee72b0a1" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:16.482755 11246 consensus_queue.cc:1048] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:16.484969 10834 leader_election.cc:304] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
W20260709 13:06:16.564626 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 79733.7us
I20260709 13:06:16.566744 10720 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
W20260709 13:06:16.512910 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 367080us
I20260709 13:06:16.515981 11022 raft_consensus.cc:2804] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.568934 11022 raft_consensus.cc:493] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.569298 11022 raft_consensus.cc:3060] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.574913 10794 raft_consensus.cc:1275] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:16.575583 10720 raft_consensus.cc:3060] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.545274 10915 raft_consensus.cc:359] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.597126 10915 raft_consensus.cc:385] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.494246 11259 consensus_queue.cc:1048] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:16.567625 11323 raft_consensus.cc:2804] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.597513 10915 raft_consensus.cc:740] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.606315 10915 consensus_queue.cc:260] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.608675 10915 ts_tablet_manager.cc:1434] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.066s	user 0.008s	sys 0.000s
I20260709 13:06:16.581302 11022 raft_consensus.cc:515] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.590699 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "77087779392747febcc9e60b2f23f96f" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:16.610543 10640 raft_consensus.cc:3060] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.611236 10868 raft_consensus.cc:3060] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.593452 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:16.322381 11413 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:16.610947 11373 consensus_queue.cc:1048] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:16.613466 11413 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.614676 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.615581 10795 raft_consensus.cc:2468] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:16.604862 10832 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.633831 10641 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" is_pre_election: true
I20260709 13:06:16.634963 11413 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:16.635872 10642 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 0.
I20260709 13:06:16.537236 10913 raft_consensus.cc:359] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:16.650681 10913 raft_consensus.cc:385] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:16.651150 10913 raft_consensus.cc:740] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:16.666823 10913 consensus_queue.cc:260] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:16.671486 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:16.672449 10642 raft_consensus.cc:3060] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:16.676885 10609 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 116157us
I20260709 13:06:16.603451 11313 leader_election.cc:290] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:16.673470 10684 leader_election.cc:304] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.680706 11022 leader_election.cc:290] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:16.681710 11022 raft_consensus.cc:2804] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.682140 11022 raft_consensus.cc:697] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:16.685621 10720 raft_consensus.cc:2468] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:16.683154 11022 consensus_queue.cc:237] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.687565 10609 leader_election.cc:304] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:16.635531 11246 raft_consensus.cc:2804] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.693595 10761 leader_election.cc:304] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:16.696331 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:16.697260 10795 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 0.
I20260709 13:06:16.699534 10642 raft_consensus.cc:2468] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:16.701040 10686 leader_election.cc:304] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:16.692041 11262 raft_consensus.cc:697] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:16.702714 11259 raft_consensus.cc:2764] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 13:06:16.703090 11313 raft_consensus.cc:2804] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.691241 10717 raft_consensus.cc:2393] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:16.691571 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79aeb4ec11254f438869b1aae326f5c6" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:16.711743 11262 consensus_queue.cc:237] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:16.700212 11246 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:06:16.735471 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 46893.2us
I20260709 13:06:16.726756 11323 raft_consensus.cc:697] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:16.746654 11323 consensus_queue.cc:237] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:16.734969 11416 raft_consensus.cc:2804] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:16.752156 10609 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:16.735159 11313 raft_consensus.cc:697] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:16.742025 11246 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.733425 11337 consensus_queue.cc:1048] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.033s
I20260709 13:06:16.764989 11416 raft_consensus.cc:697] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:16.769304 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7cf735aa97ef4935a1f8ac140c341ceb" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
W20260709 13:06:16.766171 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 198972us
I20260709 13:06:16.774588 11418 raft_consensus.cc:2804] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:16.803256 10794 raft_consensus.cc:3060] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:16.844251 10913 ts_tablet_manager.cc:1434] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: Time spent starting tablet: real 0.310s	user 0.012s	sys 0.012s
I20260709 13:06:16.805296 11418 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:16.787120 10868 raft_consensus.cc:2468] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
W20260709 13:06:16.857167 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 138774us
I20260709 13:06:16.786855 10640 raft_consensus.cc:2468] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:16.863862 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 103371us
I20260709 13:06:16.865199 11418 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:06:16.880354 10592 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:06:16.893134 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 125060us
W20260709 13:06:16.894630 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 274325us
I20260709 13:06:16.901578 10794 raft_consensus.cc:2468] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:16.918752 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 114978us
I20260709 13:06:16.870618 11313 consensus_queue.cc:237] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:16.954399 11427 log.cc:927] Time spent T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 13:06:16.878170 11416 consensus_queue.cc:237] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:16.970382 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:17.001530 10758 leader_election.cc:304] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:16.981721 11262 consensus_queue.cc:1048] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260709 13:06:17.011183 11239 raft_consensus.cc:2749] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:06:17.035833 10832 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 88141us
W20260709 13:06:17.063086 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 46494.9us
I20260709 13:06:17.044173 11418 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:17.107317 11418 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:17.110090 10786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:17.110833 10786 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.112211 11246 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:17.113899 10866 raft_consensus.cc:2393] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:17.153162 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:17.154270 10866 raft_consensus.cc:2393] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 in current term 1: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
W20260709 13:06:17.188248 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 93886.2us
I20260709 13:06:17.209908 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:17.213783 11246 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:17.224143 10575 debug-util.cc:398] Leaking SignalData structure 0x7b0800380f60 after lost signal to thread 434
W20260709 13:06:17.225674 10575 debug-util.cc:398] Leaking SignalData structure 0x7b08003ed7a0 after lost signal to thread 10578
W20260709 13:06:17.227594 10575 debug-util.cc:398] Leaking SignalData structure 0x7b0800322420 after lost signal to thread 10670
W20260709 13:06:17.229012 10575 debug-util.cc:398] Leaking SignalData structure 0x7b0800149440 after lost signal to thread 10745
W20260709 13:06:17.238510 10575 debug-util.cc:398] Leaking SignalData structure 0x7b0800263620 after lost signal to thread 10819
W20260709 13:06:17.240094 10575 debug-util.cc:398] Leaking SignalData structure 0x7b08003b1e00 after lost signal to thread 10893
I20260709 13:06:17.241128 10786 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 1.
I20260709 13:06:17.250032 10609 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:17.250911 11417 raft_consensus.cc:2804] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:17.251416 11417 raft_consensus.cc:697] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:17.252286 11417 consensus_queue.cc:237] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:17.257877 10642 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
I20260709 13:06:17.272027 10865 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:17.283479 10834 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:17.292582 11022 consensus_queue.cc:1048] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.292420 11444 raft_consensus.cc:2749] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:17.297155 11414 consensus_queue.cc:1048] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.320683 11424 consensus_queue.cc:1048] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.337610 11417 consensus_queue.cc:1048] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:17.343122 11444 consensus_queue.cc:1048] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:06:17.433352 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 70300.5us
I20260709 13:06:17.449821 10542 catalog_manager.cc:5697] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:17.460166 11440 consensus_queue.cc:1048] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.519137 10539 catalog_manager.cc:5697] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:17.594667 11315 consensus_queue.cc:1048] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.634308 11373 consensus_queue.cc:1048] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.662348 11320 consensus_queue.cc:1048] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.691671 11262 consensus_queue.cc:1048] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.705260 11423 consensus_queue.cc:1048] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:17.746572 11421 consensus_queue.cc:1048] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 13:06:17.951372 10540 catalog_manager.cc:5697] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 1 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:18.047479 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 62834.9us
I20260709 13:06:18.113296 10544 catalog_manager.cc:5697] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:18.126801 10544 catalog_manager.cc:5697] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:18.149405 10543 catalog_manager.cc:5697] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } }
I20260709 13:06:18.897588   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
W20260709 13:06:18.905733 10834 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:06:18.949736 10834 consensus_peers.cc:597] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.954483 10760 consensus_peers.cc:597] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.956621 10760 consensus_peers.cc:597] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.958356 10834 consensus_peers.cc:597] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.959329 10686 consensus_peers.cc:597] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:18.960256 10686 consensus_peers.cc:597] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.010875 10760 consensus_peers.cc:597] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.010766   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:19.015034   431 tablet_replica.cc:333] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.018100   431 raft_consensus.cc:2243] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.020006   431 raft_consensus.cc:2272] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.023061 10834 consensus_peers.cc:597] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.023171 10686 consensus_peers.cc:597] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.023943   431 tablet_replica.cc:333] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.024843   431 raft_consensus.cc:2243] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.026013   431 raft_consensus.cc:2272] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.031131   431 tablet_replica.cc:333] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.032446   431 raft_consensus.cc:2243] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.033190   431 raft_consensus.cc:2272] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.033833 10760 consensus_peers.cc:597] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.039899   431 tablet_replica.cc:333] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.041172   431 raft_consensus.cc:2243] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.041878   431 raft_consensus.cc:2272] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.042455 10760 consensus_peers.cc:597] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.045845   431 tablet_replica.cc:333] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
W20260709 13:06:19.047473 10760 consensus_peers.cc:597] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.050468   431 raft_consensus.cc:2243] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.052378   431 raft_consensus.cc:2272] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.056046   431 tablet_replica.cc:333] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.057009   431 raft_consensus.cc:2243] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.057747   431 raft_consensus.cc:2272] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.059218 10760 consensus_peers.cc:597] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.061419   431 tablet_replica.cc:333] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.062536   431 raft_consensus.cc:2243] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.063862   431 raft_consensus.cc:2272] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.067552   431 tablet_replica.cc:333] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.068557   431 raft_consensus.cc:2243] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.069222   431 raft_consensus.cc:2272] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.069468 10686 consensus_peers.cc:597] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.072621   431 tablet_replica.cc:333] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
W20260709 13:06:19.073406 10686 consensus_peers.cc:597] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.073701   431 raft_consensus.cc:2243] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.074447   431 raft_consensus.cc:2272] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.077692   431 tablet_replica.cc:333] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.078361   431 raft_consensus.cc:2243] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.078900   431 raft_consensus.cc:2272] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.082657   431 tablet_replica.cc:333] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.083705   431 raft_consensus.cc:2243] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.085000   431 raft_consensus.cc:2272] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.088660   431 tablet_replica.cc:333] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.089674   431 raft_consensus.cc:2243] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.090375   431 raft_consensus.cc:2272] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.093758   431 tablet_replica.cc:333] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.094519   431 raft_consensus.cc:2243] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.095080   431 raft_consensus.cc:2272] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.098338   431 tablet_replica.cc:333] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.099291   431 raft_consensus.cc:2243] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.101238   431 raft_consensus.cc:2272] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.104413 10760 consensus_peers.cc:597] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.104962   431 tablet_replica.cc:333] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.108158   431 raft_consensus.cc:2243] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.108727   431 raft_consensus.cc:2272] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.112176   431 tablet_replica.cc:333] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.113397   431 raft_consensus.cc:2243] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:19.114105 10686 consensus_peers.cc:597] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.114290   431 raft_consensus.cc:2272] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.119808   431 tablet_replica.cc:333] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.120860   431 raft_consensus.cc:2243] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.121593   431 raft_consensus.cc:2272] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.125090   431 tablet_replica.cc:333] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.126132   431 raft_consensus.cc:2243] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.126874   431 raft_consensus.cc:2272] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.131080   431 tablet_replica.cc:333] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.133014   431 raft_consensus.cc:2243] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.134477   431 raft_consensus.cc:2272] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.138507   431 tablet_replica.cc:333] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.139593   431 raft_consensus.cc:2243] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.140362   431 raft_consensus.cc:2272] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.138854 10834 consensus_peers.cc:597] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.144959   431 tablet_replica.cc:333] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
W20260709 13:06:19.145326 10834 consensus_peers.cc:597] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.146064   431 raft_consensus.cc:2243] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.146821   431 raft_consensus.cc:2272] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.150306   431 tablet_replica.cc:333] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
W20260709 13:06:19.150472 10686 consensus_peers.cc:597] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.151243   431 raft_consensus.cc:2243] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.151942   431 raft_consensus.cc:2272] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.154847   431 tablet_replica.cc:333] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.155494   431 raft_consensus.cc:2243] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.156493   431 raft_consensus.cc:2272] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.159252   431 tablet_replica.cc:333] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.160048   431 raft_consensus.cc:2243] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.161326   431 raft_consensus.cc:2272] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.164814   431 tablet_replica.cc:333] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
W20260709 13:06:19.165293 10834 consensus_peers.cc:597] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.165846   431 raft_consensus.cc:2243] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.166540   431 raft_consensus.cc:2272] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.169948   431 tablet_replica.cc:333] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.170778   431 raft_consensus.cc:2243] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:19.171258 10834 consensus_peers.cc:597] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.171350   431 raft_consensus.cc:2272] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.174481   431 tablet_replica.cc:333] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.175360   431 raft_consensus.cc:2243] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.176530   431 raft_consensus.cc:2272] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.179694   431 tablet_replica.cc:333] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.180547   431 raft_consensus.cc:2243] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.182015   431 raft_consensus.cc:2272] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.184980   431 tablet_replica.cc:333] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.185791   431 raft_consensus.cc:2243] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.186333   431 raft_consensus.cc:2272] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.188841   431 tablet_replica.cc:333] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.189618   431 raft_consensus.cc:2243] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:19.190688   431 raft_consensus.cc:2272] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.193483   431 tablet_replica.cc:333] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.194260   431 raft_consensus.cc:2243] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.194861   431 raft_consensus.cc:2272] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.196630 10686 consensus_peers.cc:597] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.198331   431 tablet_replica.cc:333] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.199294   431 raft_consensus.cc:2243] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.200301   431 raft_consensus.cc:2272] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.203682   431 tablet_replica.cc:333] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.204366   431 raft_consensus.cc:2243] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.204936   431 raft_consensus.cc:2272] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.208844   431 tablet_replica.cc:333] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.209743   431 raft_consensus.cc:2243] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.210397   431 raft_consensus.cc:2272] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.214036   431 tablet_replica.cc:333] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.214968   431 raft_consensus.cc:2243] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.215708   431 raft_consensus.cc:2272] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.219223   431 tablet_replica.cc:333] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.220292   431 raft_consensus.cc:2243] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.221154   431 raft_consensus.cc:2272] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.221800 10760 consensus_peers.cc:597] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.224951   431 tablet_replica.cc:333] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.226002   431 raft_consensus.cc:2243] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.226634   431 raft_consensus.cc:2272] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.229835   431 tablet_replica.cc:333] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.230887   431 raft_consensus.cc:2243] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.231594   431 raft_consensus.cc:2272] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.235769   431 tablet_replica.cc:333] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.236852   431 raft_consensus.cc:2243] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.237601   431 raft_consensus.cc:2272] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.241273   431 tablet_replica.cc:333] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.242348   431 raft_consensus.cc:2243] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.243000   431 raft_consensus.cc:2272] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.246429   431 tablet_replica.cc:333] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.247398   431 raft_consensus.cc:2243] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.248365   431 raft_consensus.cc:2272] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.252453   431 tablet_replica.cc:333] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.253783   431 raft_consensus.cc:2243] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.254719   431 raft_consensus.cc:2272] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.259183   431 tablet_replica.cc:333] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.260233   431 raft_consensus.cc:2243] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.261057   431 raft_consensus.cc:2272] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.264423   431 tablet_replica.cc:333] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.265161   431 raft_consensus.cc:2243] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:19.265588 10834 consensus_peers.cc:597] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.265908   431 raft_consensus.cc:2272] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:19.269243   431 tablet_replica.cc:333] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: stopping tablet replica
I20260709 13:06:19.269845   431 raft_consensus.cc:2243] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:19.270370   431 raft_consensus.cc:2272] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:19.290108 10834 consensus_peers.cc:597] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.333091 10760 consensus_peers.cc:597] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.347985 10686 consensus_peers.cc:597] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.354550 10686 consensus_peers.cc:597] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.366971 10686 consensus_peers.cc:597] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.369074 10834 consensus_peers.cc:597] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.370739 10834 consensus_peers.cc:597] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.408830   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:06:19.517313 10686 consensus_peers.cc:597] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.576612 10834 consensus_peers.cc:597] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:19.578385 10834 consensus_peers.cc:597] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:19.655028   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:19.664866 11489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:19.668713 11490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:19.695690 11492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:19.699640   431 server_base.cc:1061] running on GCE node
I20260709 13:06:19.704849   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:19.705364   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:19.705719   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602379705694 us; error 0 us; skew 500 ppm
I20260709 13:06:19.706586   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:19.727707   431 webserver.cc:533] Webserver started at http://127.0.107.193:43975/ using document root <none> and password file <none>
I20260709 13:06:19.728737   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:19.729023   431 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 13:06:19.759018   431 fs_manager.cc:714] Time spent opening directory manager: real 0.027s	user 0.004s	sys 0.013s
I20260709 13:06:19.768674 11498 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:19.769804   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:06:19.770251   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
format_stamp: "Formatted at 2026-07-09 13:06:03 on dist-test-slave-p8rg"
I20260709 13:06:19.770875   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:19.789103   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:19.790757   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:19.793008   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:19.840247 11505 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 13:06:20.082396 11423 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.083455 11423 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.092255 11423 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:20.107033 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:20.117909 10686 leader_election.cc:336] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.118546 11415 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.119262 11415 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.123328 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.125986 11415 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:20.132432 10760 leader_election.cc:336] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.133363 10760 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.118570 10686 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.134528 11423 raft_consensus.cc:2749] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.134511 11415 raft_consensus.cc:2749] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.166314 11416 raft_consensus.cc:493] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.166894 11416 raft_consensus.cc:515] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.169123 11416 leader_election.cc:290] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:20.170449 11506 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.171092 11506 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.173702 11506 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.175098 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:20.176515 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.178409 11429 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.179205 11429 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:20.183630 10686 leader_election.cc:336] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.184516 10686 leader_election.cc:304] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
W20260709 13:06:20.187960 10834 leader_election.cc:336] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.188589 10834 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.188802 11423 raft_consensus.cc:2749] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.190786 11506 raft_consensus.cc:2749] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.191151 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.194916 11429 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.201407 11506 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.202169 11506 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.204749 11506 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:20.204613 10686 leader_election.cc:336] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.205353 10686 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.206449 11429 raft_consensus.cc:2749] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.207973 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
W20260709 13:06:20.215322 10834 leader_election.cc:336] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.215756 10834 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.216475 11506 raft_consensus.cc:2749] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.254041 11429 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.254598 11429 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.258268 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.258944 10864 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
W20260709 13:06:20.262578 10686 leader_election.cc:336] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.266254 11429 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.271641 11416 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.272254 11416 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.271826 10684 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.274058 11423 raft_consensus.cc:2804] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.274475 11423 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.274857 11423 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.275069 11416 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.286283 11487 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.287029 11487 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.289371 11487 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.296948 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:20.304991 11506 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.305581 11506 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:20.309617 10686 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.310593 11506 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.315749 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.316530 10864 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:20.322923 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.323691 10719 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:20.322965 10684 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.326488 11423 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:20.328711 10834 leader_election.cc:336] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.331527 10834 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.329720 11429 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.332222 11429 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.332700 11429 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.333825 11496 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.334153 11423 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.334581 11496 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:20.337423 10686 leader_election.cc:336] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.337990 11506 raft_consensus.cc:2749] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.338407 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.342211 11496 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.344139 10835 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:20.344784   431 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 13:06:20.345247   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.543s	user 0.017s	sys 0.029s
I20260709 13:06:20.345705   431 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 13:06:20.345875 11506 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.346397 11506 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.346796 11506 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:20.355073 10760 leader_election.cc:336] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.357915 11506 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.358587 10760 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.360272 11415 raft_consensus.cc:2749] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.361917 11505 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:20.366110 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:20.367378 10862 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.383519 11415 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.384313 11415 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.391561 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.406981 10864 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
W20260709 13:06:20.395684 10834 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.399097 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:20.410512 11509 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.411309 11509 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.412027 11415 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.413841 10758 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.415091 11496 raft_consensus.cc:2804] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.377205 10862 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:20.415465 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.404323 11429 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.416306 10719 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:20.417790 10761 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.400708 11506 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.418383 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:20.419827 10864 raft_consensus.cc:2393] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 2: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:20.419989 11510 raft_consensus.cc:2804] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.415558 11496 raft_consensus.cc:493] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.420640 11510 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.421175 10718 raft_consensus.cc:2393] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 2: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:20.421299 11496 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.422206 10684 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.423080 11423 raft_consensus.cc:2804] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.423629 11423 raft_consensus.cc:697] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:20.424691 11423 consensus_queue.cc:237] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.425632 11429 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.428447 11506 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.421449 11509 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:20.430590 10834 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.430624 11506 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.433004 11506 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.433275 10835 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.434374 11506 raft_consensus.cc:2749] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.442230 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.443005 10719 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:20.421339 11510 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.446201 10835 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:20.447532 11487 raft_consensus.cc:2804] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.447950 11487 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.448374 11487 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.454011 11509 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.454634 11509 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:20.456514 10834 leader_election.cc:336] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.457091 11509 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.460246 11496 raft_consensus.cc:515] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.474337 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e46c52fd7feb4776839cc022f4ef27e0" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:20.475082 10862 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.462811 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.486406 10864 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:20.488600 10758 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.489768 11415 raft_consensus.cc:2804] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.490172 11415 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.490554 11415 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.493326 11422 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.494010 11422 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:20.498162 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 54211.6us
I20260709 13:06:20.499402 11422 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:20.506352 10686 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.507056 10686 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.507853 11429 raft_consensus.cc:2749] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.499452 11415 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.510057 11496 leader_election.cc:290] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.510206 11415 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.512995 11429 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.513592 11429 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.514391 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:20.515026 10864 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.516685 11429 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.516752 10862 raft_consensus.cc:2468] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
I20260709 13:06:20.530236 10758 leader_election.cc:304] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.525343 10542 catalog_manager.cc:5697] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:20.531390 11496 raft_consensus.cc:2804] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.531909 11496 raft_consensus.cc:697] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
W20260709 13:06:20.532943 10834 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.532948 11496 consensus_queue.cc:237] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.538040 11510 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.527608 11487 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.550323 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:20.551031 10719 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.552129 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.552778 11487 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.553079 10718 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:20.555626 11506 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.556146 11506 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.561519 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:20.562304 10794 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:20.562697 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:20.563372 10718 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.566215 10832 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.567915 11422 raft_consensus.cc:2804] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.568081 11415 raft_consensus.cc:493] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.568375 11422 raft_consensus.cc:493] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.568876 11422 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.568706 11415 raft_consensus.cc:515] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.571552 11415 leader_election.cc:290] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:20.572993 10760 leader_election.cc:336] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.573473 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:20.576843 10717 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 1.
I20260709 13:06:20.574617 11506 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.578660 10761 leader_election.cc:304] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.566828 10719 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
I20260709 13:06:20.579799 11415 raft_consensus.cc:2804] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.580308 11415 raft_consensus.cc:493] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.580773 10761 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:20.588877 11509 raft_consensus.cc:2804] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.589447 11509 raft_consensus.cc:697] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:20.590408 11509 consensus_queue.cc:237] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:20.579214 10760 leader_election.cc:336] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.581081 10835 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
W20260709 13:06:20.583648 10686 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.584735 11505 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:20.578591 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:20.598217 10862 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 1.
I20260709 13:06:20.598598 11415 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.608875 11505 tablet_bootstrap.cc:492] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:20.610105 11505 ts_tablet_manager.cc:1403] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.249s	user 0.018s	sys 0.008s
W20260709 13:06:20.610191 10760 leader_election.cc:336] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:20.603220 10834 leader_election.cc:336] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.612699 10684 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
W20260709 13:06:20.613663 10760 leader_election.cc:336] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:20.617259 10760 leader_election.cc:336] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.594448 11510 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:20.561324 10864 raft_consensus.cc:2468] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
I20260709 13:06:20.624539 11507 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.625030 11507 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.625387 11507 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.601143 11506 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:20.614846 11505 raft_consensus.cc:359] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.642248 11506 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.642967 11506 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.643755 11507 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.632642 11505 raft_consensus.cc:740] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
W20260709 13:06:20.647549 10760 leader_election.cc:336] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:20.648032 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 51928.4us
W20260709 13:06:20.648756 10760 leader_election.cc:336] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.597204 10718 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:20.622591 10539 catalog_manager.cc:5697] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:20.626986 10834 leader_election.cc:336] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.649262 10758 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.653367 11505 consensus_queue.cc:260] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.666898 10835 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.669209 11507 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.662285 11510 raft_consensus.cc:2804] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.677240 11510 raft_consensus.cc:697] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:20.680047 11510 consensus_queue.cc:237] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:20.680544 10686 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.681842 11422 raft_consensus.cc:515] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.685322 11517 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:20.695829 11487 raft_consensus.cc:2804] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.709353 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:20.758514 11422 leader_election.cc:290] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.770469 11487 raft_consensus.cc:697] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
W20260709 13:06:20.771629 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 65466.9us
I20260709 13:06:20.705694 11517 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.777842 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2e822ba7018446f98e776bbd1c9e8a3" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:20.778656 10794 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:20.797294 11415 raft_consensus.cc:515] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:20.799755 11415 leader_election.cc:290] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:20.801208 10834 leader_election.cc:336] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.804406 11506 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:20.818099 10865 raft_consensus.cc:2393] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 39f0ed0891e14f20bb25882450033da4 in current term 2: Already voted for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in this term.
I20260709 13:06:20.814826 11487 consensus_queue.cc:237] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:20.819981 10834 leader_election.cc:336] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.820353 10684 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.821590 11429 raft_consensus.cc:2749] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:06:20.842298 10794 raft_consensus.cc:2468] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:20.814706 11517 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:20.878456 11506 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:20.883792 10760 leader_election.cc:336] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:20.884882 10834 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:20.911931 11505 ts_tablet_manager.cc:1434] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.301s	user 0.011s	sys 0.054s
I20260709 13:06:20.935294 11505 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:20.942132 10718 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fe860f4e3a2945b28a2c390567bc043d" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:20.943138 10718 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:20.945397 10686 consensus_peers.cc:597] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:20.956264 10788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:20.945334 10832 leader_election.cc:304] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:20.964761 10788 raft_consensus.cc:2468] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 1.
I20260709 13:06:20.966602 11422 raft_consensus.cc:2804] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:20.986125 11422 raft_consensus.cc:697] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:20.987957 11422 consensus_queue.cc:237] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:20.990883 10718 raft_consensus.cc:2468] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
W20260709 13:06:20.995841 10760 consensus_peers.cc:597] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:21.000833 10761 leader_election.cc:304] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:21.004237 11523 raft_consensus.cc:2804] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:21.005206 11523 raft_consensus.cc:697] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:21.013353 10832 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:21.016695 11526 raft_consensus.cc:2804] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:21.019452 11526 raft_consensus.cc:493] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:21.020881 11526 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:21.006945 11523 consensus_queue.cc:237] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:21.038401 10758 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 261694us
I20260709 13:06:21.060583 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:21.061777 10717 raft_consensus.cc:2393] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in current term 2: Already voted for candidate 39f0ed0891e14f20bb25882450033da4 in this term.
I20260709 13:06:21.096796 10862 raft_consensus.cc:1275] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.037535 10542 catalog_manager.cc:5697] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:21.133129 10865 raft_consensus.cc:1275] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.152649 10542 catalog_manager.cc:5697] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:21.162549 11505 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:21.163756 11505 tablet_bootstrap.cc:492] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:21.164945 11505 ts_tablet_manager.cc:1403] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.247s	user 0.025s	sys 0.000s
I20260709 13:06:21.168211 11505 raft_consensus.cc:359] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:21.169025 11505 raft_consensus.cc:740] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:21.170737 11505 consensus_queue.cc:260] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:21.173216 11505 ts_tablet_manager.cc:1434] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 13:06:21.174242 11505 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:21.154902 11524 consensus_queue.cc:1048] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:21.194425 10868 raft_consensus.cc:1275] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.204085 10719 raft_consensus.cc:1275] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.226979 10835 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:21.231663 11422 raft_consensus.cc:2749] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 13:06:21.233842 11508 consensus_queue.cc:1048] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:21.237459 10760 consensus_peers.cc:597] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:21.243021 11526 raft_consensus.cc:515] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:21.260416 11526 leader_election.cc:290] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:21.263671 10834 leader_election.cc:336] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:21.222226 11505 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:21.267658 11505 tablet_bootstrap.cc:492] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:21.273677 11505 ts_tablet_manager.cc:1403] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.100s	user 0.012s	sys 0.009s
I20260709 13:06:21.279910 10539 catalog_manager.cc:5697] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:21.290745 10539 catalog_manager.cc:5697] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:21.292768 10539 catalog_manager.cc:5697] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:21.295395 11523 consensus_queue.cc:1048] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:21.304792 10760 consensus_peers.cc:597] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.326753 10834 consensus_peers.cc:597] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.333364 11509 consensus_queue.cc:1048] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:21.337798 11505 raft_consensus.cc:359] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.357779 11505 raft_consensus.cc:740] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:21.345899 10788 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5ce3e6e278894017aa528fe91907d31c" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6"
W20260709 13:06:21.338897 10834 consensus_peers.cc:597] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.364727 10788 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:21.363725   431 ts_tablet_manager.cc:600] Registering tablets (40/45 complete)
I20260709 13:06:21.368000 10719 raft_consensus.cc:1275] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 13:06:21.372573 10834 consensus_peers.cc:597] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:21.379215 10788 raft_consensus.cc:2468] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:21.388572 11422 consensus_queue.cc:1048] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:21.393100 11505 consensus_queue.cc:260] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.396523 11505 ts_tablet_manager.cc:1434] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.107s	user 0.007s	sys 0.000s
I20260709 13:06:21.401446 11505 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:21.428478 10760 consensus_peers.cc:597] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:21.438884 10686 consensus_peers.cc:597] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.454650 10686 consensus_peers.cc:597] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.424382   431 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 13:06:21.457947   431 ts_tablet_manager.cc:595] Time spent register tablets: real 1.112s	user 0.356s	sys 0.013s
I20260709 13:06:21.490191 10832 leader_election.cc:304] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:21.493561 11526 raft_consensus.cc:2804] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:21.501022 11526 raft_consensus.cc:697] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:21.502156 11526 consensus_queue.cc:237] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
W20260709 13:06:21.523300 10834 consensus_peers.cc:597] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.546099 10760 consensus_peers.cc:597] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.546765 10760 consensus_peers.cc:597] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:21.572685 10760 consensus_peers.cc:597] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.585697 10834 consensus_peers.cc:597] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:21.599412 10834 consensus_peers.cc:597] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.604400 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 72870.1us
W20260709 13:06:21.623347 10686 consensus_peers.cc:597] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.643513 10686 consensus_peers.cc:597] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.630338 10760 consensus_peers.cc:597] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.644657 10760 consensus_peers.cc:597] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.651467 11505 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:21.652993 11505 tablet_bootstrap.cc:492] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
W20260709 13:06:21.652864 10760 consensus_peers.cc:597] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.654194 10760 consensus_peers.cc:597] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.654376 11505 ts_tablet_manager.cc:1403] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.253s	user 0.025s	sys 0.001s
W20260709 13:06:21.660328 10686 consensus_peers.cc:597] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.660701 11505 raft_consensus.cc:359] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.665109 11505 raft_consensus.cc:740] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:21.666342 11505 consensus_queue.cc:260] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:21.668251 10834 consensus_peers.cc:597] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.670603 11505 ts_tablet_manager.cc:1434] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:06:21.672616 11505 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:21.678555 10686 consensus_peers.cc:597] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.683219 10717 raft_consensus.cc:1275] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.688181 11510 consensus_queue.cc:1048] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:21.700614 10791 raft_consensus.cc:1275] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.705256 11520 consensus_queue.cc:1048] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:21.694621 10539 catalog_manager.cc:5697] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:21.721159 10834 consensus_peers.cc:597] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.747319 10760 consensus_peers.cc:597] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.754385 10834 consensus_peers.cc:597] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.767701 10686 consensus_peers.cc:597] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.787861 11505 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:21.792093 11505 tablet_bootstrap.cc:492] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:21.792960 11505 ts_tablet_manager.cc:1403] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.121s	user 0.021s	sys 0.004s
I20260709 13:06:21.796334 11505 raft_consensus.cc:359] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:21.797238 11505 raft_consensus.cc:740] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:21.798399 11505 consensus_queue.cc:260] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:21.809267 11505 ts_tablet_manager.cc:1434] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.008s
I20260709 13:06:21.810410 11505 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:21.827764 10760 consensus_peers.cc:597] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.836087 10760 consensus_peers.cc:597] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.905193 10834 consensus_peers.cc:597] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.905843 10834 consensus_peers.cc:597] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.910713 10791 raft_consensus.cc:1275] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:21.913177 11520 consensus_queue.cc:1048] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:21.917905 10686 consensus_peers.cc:597] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:21.935010 10686 consensus_peers.cc:597] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:21.965807 11506 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:21.966470 11506 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.967512 11505 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:21.968626 11505 tablet_bootstrap.cc:492] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:21.974086 11506 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:21.969342 11505 ts_tablet_manager.cc:1403] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.159s	user 0.013s	sys 0.013s
I20260709 13:06:21.980248 11505 raft_consensus.cc:359] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.981141 11505 raft_consensus.cc:740] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:21.982031 11505 consensus_queue.cc:260] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:21.985792 11505 ts_tablet_manager.cc:1434] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:06:21.987345 11505 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:21.999985 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:22.000968 10713 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:22.003620 10835 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
W20260709 13:06:22.004944 10686 consensus_peers.cc:597] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:22.006263 11551 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:06:22.007045 11551 raft_consensus.cc:493] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:22.007531 11551 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:22.016167 11527 raft_consensus.cc:493] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:22.016911 11527 raft_consensus.cc:515] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:22.031874 10834 consensus_peers.cc:597] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:22.032753 10834 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:22.034713 11527 leader_election.cc:290] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
W20260709 13:06:22.092555 10834 consensus_peers.cc:597] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:22.092772 10686 consensus_peers.cc:597] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:22.093688 10686 consensus_peers.cc:597] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:22.126262 11551 raft_consensus.cc:515] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.130441 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:22.137699 10834 leader_election.cc:336] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:22.140913 10834 consensus_peers.cc:597] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:22.145969 10834 consensus_peers.cc:597] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:22.152381 11551 leader_election.cc:290] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 election: Requested vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:22.157994 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a375c777784941d4a0945680b7406ed1" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:22.160326 11499 leader_election.cc:304] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:22.161227 11527 raft_consensus.cc:2749] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:22.161778 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d50a168e65034266bc912284a133cb88" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:22.163164 10713 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:22.185281 11505 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:22.186657 11505 tablet_bootstrap.cc:492] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:22.187587 11505 ts_tablet_manager.cc:1403] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.201s	user 0.034s	sys 0.016s
I20260709 13:06:22.194353 11505 raft_consensus.cc:359] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:22.195139 11505 raft_consensus.cc:740] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
W20260709 13:06:22.195631 10834 consensus_peers.cc:597] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:22.195905 11505 consensus_queue.cc:260] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:22.198098 10713 raft_consensus.cc:2468] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 3.
I20260709 13:06:22.198531 11505 ts_tablet_manager.cc:1434] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.004s
I20260709 13:06:22.199527 11505 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:22.199783 10835 leader_election.cc:304] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:22.200829 11551 raft_consensus.cc:2804] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:22.202817 11551 raft_consensus.cc:697] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:22.203974 11551 consensus_queue.cc:237] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.230830 10542 catalog_manager.cc:5697] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 3, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 3 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:22.309418 11505 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:22.310581 11505 tablet_bootstrap.cc:492] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:22.311331 11505 ts_tablet_manager.cc:1403] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.112s	user 0.018s	sys 0.001s
I20260709 13:06:22.314486 11505 raft_consensus.cc:359] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:22.315234 11505 raft_consensus.cc:740] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:22.316253 11505 consensus_queue.cc:260] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:22.323137 11505 ts_tablet_manager.cc:1434] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:06:22.324280 11505 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:22.437760 11505 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:22.438897 11505 tablet_bootstrap.cc:492] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:22.439718 11505 ts_tablet_manager.cc:1403] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.116s	user 0.019s	sys 0.018s
I20260709 13:06:22.442507 11505 raft_consensus.cc:359] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.443267 11505 raft_consensus.cc:740] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:22.444370 11505 consensus_queue.cc:260] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.447541 11505 ts_tablet_manager.cc:1434] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:22.448396 11505 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:22.610117 11505 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:22.611526 11505 tablet_bootstrap.cc:492] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:22.612380 11505 ts_tablet_manager.cc:1403] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.164s	user 0.034s	sys 0.026s
I20260709 13:06:22.638553 11505 raft_consensus.cc:359] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:22.639593 11505 raft_consensus.cc:740] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:22.644845 11505 consensus_queue.cc:260] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:22.649436 11505 ts_tablet_manager.cc:1434] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.037s	user 0.009s	sys 0.003s
I20260709 13:06:22.650969 11505 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:22.668603 11580 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:22.669273 11580 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.678414 11580 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:22.680586 11505 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:22.681769 11505 tablet_bootstrap.cc:492] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:22.682493 11505 ts_tablet_manager.cc:1403] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.032s	user 0.013s	sys 0.013s
W20260709 13:06:22.683390 10686 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:22.687724 11505 raft_consensus.cc:359] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:22.689105 11505 raft_consensus.cc:740] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:22.690220 11505 consensus_queue.cc:260] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:22.695101 11505 ts_tablet_manager.cc:1434] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 13:06:22.696295 11505 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:22.714322 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:22.715359 10868 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:22.778036 10684 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:22.778838 10719 raft_consensus.cc:1275] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:22.790879 11614 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:06:22.791399 11614 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:22.791740 11614 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:22.792457 11527 raft_consensus.cc:493] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:22.793322 11527 raft_consensus.cc:515] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:22.799588 11527 leader_election.cc:290] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:22.804358 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:22.821336 11551 consensus_queue.cc:1048] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:22.844445 10834 consensus_peers.cc:597] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:22.872814 11614 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:22.919857 11614 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:22.957944 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c66cc57cc4ac4587a267846214205210" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:22.972945 11499 leader_election.cc:304] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
W20260709 13:06:22.970122 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 51102.7us
I20260709 13:06:22.974869 11527 raft_consensus.cc:2749] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:06:22.973634 10686 leader_election.cc:336] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:23.030045 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 55317.3us
I20260709 13:06:23.039855 11505 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.041450 11505 tablet_bootstrap.cc:492] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.046226 11505 ts_tablet_manager.cc:1403] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.350s	user 0.076s	sys 0.063s
I20260709 13:06:23.049299 11505 raft_consensus.cc:359] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.050087 11505 raft_consensus.cc:740] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.050959 11505 consensus_queue.cc:260] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.053207 11505 ts_tablet_manager.cc:1434] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:06:23.058411 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:23.059312 10862 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.066285 11505 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:23.090571 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 55789.2us
I20260709 13:06:23.117671 11527 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.118413 11527 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.122354 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.125223 11527 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:23.158402 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:23.160478 11502 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
W20260709 13:06:23.164317 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 47729.3us
I20260709 13:06:23.166567 11527 raft_consensus.cc:2749] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:06:23.177480 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 74023.2us
I20260709 13:06:23.194024 11527 raft_consensus.cc:493] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.194765 11527 raft_consensus.cc:515] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.198326 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.201702 10717 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80f861846a004804a0c1cca12f60514e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:23.202589 10717 raft_consensus.cc:2368] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 for earlier term 2. Current term is 3.
I20260709 13:06:23.204061 11502 leader_election.cc:400] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783) with higher term. Message: Invalid argument: T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 for earlier term 2. Current term is 3.
I20260709 13:06:23.204373 11502 leader_election.cc:403] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 13:06:23.213864 10862 raft_consensus.cc:2468] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 3.
I20260709 13:06:23.235566 11505 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.215564 11527 leader_election.cc:290] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:23.242794 11632 raft_consensus.cc:3060] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.243083 10684 leader_election.cc:304] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:23.247689 11534 raft_consensus.cc:2804] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:23.248219 11534 raft_consensus.cc:697] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:23.249419 11534 consensus_queue.cc:237] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.263103 11505 tablet_bootstrap.cc:492] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.264979 11505 ts_tablet_manager.cc:1403] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.199s	user 0.045s	sys 0.035s
I20260709 13:06:23.296304 11527 raft_consensus.cc:493] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.296895 11527 raft_consensus.cc:515] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.300311 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.302328 11527 leader_election.cc:290] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:23.296828 11505 raft_consensus.cc:359] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.305914 11505 raft_consensus.cc:740] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.310686 11632 raft_consensus.cc:2749] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783) with higher term. Message: Invalid argument: T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bf9fc3264f844470bc0ea04312ff2dd6 for earlier term 2. Current term is 3.
I20260709 13:06:23.318033 11505 consensus_queue.cc:260] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.321199 11505 ts_tablet_manager.cc:1434] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.053s	user 0.005s	sys 0.000s
I20260709 13:06:23.322731 11505 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:23.351622 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73ff90c30e40eaa01d6254d379aa3a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
W20260709 13:06:23.354202 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 69151.5us
I20260709 13:06:23.356724 11502 leader_election.cc:304] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:23.358192 11632 raft_consensus.cc:2749] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:23.391817 11505 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.393185 11505 tablet_bootstrap.cc:492] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.394032 11505 ts_tablet_manager.cc:1403] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.072s	user 0.024s	sys 0.005s
I20260709 13:06:23.397280 11505 raft_consensus.cc:359] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.398119 11505 raft_consensus.cc:740] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.398910 11505 consensus_queue.cc:260] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.402071 11505 ts_tablet_manager.cc:1434] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 13:06:23.403041 11505 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:23.429926 11505 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.431102 11505 tablet_bootstrap.cc:492] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.432030 11505 ts_tablet_manager.cc:1403] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.029s	user 0.022s	sys 0.000s
I20260709 13:06:23.434268 10542 catalog_manager.cc:5697] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 1 to 3, leader changed from bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193) to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 3 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:23.434898 11505 raft_consensus.cc:359] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.438133 11505 raft_consensus.cc:740] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.438925 11505 consensus_queue.cc:260] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.443478 11505 ts_tablet_manager.cc:1434] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:06:23.444419 11505 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:23.444656 10760 consensus_peers.cc:597] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:23.469416 11632 raft_consensus.cc:493] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.470476 11632 raft_consensus.cc:515] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.476678 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:23.480592 10866 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d43a3401e55d4d8687de1b5dad8edc84" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.484680 11527 raft_consensus.cc:493] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.485394 11527 raft_consensus.cc:515] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.488406 11527 leader_election.cc:290] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:23.491190 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7133813a08fe45908399f19d1e019ca6" is_pre_election: true
I20260709 13:06:23.472864 11632 leader_election.cc:290] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:23.498936 11499 leader_election.cc:304] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:23.506410 11632 raft_consensus.cc:2749] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:23.508939 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd8c76caaf4443508347a95b9085bf9a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.510340 11505 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.511433 11505 tablet_bootstrap.cc:492] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.512300 11505 ts_tablet_manager.cc:1403] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.068s	user 0.031s	sys 0.012s
I20260709 13:06:23.526194 11505 raft_consensus.cc:359] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.527256 11505 raft_consensus.cc:740] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.528224 11505 consensus_queue.cc:260] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.529135 11499 leader_election.cc:304] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:23.530339 11632 raft_consensus.cc:2749] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:23.531005 11505 ts_tablet_manager.cc:1434] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.001s
I20260709 13:06:23.532027 11505 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:23.540087 10834 consensus_peers.cc:597] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:23.545605 10686 consensus_peers.cc:597] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:23.567411   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:46307
I20260709 13:06:23.568135 11654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:46307 every 8 connection(s)
I20260709 13:06:23.581913 11505 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.585366 11505 tablet_bootstrap.cc:492] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.587097 11505 ts_tablet_manager.cc:1403] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.055s	user 0.028s	sys 0.003s
I20260709 13:06:23.610077 11505 raft_consensus.cc:359] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.610911 11505 raft_consensus.cc:740] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.614145 11505 consensus_queue.cc:260] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:23.629074 11505 ts_tablet_manager.cc:1434] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.041s	user 0.008s	sys 0.000s
I20260709 13:06:23.636099 11505 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:23.643693 11632 raft_consensus.cc:493] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:23.741287 10862 raft_consensus.cc:1275] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 13:06:23.730827 11632 raft_consensus.cc:515] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.775759 11632 leader_election.cc:290] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:23.794939 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "39f0ed0891e14f20bb25882450033da4" is_pre_election: true
I20260709 13:06:23.804288 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97d4e12c4db14c3397986584840b233b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:23.809207 11638 consensus_queue.cc:1048] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 13:06:23.865851 11604 raft_consensus.cc:3060] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:23.876607 10834 consensus_peers.cc:597] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:23.884541 11603 raft_consensus.cc:3060] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:23.896768 11598 raft_consensus.cc:1275] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 13:06:23.898244 10835 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 61582.5us
I20260709 13:06:23.906917 11638 consensus_queue.cc:1048] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:23.923131 11505 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:23.924472 11505 tablet_bootstrap.cc:492] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:23.925354 11505 ts_tablet_manager.cc:1403] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.295s	user 0.020s	sys 0.015s
I20260709 13:06:23.928720 11505 raft_consensus.cc:359] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:23.936532 11505 raft_consensus.cc:740] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:23.922787 11499 leader_election.cc:304] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bf9fc3264f844470bc0ea04312ff2dd6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:23.938246 11632 raft_consensus.cc:2749] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:23.939169 11505 consensus_queue.cc:260] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
W20260709 13:06:23.941365 10686 consensus_peers.cc:597] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:23.947654 10760 consensus_peers.cc:597] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:23.951898 10760 consensus_peers.cc:597] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:23.956678 11505 ts_tablet_manager.cc:1434] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.001s
I20260709 13:06:23.971534 11505 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:23.985173 11660 heartbeater.cc:344] Connected to a master server at 127.0.107.254:39861
I20260709 13:06:23.985819 11660 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:23.986966 11660 heartbeater.cc:507] Master 127.0.107.254:39861 requested a full tablet report, sending...
W20260709 13:06:24.016245 10760 consensus_peers.cc:597] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.016206 10686 consensus_peers.cc:597] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.018870 10760 consensus_peers.cc:597] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.046073 10760 consensus_peers.cc:597] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.069607 11505 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.073426 11505 tablet_bootstrap.cc:492] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.074309 11505 ts_tablet_manager.cc:1403] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.103s	user 0.018s	sys 0.005s
I20260709 13:06:24.077885 11505 raft_consensus.cc:359] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.078600 11505 raft_consensus.cc:740] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.079334 11505 consensus_queue.cc:260] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.081067 10542 ts_manager.cc:194] Re-registered known tserver with Master: bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:24.081645 11505 ts_tablet_manager.cc:1434] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:06:24.082549 11505 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.109898 11505 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.111099 11505 tablet_bootstrap.cc:492] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.111817 11505 ts_tablet_manager.cc:1403] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.005s
I20260709 13:06:24.115341 11505 raft_consensus.cc:359] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.116160 11505 raft_consensus.cc:740] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.117463 11505 consensus_queue.cc:260] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.120525 11505 ts_tablet_manager.cc:1434] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:24.121450 11505 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.136658 10542 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57976
I20260709 13:06:24.142318 11660 heartbeater.cc:499] Master 127.0.107.254:39861 was elected leader, sending a full tablet report...
I20260709 13:06:24.147854 11505 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.149022 11505 tablet_bootstrap.cc:492] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.149808 11505 ts_tablet_manager.cc:1403] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.029s	user 0.016s	sys 0.003s
I20260709 13:06:24.152596 11505 raft_consensus.cc:359] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.153354 11505 raft_consensus.cc:740] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.154079 11505 consensus_queue.cc:260] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.156402 11505 ts_tablet_manager.cc:1434] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:06:24.157439 11505 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.175154 10760 consensus_peers.cc:597] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.177343 11505 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.178563 11505 tablet_bootstrap.cc:492] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.179436 11505 ts_tablet_manager.cc:1403] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.003s
I20260709 13:06:24.182909 11505 raft_consensus.cc:359] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.183703 11505 raft_consensus.cc:740] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.184442 11505 consensus_queue.cc:260] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.192301 11505 ts_tablet_manager.cc:1434] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:06:24.193378 11505 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.224527 10760 consensus_peers.cc:597] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.236121 10834 consensus_peers.cc:597] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:24.237108 10834 consensus_peers.cc:597] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.247186 11505 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.259367 11505 tablet_bootstrap.cc:492] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.261660 11505 ts_tablet_manager.cc:1403] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.069s	user 0.018s	sys 0.002s
W20260709 13:06:24.262504 10760 consensus_peers.cc:597] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.265816 11505 raft_consensus.cc:359] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.266656 11505 raft_consensus.cc:740] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.267616 11505 consensus_queue.cc:260] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.269853 11505 ts_tablet_manager.cc:1434] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:24.270689 11505 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.309930 10686 consensus_peers.cc:597] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.307194 11505 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.311542 11505 tablet_bootstrap.cc:492] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.312325 11505 ts_tablet_manager.cc:1403] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.042s	user 0.016s	sys 0.005s
I20260709 13:06:24.318569 11505 raft_consensus.cc:359] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.319391 11505 raft_consensus.cc:740] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.320488 11505 consensus_queue.cc:260] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.357077 11505 ts_tablet_manager.cc:1434] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.000s
I20260709 13:06:24.365309 11505 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.395432 11505 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.396512 11505 tablet_bootstrap.cc:492] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.397257 11505 ts_tablet_manager.cc:1403] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.032s	user 0.015s	sys 0.006s
I20260709 13:06:24.400314 11505 raft_consensus.cc:359] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.401479 11505 raft_consensus.cc:740] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.402467 11505 consensus_queue.cc:260] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.405473 11505 ts_tablet_manager.cc:1434] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:06:24.406694 11505 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.432040 10760 consensus_peers.cc:597] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:24.435101 11505 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.436563 11505 tablet_bootstrap.cc:492] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.437283 11505 ts_tablet_manager.cc:1403] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.031s	user 0.012s	sys 0.010s
W20260709 13:06:24.437362 10686 consensus_peers.cc:597] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.440162 11505 raft_consensus.cc:359] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.440950 11505 raft_consensus.cc:740] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.441893 11505 consensus_queue.cc:260] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.444415 11505 ts_tablet_manager.cc:1434] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:06:24.445333 11505 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.530200 10834 consensus_peers.cc:597] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.587649 10686 consensus_peers.cc:597] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.588820 10834 consensus_peers.cc:597] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:24.590060 10834 consensus_peers.cc:597] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 13:06:24.598078 11505 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.603835 11505 tablet_bootstrap.cc:492] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.604905 11505 ts_tablet_manager.cc:1403] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.160s	user 0.071s	sys 0.075s
I20260709 13:06:24.607931 11505 raft_consensus.cc:359] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.608953 11505 raft_consensus.cc:740] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.609755 11505 consensus_queue.cc:260] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.617760 11505 ts_tablet_manager.cc:1434] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.003s
I20260709 13:06:24.618808 11505 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.688284 11603 raft_consensus.cc:3060] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:24.709062 11505 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.710414 11505 tablet_bootstrap.cc:492] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.711320 11505 ts_tablet_manager.cc:1403] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.093s	user 0.028s	sys 0.042s
I20260709 13:06:24.714604 11505 raft_consensus.cc:359] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.718695 11505 raft_consensus.cc:740] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.720265 11505 consensus_queue.cc:260] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.723568 11505 ts_tablet_manager.cc:1434] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:24.724759 11505 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.748154 11505 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.749267 11505 tablet_bootstrap.cc:492] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.749935 11505 ts_tablet_manager.cc:1403] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.025s	user 0.022s	sys 0.000s
I20260709 13:06:24.752605 11505 raft_consensus.cc:359] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.753324 11505 raft_consensus.cc:740] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.754132 11505 consensus_queue.cc:260] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:24.756313 11505 ts_tablet_manager.cc:1434] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:06:24.757189 11505 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.780797 11505 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.782107 11505 tablet_bootstrap.cc:492] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.784315 11505 ts_tablet_manager.cc:1403] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.027s	user 0.021s	sys 0.003s
I20260709 13:06:24.787865 11505 raft_consensus.cc:359] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.789069 11505 raft_consensus.cc:740] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.790278 11505 consensus_queue.cc:260] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.792987 11505 ts_tablet_manager.cc:1434] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:24.794034 11505 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.830826 11505 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.832199 11505 tablet_bootstrap.cc:492] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.833026 11505 ts_tablet_manager.cc:1403] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.039s	user 0.016s	sys 0.017s
I20260709 13:06:24.835956 11505 raft_consensus.cc:359] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.836758 11505 raft_consensus.cc:740] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.837729 11505 consensus_queue.cc:260] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.840847 11505 ts_tablet_manager.cc:1434] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 13:06:24.842239 11505 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
W20260709 13:06:24.865572 10686 consensus_peers.cc:597] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:24.872640 11505 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.874094 11505 tablet_bootstrap.cc:492] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.874883 11505 ts_tablet_manager.cc:1403] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.009s
I20260709 13:06:24.877719 11505 raft_consensus.cc:359] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.878511 11505 raft_consensus.cc:740] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.879252 11505 consensus_queue.cc:260] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:24.881215 11505 ts_tablet_manager.cc:1434] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:06:24.883064 11505 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.907819 11505 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.908686 11505 tablet_bootstrap.cc:492] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.909420 11505 ts_tablet_manager.cc:1403] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.008s
I20260709 13:06:24.912747 11505 raft_consensus.cc:359] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.913594 11505 raft_consensus.cc:740] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.914551 11505 consensus_queue.cc:260] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.916987 11505 ts_tablet_manager.cc:1434] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:06:24.922935 11505 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:24.947594 11505 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:24.957844 11603 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:24.967852 11505 tablet_bootstrap.cc:492] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:24.979159 11505 ts_tablet_manager.cc:1403] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.056s	user 0.016s	sys 0.011s
I20260709 13:06:24.982450 11505 raft_consensus.cc:359] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.983219 11505 raft_consensus.cc:740] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:24.985380 11505 consensus_queue.cc:260] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:24.988082 11505 ts_tablet_manager.cc:1434] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:06:24.995028 11604 raft_consensus.cc:3060] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.047849 11505 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.100447 11505 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.101425 11505 tablet_bootstrap.cc:492] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.102240 11505 ts_tablet_manager.cc:1403] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.055s	user 0.019s	sys 0.009s
I20260709 13:06:25.106104 11505 raft_consensus.cc:359] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:25.106899 11505 raft_consensus.cc:740] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.107662 11505 consensus_queue.cc:260] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:25.110129 11505 ts_tablet_manager.cc:1434] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 13:06:25.110929 11505 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.130232 11505 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.132105 11505 tablet_bootstrap.cc:492] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.132787 11505 ts_tablet_manager.cc:1403] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.004s
I20260709 13:06:25.135353 11505 raft_consensus.cc:359] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.136751 11505 raft_consensus.cc:740] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.137815 11505 consensus_queue.cc:260] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.140617 11505 ts_tablet_manager.cc:1434] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:25.141865 11505 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.166311 11505 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.167122 11505 tablet_bootstrap.cc:492] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.167732 11505 ts_tablet_manager.cc:1403] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.026s	user 0.022s	sys 0.000s
I20260709 13:06:25.169889 11505 raft_consensus.cc:359] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.170410 11505 raft_consensus.cc:740] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.170961 11505 consensus_queue.cc:260] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.172783 11505 ts_tablet_manager.cc:1434] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:25.173810 11505 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.197152 11505 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.198325 11505 tablet_bootstrap.cc:492] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.199064 11505 ts_tablet_manager.cc:1403] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.025s	user 0.020s	sys 0.003s
I20260709 13:06:25.201609 11505 raft_consensus.cc:359] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.202319 11505 raft_consensus.cc:740] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.203158 11505 consensus_queue.cc:260] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.206367 11505 ts_tablet_manager.cc:1434] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 13:06:25.207541 11505 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.240672 11505 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.242708 11505 tablet_bootstrap.cc:492] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.243610 11505 ts_tablet_manager.cc:1403] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.005s
I20260709 13:06:25.246980 11505 raft_consensus.cc:359] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.247884 11505 raft_consensus.cc:740] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.248646 11505 consensus_queue.cc:260] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.251128 11505 ts_tablet_manager.cc:1434] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 13:06:25.252624 11505 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.299523 11600 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.304926 11505 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.307934 11505 tablet_bootstrap.cc:492] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.308689 11505 ts_tablet_manager.cc:1403] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.056s	user 0.028s	sys 0.010s
I20260709 13:06:25.311825 11505 raft_consensus.cc:359] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.312608 11505 raft_consensus.cc:740] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.313344 11505 consensus_queue.cc:260] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.316931 11505 ts_tablet_manager.cc:1434] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:25.317857 11505 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.358528 11603 raft_consensus.cc:3060] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.358670 11505 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.360033 11505 tablet_bootstrap.cc:492] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.360971 11505 ts_tablet_manager.cc:1403] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.043s	user 0.010s	sys 0.010s
I20260709 13:06:25.364224 11505 raft_consensus.cc:359] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.365200 11505 raft_consensus.cc:740] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.366307 11505 consensus_queue.cc:260] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.369863 11505 ts_tablet_manager.cc:1434] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260709 13:06:25.370903 11505 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.413029 11505 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.414281 11505 tablet_bootstrap.cc:492] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.415314 11505 ts_tablet_manager.cc:1403] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.045s	user 0.027s	sys 0.012s
I20260709 13:06:25.418433 11505 raft_consensus.cc:359] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.419287 11505 raft_consensus.cc:740] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.420045 11505 consensus_queue.cc:260] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:25.422196 11505 ts_tablet_manager.cc:1434] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 13:06:25.423105 11505 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.465951 11505 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.473767 11505 tablet_bootstrap.cc:492] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.475864 11505 ts_tablet_manager.cc:1403] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.053s	user 0.013s	sys 0.024s
I20260709 13:06:25.479220 11505 raft_consensus.cc:359] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.480041 11505 raft_consensus.cc:740] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.481283 11505 consensus_queue.cc:260] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.484122 11505 ts_tablet_manager.cc:1434] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:06:25.496985 11505 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.531388 11505 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.533020 11505 tablet_bootstrap.cc:492] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.534111 11505 ts_tablet_manager.cc:1403] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.037s	user 0.025s	sys 0.003s
I20260709 13:06:25.537201 11505 raft_consensus.cc:359] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.538106 11505 raft_consensus.cc:740] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.539062 11505 consensus_queue.cc:260] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:25.542073 11505 ts_tablet_manager.cc:1434] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:06:25.543090 11505 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap starting.
I20260709 13:06:25.597740 11505 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:25.598987 11505 tablet_bootstrap.cc:492] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Bootstrap complete.
I20260709 13:06:25.599833 11505 ts_tablet_manager.cc:1403] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent bootstrapping tablet: real 0.057s	user 0.028s	sys 0.020s
I20260709 13:06:25.620850 11505 raft_consensus.cc:359] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:25.622813 11505 raft_consensus.cc:740] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Initialized, Role: FOLLOWER
I20260709 13:06:25.625883 11505 consensus_queue.cc:260] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:25.629096 11505 ts_tablet_manager.cc:1434] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.001s
I20260709 13:06:25.687981 11603 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:25.688023 11606 raft_consensus.cc:3060] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:27.806033 10761 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:45783, user_credentials={real_user=slave}} blocked reactor thread for 44988.3us
I20260709 13:06:33.842545   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:33.909837 10862 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "017730403fb44573ae22679b6d9d461f"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:33.910625 10862 raft_consensus.cc:606] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.914448 10862 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3077ac317c854af19123e5a37389d01b"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:33.915130 10862 raft_consensus.cc:606] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.939308 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:33.939908 10714 raft_consensus.cc:606] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.942533 10862 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:33.943003 10862 raft_consensus.cc:606] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.945576 10862 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:33.946033 10862 raft_consensus.cc:606] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.948380 10862 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:33.948827 10862 raft_consensus.cc:606] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.963137 10791 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fcce4acfbd7240178278200df749235a"
dest_uuid: "7133813a08fe45908399f19d1e019ca6"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:50768
I20260709 13:06:33.963711 10791 raft_consensus.cc:606] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:33.963797 11746 raft_consensus.cc:993] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:33.964553 11751 raft_consensus.cc:1081] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:33.965015   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 13:06:33.966504 11602 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "017730403fb44573ae22679b6d9d461f"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44840
I20260709 13:06:33.966884   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:33.967068 11602 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:33.967553 11602 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.975445 11602 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:33.977586 11602 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:33.978484 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:33.979076 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "017730403fb44573ae22679b6d9d461f" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:33.979573 10862 raft_consensus.cc:3055] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:33.979715 10714 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.979947 10862 raft_consensus.cc:740] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:33.981009 10862 consensus_queue.cc:260] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:33.982579 10862 raft_consensus.cc:3060] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:33.988672 10714 raft_consensus.cc:2468] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:33.990957 11502 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:33.991780 10862 raft_consensus.cc:2468] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:33.993273 11758 raft_consensus.cc:2804] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:33.994632 11758 raft_consensus.cc:697] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:33.995729 11758 consensus_queue.cc:237] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.013456 11746 raft_consensus.cc:993] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.014348 11751 raft_consensus.cc:1081] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.016574 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44840
I20260709 13:06:34.017233 11606 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.017668 11606 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.018723 10544 catalog_manager.cc:5697] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.028753 11606 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:34.031355 11606 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:34.033162 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:34.033288 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:34.034265 10791 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.033775 10862 raft_consensus.cc:3055] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.037134 10862 raft_consensus.cc:740] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:34.038311 10862 consensus_queue.cc:260] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:34.039917 10862 raft_consensus.cc:3060] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.053424 11745 raft_consensus.cc:993] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.053340 10791 raft_consensus.cc:2468] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.054384 11755 raft_consensus.cc:1081] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.055083 11499 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.055735 10862 raft_consensus.cc:2468] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.056033 11758 raft_consensus.cc:2804] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.056548 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fcce4acfbd7240178278200df749235a"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44832
I20260709 13:06:34.056581 11758 raft_consensus.cc:697] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.057242 11606 raft_consensus.cc:493] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.057698 11606 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.058022 11758 consensus_queue.cc:237] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:34.064577 11606 raft_consensus.cc:515] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:34.066869 11606 leader_election.cc:290] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:34.067148 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:34.067533 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fcce4acfbd7240178278200df749235a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:34.067778 10714 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.068266 10791 raft_consensus.cc:3055] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.068578 10791 raft_consensus.cc:740] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:34.069439 10791 consensus_queue.cc:260] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:34.069197 10544 catalog_manager.cc:5697] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:34.070654 10791 raft_consensus.cc:3060] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.075098 10714 raft_consensus.cc:2468] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.076499 11502 leader_election.cc:304] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.077286 11758 raft_consensus.cc:2804] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.078001 11758 raft_consensus.cc:697] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.078879 11758 consensus_queue.cc:237] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:34.082859 10791 raft_consensus.cc:2468] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.095077 10544 catalog_manager.cc:5697] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 7133813a08fe45908399f19d1e019ca6 (127.0.107.195) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.141649 11751 raft_consensus.cc:993] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.142216 11746 raft_consensus.cc:1081] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.143901 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44840
I20260709 13:06:34.144413 11606 raft_consensus.cc:493] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.144662 11606 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.153875 11606 raft_consensus.cc:515] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.156280 11606 leader_election.cc:290] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:34.156721 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:34.156965 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3f299da07404b66be3f547c4629f7b6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:34.157727 10862 raft_consensus.cc:3055] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.158125 10862 raft_consensus.cc:740] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:34.158006 10714 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.158987 10862 consensus_queue.cc:260] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.160409 10862 raft_consensus.cc:3060] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.166870 10714 raft_consensus.cc:2468] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.167068 10862 raft_consensus.cc:2468] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.168093 11499 leader_election.cc:304] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.169049 11758 raft_consensus.cc:2804] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.169553 11758 raft_consensus.cc:697] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.170607 11758 consensus_queue.cc:237] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.186895 10544 catalog_manager.cc:5697] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.212229 11761 raft_consensus.cc:993] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.213142 11761 raft_consensus.cc:1081] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.216050 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3077ac317c854af19123e5a37389d01b"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44840
I20260709 13:06:34.217181 11606 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.217732 11606 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.229885 11606 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.231987 11606 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:34.232519 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:34.233685 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3077ac317c854af19123e5a37389d01b" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:34.234830 10714 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.233280 10862 raft_consensus.cc:3055] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:34.235692 10862 raft_consensus.cc:740] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:34.236851 10862 consensus_queue.cc:260] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.238533 10862 raft_consensus.cc:3060] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.243888 10714 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:34.245117 11502 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.245997 11758 raft_consensus.cc:2804] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:34.246491 11758 raft_consensus.cc:697] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.246946 10862 raft_consensus.cc:2468] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:34.247493 11758 consensus_queue.cc:237] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.259631 10544 catalog_manager.cc:5697] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 2 to 3, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.391763 11727 raft_consensus.cc:993] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.392729 11752 raft_consensus.cc:1081] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.394937 11603 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:34.395500 11603 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.395912 11603 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.404066 11603 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.406571 11603 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:34.407895 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:34.408679 10714 raft_consensus.cc:3055] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:34.409024 10714 raft_consensus.cc:740] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:34.410895 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cbb2820ebd1414e98070a8a401c92a0" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:34.411664 10868 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.410688 10714 consensus_queue.cc:260] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.413286 10714 raft_consensus.cc:3060] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:34.423132 10714 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:34.425329 11502 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.427358 11758 raft_consensus.cc:2804] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:34.428068 11758 raft_consensus.cc:697] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.429263 11758 consensus_queue.cc:237] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.430687 10868 raft_consensus.cc:2468] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:34.447113 11769 raft_consensus.cc:993] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.450604 11751 raft_consensus.cc:1081] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:34.449638 10868 raft_consensus.cc:1275] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.453238 11603 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44840
I20260709 13:06:34.452458 10539 catalog_manager.cc:5697] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 2 to 3, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:34.453891 11603 raft_consensus.cc:493] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:34.453744 11758 consensus_queue.cc:1048] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.454401 11603 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.466594 11603 raft_consensus.cc:515] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.473042 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:34.473814 10714 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.479651 10862 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d744005e4a084fd29059cb6140b6d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:34.480728 10862 raft_consensus.cc:3055] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:34.481413 10862 raft_consensus.cc:740] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:34.485410 10862 consensus_queue.cc:260] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.486402 10714 raft_consensus.cc:2468] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.487905 11502 leader_election.cc:304] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:34.489861 11759 raft_consensus.cc:2804] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:34.494204 10719 raft_consensus.cc:1275] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.487543 10862 raft_consensus.cc:3060] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:34.500691 11759 raft_consensus.cc:697] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:34.501235 11774 consensus_queue.cc:1048] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.502064 11759 consensus_queue.cc:237] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:34.504778 11603 leader_election.cc:290] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:34.528787 10862 raft_consensus.cc:2468] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:34.539131 10791 raft_consensus.cc:1275] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.546295 11774 consensus_queue.cc:1048] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.578094 10862 raft_consensus.cc:1275] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.616681 11781 consensus_queue.cc:1048] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 13:06:34.632493 10719 raft_consensus.cc:1275] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:34.634783 11783 consensus_queue.cc:1048] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:34.638929 10794 raft_consensus.cc:1275] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.640687 11783 consensus_queue.cc:1048] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.661572 10539 catalog_manager.cc:5697] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:34.693296 10686 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 47050.6us
I20260709 13:06:34.708750 10719 raft_consensus.cc:1275] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.709798 10868 raft_consensus.cc:1275] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:34.711582 10862 raft_consensus.cc:1275] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.714361 11777 consensus_queue.cc:1048] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.716557 11777 consensus_queue.cc:1048] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:34.728235 11777 consensus_queue.cc:1048] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.731165 10719 raft_consensus.cc:1275] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.733296 11783 consensus_queue.cc:1048] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:34.905786 10868 raft_consensus.cc:1275] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:34.907253 11783 consensus_queue.cc:1048] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:34.930608 10714 raft_consensus.cc:1275] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:34.932399 11783 consensus_queue.cc:1048] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:34.941612 10862 raft_consensus.cc:1275] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:34.943343 11783 consensus_queue.cc:1048] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:34.968502   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:34.989217 10719 raft_consensus.cc:1275] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.003150 11783 consensus_queue.cc:1048] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:35.052430   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:35.101133 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "193ca6746d1344849958720afeb79325"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:35.101908 10714 raft_consensus.cc:606] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:35.105345 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:35.106221 10714 raft_consensus.cc:606] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:35.109689 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "410064a793614346a815b98b7c6d5316"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:35.110558 10714 raft_consensus.cc:606] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:35.113842 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a32195885f27470e85f07731d06d85d6"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:35.114511 10714 raft_consensus.cc:606] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:35.117455 10791 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56"
dest_uuid: "7133813a08fe45908399f19d1e019ca6"
mode: GRACEFUL
new_leader_uuid: "39f0ed0891e14f20bb25882450033da4"
 from {username='slave'} at 127.0.0.1:50768
I20260709 13:06:35.118041 10791 raft_consensus.cc:606] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Received request to transfer leadership to TS 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:35.121049 11752 raft_consensus.cc:993] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.121639 11790 raft_consensus.cc:1081] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.121768 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:35.122431 10714 raft_consensus.cc:606] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:35.123977 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "193ca6746d1344849958720afeb79325"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:35.124661 11606 raft_consensus.cc:493] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.125061 11606 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.125732   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 13:06:35.127489   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:35.133402 11606 raft_consensus.cc:515] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.135430 11606 leader_election.cc:290] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:35.136633 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:35.137346 10714 raft_consensus.cc:3055] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.137722 10714 raft_consensus.cc:740] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.138006 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "193ca6746d1344849958720afeb79325" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:35.138666 10791 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.138655 10714 consensus_queue.cc:260] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.140031 10714 raft_consensus.cc:3060] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.149860 10791 raft_consensus.cc:2468] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.151015 10714 raft_consensus.cc:2468] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.152221 11499 leader_election.cc:304] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:35.153609 11759 raft_consensus.cc:2804] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.154178 11759 raft_consensus.cc:697] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:35.155300 11759 consensus_queue.cc:237] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.170576 10543 catalog_manager.cc:5697] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:35.172942 11752 raft_consensus.cc:993] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.173671 11790 raft_consensus.cc:1081] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.175335 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:35.177444 11606 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.177887 11606 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.187179 11606 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.191468 11606 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:35.195252 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:35.196041 10719 raft_consensus.cc:3055] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.196344 10719 raft_consensus.cc:740] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.197044 10719 consensus_queue.cc:260] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.197943 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3f27b1950c54af2b8d48d3e4c7cb7e9" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:35.198585 10868 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.198624 10719 raft_consensus.cc:3060] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.207804 10868 raft_consensus.cc:2468] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.206878 10719 raft_consensus.cc:2468] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.209779 11502 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:35.211122 11759 raft_consensus.cc:2804] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.211686 11759 raft_consensus.cc:697] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:35.212673 11759 consensus_queue.cc:237] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.242372 10543 catalog_manager.cc:5697] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:35.335502 11790 raft_consensus.cc:993] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.336139 11790 raft_consensus.cc:1081] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.338438 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a32195885f27470e85f07731d06d85d6"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:35.339093 11606 raft_consensus.cc:493] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.339514 11606 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.347437 11606 raft_consensus.cc:515] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.349220 11606 leader_election.cc:290] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:35.350164 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:35.350268 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a32195885f27470e85f07731d06d85d6" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:35.352473 10719 raft_consensus.cc:3055] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.352922 10719 raft_consensus.cc:740] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.353947 10719 consensus_queue.cc:260] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.355368 10719 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.351094 10868 raft_consensus.cc:3060] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.361817 10719 raft_consensus.cc:2468] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.363034 11502 leader_election.cc:304] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:35.363850 11759 raft_consensus.cc:2804] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.363885 10868 raft_consensus.cc:2468] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.364478 11759 raft_consensus.cc:697] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:35.366151 11759 consensus_queue.cc:237] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:35.377831 10543 catalog_manager.cc:5697] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:35.384989 11791 raft_consensus.cc:993] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower 148a4048db5a444c9daaa8cbc7b7e9d7 to start an election
I20260709 13:06:35.385691 11752 raft_consensus.cc:1081] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer 148a4048db5a444c9daaa8cbc7b7e9d7 to start an election
I20260709 13:06:35.389777 10868 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "410064a793614346a815b98b7c6d5316"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
 from {username='slave'} at 127.0.0.1:50298
I20260709 13:06:35.390404 10868 raft_consensus.cc:493] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.390803 10868 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.397589 10868 raft_consensus.cc:515] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.400959 10868 leader_election.cc:290] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:35.401687 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:35.402494 10714 raft_consensus.cc:3055] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.402796 10714 raft_consensus.cc:740] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.403574 10714 consensus_queue.cc:260] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.403970 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "410064a793614346a815b98b7c6d5316" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:35.404699 10714 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.404858 10794 raft_consensus.cc:3060] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.412946 10714 raft_consensus.cc:2468] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:35.414115 10794 raft_consensus.cc:2468] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:35.414489 10835 leader_election.cc:304] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:35.415594 11751 raft_consensus.cc:2804] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.416131 11751 raft_consensus.cc:697] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:35.417086 11751 consensus_queue.cc:237] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.434921 10543 catalog_manager.cc:5697] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:35.546330 11745 raft_consensus.cc:993] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: : Instructing follower 39f0ed0891e14f20bb25882450033da4 to start an election
I20260709 13:06:35.554884 10714 raft_consensus.cc:1275] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.556967 11745 raft_consensus.cc:1081] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Signalling peer 39f0ed0891e14f20bb25882450033da4 to start an election
I20260709 13:06:35.559389 10719 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
 from {username='slave'} at 127.0.0.1:47348
I20260709 13:06:35.560170 10719 raft_consensus.cc:493] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.560613 10719 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.584782 11791 raft_consensus.cc:993] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.585793 11791 raft_consensus.cc:1081] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:35.589069 11603 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:35.589990 11603 raft_consensus.cc:493] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:35.590446 11603 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.597471 11781 consensus_queue.cc:1048] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:35.598958 10794 raft_consensus.cc:1275] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.601033 11783 consensus_queue.cc:1048] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:06:35.605376 10760 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 58174.8us
I20260709 13:06:35.606881 11603 raft_consensus.cc:515] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.610778 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:35.611622 10714 raft_consensus.cc:3055] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.612021 10714 raft_consensus.cc:740] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.613268 10714 consensus_queue.cc:260] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.597309 10719 raft_consensus.cc:515] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.615192 10714 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.620960 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25f5b1ed27b34d46b55c0e684d98d935" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:35.622869 10791 raft_consensus.cc:3060] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.623220 11603 leader_election.cc:290] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:35.633589 10714 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.635555 11502 leader_election.cc:304] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:35.636981 11783 raft_consensus.cc:2804] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.654735 11783 raft_consensus.cc:697] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:35.656917 11783 consensus_queue.cc:237] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:35.660301 10794 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:35.660926 10791 raft_consensus.cc:2468] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:35.661032 10794 raft_consensus.cc:3055] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:35.661583 10794 raft_consensus.cc:740] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:35.662237 10719 leader_election.cc:290] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:35.663924 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca1d8934020d46ad9b3a2a19c509cf56" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:35.664518 10868 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.662413 10794 consensus_queue.cc:260] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.671859 10794 raft_consensus.cc:3060] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:35.691330 10868 raft_consensus.cc:2468] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:35.698925 10684 leader_election.cc:304] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 39f0ed0891e14f20bb25882450033da4; no voters: 
I20260709 13:06:35.701130 11791 raft_consensus.cc:2804] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:35.701766 11791 raft_consensus.cc:697] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:35.703459 11791 consensus_queue.cc:237] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:35.709630 10794 raft_consensus.cc:2468] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:35.706240 10543 catalog_manager.cc:5697] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:35.727766 10544 catalog_manager.cc:5697] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 1 to 2, leader changed from 7133813a08fe45908399f19d1e019ca6 (127.0.107.195) to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:35.777587 10719 raft_consensus.cc:1275] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.779340 11811 consensus_queue.cc:1048] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:35.794297 10862 raft_consensus.cc:1275] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.794486 10868 raft_consensus.cc:1275] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.797760 11783 consensus_queue.cc:1048] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:35.798233 11811 consensus_queue.cc:1048] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:35.833853 10714 raft_consensus.cc:1275] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.835772 11751 consensus_queue.cc:1048] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:35.843097 10719 raft_consensus.cc:1275] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.845650 11759 consensus_queue.cc:1048] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:35.854297 10791 raft_consensus.cc:1275] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:35.856029 11810 consensus_queue.cc:1048] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:36.128600   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:36.146481 10713 raft_consensus.cc:1275] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.149021 11783 consensus_queue.cc:1048] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:36.172106 10791 raft_consensus.cc:1275] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.173733 11783 consensus_queue.cc:1048] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:36.182982 10864 raft_consensus.cc:1275] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.184603 11791 consensus_queue.cc:1048] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:36.187621   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:36.264324 10794 raft_consensus.cc:1275] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.266101 10865 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "39f0ed0891e14f20bb25882450033da4"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:36.266912 10865 raft_consensus.cc:606] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS 39f0ed0891e14f20bb25882450033da4
I20260709 13:06:36.269482 11790 consensus_queue.cc:1048] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:36.273166 10794 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e"
dest_uuid: "7133813a08fe45908399f19d1e019ca6"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:50768
I20260709 13:06:36.274040 10794 raft_consensus.cc:606] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:36.282058 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d52e964d865840ee885c5cb005982d82"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:36.282794 10714 raft_consensus.cc:606] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS 148a4048db5a444c9daaa8cbc7b7e9d7
I20260709 13:06:36.285414   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:06:36.289376   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:36.372185 11828 raft_consensus.cc:993] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower 148a4048db5a444c9daaa8cbc7b7e9d7 to start an election
I20260709 13:06:36.372753 11828 raft_consensus.cc:1081] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer 148a4048db5a444c9daaa8cbc7b7e9d7 to start an election
I20260709 13:06:36.374826 10864 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d52e964d865840ee885c5cb005982d82"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
 from {username='slave'} at 127.0.0.1:50298
I20260709 13:06:36.375430 10864 raft_consensus.cc:493] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:36.375752 10864 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.384070 10864 raft_consensus.cc:515] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.386260 10864 leader_election.cc:290] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:36.387164 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:36.387357 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d52e964d865840ee885c5cb005982d82" candidate_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:36.388006 10791 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.388006 10714 raft_consensus.cc:3055] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:36.388612 10714 raft_consensus.cc:740] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:36.389891 10714 consensus_queue.cc:260] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.391605 10714 raft_consensus.cc:3060] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.397373 10791 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:36.399322 10832 leader_election.cc:304] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:36.400689 11809 raft_consensus.cc:2804] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:36.401623 11809 raft_consensus.cc:697] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Becoming Leader. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:36.403141 11809 consensus_queue.cc:237] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.403759 10714 raft_consensus.cc:2468] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 148a4048db5a444c9daaa8cbc7b7e9d7 in term 2.
I20260709 13:06:36.416913 10544 catalog_manager.cc:5697] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196). New cstate: current_term: 2 leader_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:36.494550 11806 raft_consensus.cc:993] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:36.495105 11805 raft_consensus.cc:1081] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:36.496611 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44832
I20260709 13:06:36.497097 11606 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:36.497362 11606 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:36.503829 11606 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:36.505566 11606 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001)
I20260709 13:06:36.506283 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:36.506739 10791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96da319bd6aa4afb90b067caaedacf3e" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:36.507027 10713 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:36.507314 10791 raft_consensus.cc:3055] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:36.507725 10791 raft_consensus.cc:740] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:36.508471 10791 consensus_queue.cc:260] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:36.509716 10791 raft_consensus.cc:3060] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:36.513788 10713 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:36.515151 11502 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:36.516043 11781 raft_consensus.cc:2804] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:36.516516 11781 raft_consensus.cc:697] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:36.517352 11781 consensus_queue.cc:237] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:36.518474 10791 raft_consensus.cc:2468] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 3.
I20260709 13:06:36.530526 10544 catalog_manager.cc:5697] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 2 to 3, leader changed from 7133813a08fe45908399f19d1e019ca6 (127.0.107.195) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 3 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:36.690660 11809 raft_consensus.cc:993] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower 39f0ed0891e14f20bb25882450033da4 to start an election
I20260709 13:06:36.691567 11810 raft_consensus.cc:1081] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer 39f0ed0891e14f20bb25882450033da4 to start an election
I20260709 13:06:36.695188 10713 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
 from {username='slave'} at 127.0.0.1:47342
I20260709 13:06:36.695981 10713 raft_consensus.cc:493] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:36.696475 10713 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.707206 10713 raft_consensus.cc:515] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.709052 10713 leader_election.cc:290] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:36.710644 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:36.710637 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76a046317986428fbc51a3609a2bf91e" candidate_uuid: "39f0ed0891e14f20bb25882450033da4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:36.711521 10864 raft_consensus.cc:3055] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:36.711628 10795 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.711846 10864 raft_consensus.cc:740] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:36.712863 10864 consensus_queue.cc:260] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.714570 10864 raft_consensus.cc:3060] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:36.724325 10795 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:36.725992 10684 leader_election.cc:304] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:36.726440 10864 raft_consensus.cc:2468] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39f0ed0891e14f20bb25882450033da4 in term 2.
I20260709 13:06:36.727056 11790 raft_consensus.cc:2804] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:36.727749 11790 raft_consensus.cc:697] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Becoming Leader. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:36.728705 11790 consensus_queue.cc:237] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:36.740816 10544 catalog_manager.cc:5697] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to 39f0ed0891e14f20bb25882450033da4 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "39f0ed0891e14f20bb25882450033da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:36.797354 10714 raft_consensus.cc:1275] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.799034 11810 consensus_queue.cc:1048] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:36.834667 10795 raft_consensus.cc:1275] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 148a4048db5a444c9daaa8cbc7b7e9d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:36.837137 11769 consensus_queue.cc:1048] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:06:37.030768 10794 raft_consensus.cc:1275] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:37.032282 11783 consensus_queue.cc:1048] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:37.067262 10713 raft_consensus.cc:1275] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:37.072463 11781 consensus_queue.cc:1048] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 13:06:37.120015 10865 raft_consensus.cc:1275] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:37.121623 11827 consensus_queue.cc:1048] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:37.144178 10794 raft_consensus.cc:1275] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer 39f0ed0891e14f20bb25882450033da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:37.147009 11790 consensus_queue.cc:1048] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:37.291579   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:37.382032   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:37.444840 10714 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806"
dest_uuid: "39f0ed0891e14f20bb25882450033da4"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:41844
I20260709 13:06:37.445559 10714 raft_consensus.cc:606] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:37.450233 10864 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc"
dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
mode: GRACEFUL
new_leader_uuid: "7133813a08fe45908399f19d1e019ca6"
 from {username='slave'} at 127.0.0.1:46230
I20260709 13:06:37.451179 10864 raft_consensus.cc:606] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Received request to transfer leadership to TS 7133813a08fe45908399f19d1e019ca6
I20260709 13:06:37.452648   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:37.454521   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:37.565649 11827 raft_consensus.cc:993] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:37.566433 11827 raft_consensus.cc:1081] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:37.580787 11603 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44852
I20260709 13:06:37.581712 11603 raft_consensus.cc:493] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:37.582554 11603 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.603461 11603 raft_consensus.cc:515] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:37.606266 11603 leader_election.cc:290] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:37.609943 10714 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:37.611198 10714 raft_consensus.cc:3055] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:37.611542 10714 raft_consensus.cc:740] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39f0ed0891e14f20bb25882450033da4, State: Running, Role: LEADER
I20260709 13:06:37.612054 10865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a8f7860dfb8447a8960a50b1e087806" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:37.612864 10865 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.613030 10714 consensus_queue.cc:260] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:37.614638 10714 raft_consensus.cc:3060] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.622503 10865 raft_consensus.cc:2468] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:37.624404 10714 raft_consensus.cc:2468] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:37.625161 11499 leader_election.cc:304] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:37.626170 11781 raft_consensus.cc:2804] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:37.627059 11781 raft_consensus.cc:697] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:37.628108 11781 consensus_queue.cc:237] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:37.645260 10543 catalog_manager.cc:5697] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 39f0ed0891e14f20bb25882450033da4 (127.0.107.194) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:37.656635 11810 raft_consensus.cc:993] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: : Instructing follower 7133813a08fe45908399f19d1e019ca6 to start an election
I20260709 13:06:37.657640 11810 raft_consensus.cc:1081] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Signalling peer 7133813a08fe45908399f19d1e019ca6 to start an election
I20260709 13:06:37.661166 10794 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc"
dest_uuid: "7133813a08fe45908399f19d1e019ca6"
 from {username='slave'} at 127.0.0.1:40606
I20260709 13:06:37.661986 10794 raft_consensus.cc:493] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:37.662568 10794 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.675137 10794 raft_consensus.cc:515] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:37.679100 10794 leader_election.cc:290] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Requested vote from peers 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783), 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819)
I20260709 13:06:37.679651 10864 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7"
I20260709 13:06:37.680866 10864 raft_consensus.cc:3055] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:37.680948 10713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5f424821b5444c69b981f137e64f96fc" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:37.682732 10713 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.682137 10864 raft_consensus.cc:740] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 148a4048db5a444c9daaa8cbc7b7e9d7, State: Running, Role: LEADER
I20260709 13:06:37.686156 10864 consensus_queue.cc:260] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
I20260709 13:06:37.691390 10864 raft_consensus.cc:3060] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:37.710348 10864 raft_consensus.cc:2468] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
W20260709 13:06:37.750387 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33001, user_credentials={real_user=slave}} blocked reactor thread for 58371.6us
I20260709 13:06:37.867297 10758 leader_election.cc:304] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7, 7133813a08fe45908399f19d1e019ca6; no voters: 
I20260709 13:06:37.908788 10713 raft_consensus.cc:2468] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7133813a08fe45908399f19d1e019ca6 in term 2.
W20260709 13:06:37.945361 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 65958.6us
I20260709 13:06:37.893278 11806 raft_consensus.cc:2804] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:37.974707 11806 raft_consensus.cc:697] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Becoming Leader. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:37.983212 11806 consensus_queue.cc:237] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } }
W20260709 13:06:38.094247 10684 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.196:36819, user_credentials={real_user=slave}} blocked reactor thread for 68968.1us
W20260709 13:06:38.132697 10834 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:46307, user_credentials={real_user=slave}} blocked reactor thread for 107477us
I20260709 13:06:38.156879 10862 raft_consensus.cc:1275] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:38.201669 11859 consensus_queue.cc:1048] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:38.222616 10719 raft_consensus.cc:1275] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:38.227561 11781 consensus_queue.cc:1048] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:38.214004 10543 catalog_manager.cc:5697] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 reported cstate change: term changed from 1 to 2, leader changed from 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196) to 7133813a08fe45908399f19d1e019ca6 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "7133813a08fe45908399f19d1e019ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:38.388420 10868 raft_consensus.cc:1275] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:38.392586 11860 consensus_queue.cc:1048] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:38.435784 10719 raft_consensus.cc:1275] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer 7133813a08fe45908399f19d1e019ca6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:38.438094 11806 consensus_queue.cc:1048] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:38.456193   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:38.553081   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:38.615641 10795 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "97767de992584ed3962c38a716e7122a"
dest_uuid: "7133813a08fe45908399f19d1e019ca6"
mode: GRACEFUL
new_leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:50768
I20260709 13:06:38.616408 10795 raft_consensus.cc:606] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Received request to transfer leadership to TS bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:38.618108   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260709 13:06:38.619689   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:38.725064 11806 raft_consensus.cc:993] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: : Instructing follower bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:38.725833 11806 raft_consensus.cc:1081] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Signalling peer bf9fc3264f844470bc0ea04312ff2dd6 to start an election
I20260709 13:06:38.727591 11606 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "97767de992584ed3962c38a716e7122a"
dest_uuid: "bf9fc3264f844470bc0ea04312ff2dd6"
 from {username='slave'} at 127.0.0.1:44832
I20260709 13:06:38.728109 11606 raft_consensus.cc:493] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:38.728477 11606 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.737988 11606 raft_consensus.cc:515] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:38.740806 11606 leader_election.cc:290] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Requested vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:38.741813 10795 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7133813a08fe45908399f19d1e019ca6"
I20260709 13:06:38.742054 10719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "97767de992584ed3962c38a716e7122a" candidate_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "39f0ed0891e14f20bb25882450033da4"
I20260709 13:06:38.742746 10795 raft_consensus.cc:3055] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:38.742807 10719 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.743191 10795 raft_consensus.cc:740] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7133813a08fe45908399f19d1e019ca6, State: Running, Role: LEADER
I20260709 13:06:38.744143 10795 consensus_queue.cc:260] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:38.745751 10795 raft_consensus.cc:3060] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:38.754361 10719 raft_consensus.cc:2468] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:38.757221 11502 leader_election.cc:304] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6; no voters: 
I20260709 13:06:38.758889 11858 raft_consensus.cc:2804] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:38.759431 11858 raft_consensus.cc:697] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Becoming Leader. State: Replica: bf9fc3264f844470bc0ea04312ff2dd6, State: Running, Role: LEADER
I20260709 13:06:38.762651 11858 consensus_queue.cc:237] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:38.756439 10795 raft_consensus.cc:2468] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf9fc3264f844470bc0ea04312ff2dd6 in term 2.
I20260709 13:06:38.791180 10542 catalog_manager.cc:5697] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 reported cstate change: term changed from 1 to 2, leader changed from 7133813a08fe45908399f19d1e019ca6 (127.0.107.195) to bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:39.176543 10792 raft_consensus.cc:1275] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.178823 11859 consensus_queue.cc:1048] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:39.196338 10714 raft_consensus.cc:1275] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Refusing update from remote peer bf9fc3264f844470bc0ea04312ff2dd6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:39.198755 11880 consensus_queue.cc:1048] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:39.621011   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:39.673892   431 tablet_server.cc:179] TabletServer@127.0.107.193:46307 shutting down...
W20260709 13:06:39.682539 10834 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 393 similar messages]
W20260709 13:06:39.692692 10834 consensus_peers.cc:597] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:39.695550 10834 consensus_peers.cc:597] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:39.726845 10760 consensus_peers.cc:597] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:39.757941 10686 consensus_peers.cc:597] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.761921   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:39.764017   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.765944   431 raft_consensus.cc:2243] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.766628   431 raft_consensus.cc:2272] T 1a0288d9025e4e1fa0e2269184186f0c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.771167   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.772421   431 raft_consensus.cc:2243] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.773219   431 raft_consensus.cc:2272] T 8d56437e06784282b4cb42f93fa4094a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.777771   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.779076   431 raft_consensus.cc:2243] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.779795   431 raft_consensus.cc:2272] T de828056db2a4621ba3a4e4ec0cc5d53 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.780332 10760 consensus_peers.cc:597] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:39.781935 10834 consensus_peers.cc:597] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.784745   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.786690   431 raft_consensus.cc:2243] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.788065   431 raft_consensus.cc:2272] T 5ce3e6e278894017aa528fe91907d31c P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.794837   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.796561   431 raft_consensus.cc:2243] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.797853   431 raft_consensus.cc:2272] T fe860f4e3a2945b28a2c390567bc043d P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.804935   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.806137   431 raft_consensus.cc:2243] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.807447   431 raft_consensus.cc:2272] T 3077ac317c854af19123e5a37389d01b P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.811311 10834 consensus_peers.cc:597] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.811630   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.812897   431 raft_consensus.cc:2243] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.814128   431 raft_consensus.cc:2272] T 96da319bd6aa4afb90b067caaedacf3e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.818470   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.819495   431 raft_consensus.cc:2243] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 LEADER]: Raft consensus shutting down.
W20260709 13:06:39.819734 10834 consensus_peers.cc:597] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.822811   431 raft_consensus.cc:2272] T 3cbb2820ebd1414e98070a8a401c92a0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.829090 10760 consensus_peers.cc:597] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.833182   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.834560   431 raft_consensus.cc:2243] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
W20260709 13:06:39.835399 10760 consensus_peers.cc:597] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.836090   431 raft_consensus.cc:2272] T a32195885f27470e85f07731d06d85d6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.839594   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.840512   431 raft_consensus.cc:2243] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.841291   431 raft_consensus.cc:2272] T e46c52fd7feb4776839cc022f4ef27e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.845850   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.846695   431 raft_consensus.cc:2243] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.847627   431 raft_consensus.cc:2272] T 3ebe7292f3cf4695813d0c4e643fb632 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.858920   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.860160   431 raft_consensus.cc:2243] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.862354   431 raft_consensus.cc:2272] T c2e822ba7018446f98e776bbd1c9e8a3 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.868193   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.869160   431 raft_consensus.cc:2243] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.870285   431 raft_consensus.cc:2272] T 588969ac5b174336819392cffb0cd9e7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.874454   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.875234   431 raft_consensus.cc:2243] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.875762   431 raft_consensus.cc:2272] T 38f642f247be413085d971feb65e94be P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.879069   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.880142   431 raft_consensus.cc:2243] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.881489   431 raft_consensus.cc:2272] T 25f5b1ed27b34d46b55c0e684d98d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.887456   431 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:39.888693 10834 consensus_peers.cc:597] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.889266   431 raft_consensus.cc:2243] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.890470   431 raft_consensus.cc:2272] T 5a8f7860dfb8447a8960a50b1e087806 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.892948   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.893864   431 raft_consensus.cc:2243] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.894676   431 raft_consensus.cc:2272] T 7d73ff90c30e40eaa01d6254d379aa3a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.899173   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.900044   431 raft_consensus.cc:2243] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.900581   431 raft_consensus.cc:2272] T 1260e07a0f924664b09fbb38ee72b0a1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.904492   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.906002   431 raft_consensus.cc:2243] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.907285   431 raft_consensus.cc:2272] T d744005e4a084fd29059cb6140b6d935 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.911266   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.912468   431 raft_consensus.cc:2243] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:39.911284 10760 consensus_peers.cc:597] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.913321   431 raft_consensus.cc:2272] T 4cf8b7376abf4fc68a82af6386b6a6b7 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.917753   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.918784   431 raft_consensus.cc:2243] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.919440   431 raft_consensus.cc:2272] T 8b781a7ee3874351bfa70a5ab1452a6f P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.922155 10834 consensus_peers.cc:597] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.923373   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.924366   431 raft_consensus.cc:2243] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.924959   431 raft_consensus.cc:2272] T 58ce1e65607f4e02b082593d502be0e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.925945 10760 consensus_peers.cc:597] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.929381   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.930634   431 raft_consensus.cc:2243] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:39.930485 10760 consensus_peers.cc:597] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.931478   431 raft_consensus.cc:2272] T d61cf6b014a34a9f8758702b48e6f5e0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.933476 10760 consensus_peers.cc:597] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.935632   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.936615   431 raft_consensus.cc:2243] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.937994   431 raft_consensus.cc:2272] T c3f299da07404b66be3f547c4629f7b6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.941768   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.942926   431 raft_consensus.cc:2243] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.943580   431 raft_consensus.cc:2272] T 79aeb4ec11254f438869b1aae326f5c6 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.948597 10686 consensus_peers.cc:597] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.948673   431 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:39.949409 10760 consensus_peers.cc:597] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.949841   431 raft_consensus.cc:2243] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.950894   431 raft_consensus.cc:2272] T 4ff22da804864a31992dc8a587b224f2 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.954599   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.955521   431 raft_consensus.cc:2243] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.956064   431 raft_consensus.cc:2272] T c66cc57cc4ac4587a267846214205210 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.959569   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.960572   431 raft_consensus.cc:2243] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.961074   431 raft_consensus.cc:2272] T cd614e94b3a24216a9ae872cda1709c0 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.964180   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.965013   431 raft_consensus.cc:2243] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.965553   431 raft_consensus.cc:2272] T d43a3401e55d4d8687de1b5dad8edc84 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.968312   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.969198   431 raft_consensus.cc:2243] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.969826   431 raft_consensus.cc:2272] T ec1ed15631604b35b414fe86a4c3083c P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.974166   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.975111   431 raft_consensus.cc:2243] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.975685   431 raft_consensus.cc:2272] T 7cf735aa97ef4935a1f8ac140c341ceb P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.976581 10686 consensus_peers.cc:597] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.978952   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.979944   431 raft_consensus.cc:2243] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.980471   431 raft_consensus.cc:2272] T 97d4e12c4db14c3397986584840b233b P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.986421   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.987599   431 raft_consensus.cc:2243] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:39.988152   431 raft_consensus.cc:2272] T c394fa4e36384c6eb804d29258eb7ec1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:39.991356 10834 consensus_peers.cc:597] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:39.992446   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.993242   431 raft_consensus.cc:2243] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.994259   431 raft_consensus.cc:2272] T 193ca6746d1344849958720afeb79325 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:39.996800   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:39.997430   431 raft_consensus.cc:2243] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:39.998433   431 raft_consensus.cc:2272] T fcce4acfbd7240178278200df749235a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.001881   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.002882   431 raft_consensus.cc:2243] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.003455   431 raft_consensus.cc:2272] T 80f861846a004804a0c1cca12f60514e P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.008507   431 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:40.008713 10760 consensus_peers.cc:597] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.010188   431 raft_consensus.cc:2243] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.011478   431 raft_consensus.cc:2272] T 017730403fb44573ae22679b6d9d461f P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.014401   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.015179   431 raft_consensus.cc:2243] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.015873   431 raft_consensus.cc:2272] T d50a168e65034266bc912284a133cb88 P bf9fc3264f844470bc0ea04312ff2dd6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.018828   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.019695   431 raft_consensus.cc:2243] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.020187   431 raft_consensus.cc:2272] T cd8c76caaf4443508347a95b9085bf9a P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.025195   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.026731   431 raft_consensus.cc:2243] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.027992   431 raft_consensus.cc:2272] T 97767de992584ed3962c38a716e7122a P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.028383 10686 consensus_peers.cc:597] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.032155   431 tablet_replica.cc:333] stopping tablet replica
W20260709 13:06:40.032538 10686 consensus_peers.cc:597] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.033198   431 raft_consensus.cc:2243] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.034260   431 raft_consensus.cc:2272] T bddfc21009464cbd8334356be1d409fb P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.037720   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.038961   431 raft_consensus.cc:2243] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.039618   431 raft_consensus.cc:2272] T a233da3c2e7d4fe795e5191bd32c8daf P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.043696   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.044996   431 raft_consensus.cc:2243] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.046303   431 raft_consensus.cc:2272] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P bf9fc3264f844470bc0ea04312ff2dd6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.046806 10686 consensus_peers.cc:597] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.049382   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.050065   431 raft_consensus.cc:2243] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.050457   431 raft_consensus.cc:2272] T 95a19fc069014b2db397a4c45961acda P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.053134   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:40.053726   431 raft_consensus.cc:2243] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.054148   431 raft_consensus.cc:2272] T a375c777784941d4a0945680b7406ed1 P bf9fc3264f844470bc0ea04312ff2dd6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.084302 10760 consensus_peers.cc:597] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.094966 10834 consensus_peers.cc:597] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.100376 10834 consensus_peers.cc:597] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.101854 10686 consensus_peers.cc:597] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.182327   431 tablet_server.cc:196] TabletServer@127.0.107.193:46307 shutdown complete.
W20260709 13:06:40.185930 10834 consensus_peers.cc:597] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.238365 10686 consensus_peers.cc:597] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.250173 10834 consensus_peers.cc:597] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Couldn't send request to peer bf9fc3264f844470bc0ea04312ff2dd6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.317401   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:06:40.334785 10761 consensus_peers.cc:597] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.360030 10761 consensus_peers.cc:597] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.381150 10835 consensus_peers.cc:597] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.396847 10761 consensus_peers.cc:597] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.405364 10835 consensus_peers.cc:597] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.418246   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:40.422240   431 tablet_replica.cc:333] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.423800   431 raft_consensus.cc:2243] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.424923   431 raft_consensus.cc:2272] T 3077ac317c854af19123e5a37389d01b P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.432502   431 tablet_replica.cc:333] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.433701   431 raft_consensus.cc:2243] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
W20260709 13:06:40.447593 10835 consensus_peers.cc:597] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.449074 10835 consensus_peers.cc:597] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.453610   431 raft_consensus.cc:2272] T 7cf735aa97ef4935a1f8ac140c341ceb P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.457442   431 tablet_replica.cc:333] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.458446   431 raft_consensus.cc:2243] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.459112   431 raft_consensus.cc:2272] T a32195885f27470e85f07731d06d85d6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.462615   431 tablet_replica.cc:333] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.465663   431 raft_consensus.cc:2243] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.468457   431 raft_consensus.cc:2272] T 3ebe7292f3cf4695813d0c4e643fb632 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.472517   431 tablet_replica.cc:333] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.473601   431 raft_consensus.cc:2243] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.474409   431 raft_consensus.cc:2272] T 5f424821b5444c69b981f137e64f96fc P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.478080   431 tablet_replica.cc:333] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.479578   431 raft_consensus.cc:2243] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.480856   431 raft_consensus.cc:2272] T 6aa9d4273100419bb29366421f9a294c P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.484688   431 tablet_replica.cc:333] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.485584   431 raft_consensus.cc:2243] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.486366   431 raft_consensus.cc:2272] T 3cbb2820ebd1414e98070a8a401c92a0 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.489197 10835 consensus_peers.cc:597] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.490283   431 tablet_replica.cc:333] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.491590   431 raft_consensus.cc:2243] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.492915   431 raft_consensus.cc:2272] T 37b295fd9eb746c78bae881d54b57351 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.496330   431 tablet_replica.cc:333] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.497175   431 raft_consensus.cc:2243] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.498451   431 raft_consensus.cc:2272] T 97d4e12c4db14c3397986584840b233b P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.502153   431 tablet_replica.cc:333] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.503490   431 raft_consensus.cc:2243] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.504765   431 raft_consensus.cc:2272] T 588969ac5b174336819392cffb0cd9e7 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.509927   431 tablet_replica.cc:333] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.511623   431 raft_consensus.cc:2243] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.512356   431 raft_consensus.cc:2272] T 96da319bd6aa4afb90b067caaedacf3e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.515969   431 tablet_replica.cc:333] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.517099   431 raft_consensus.cc:2243] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.518373   431 raft_consensus.cc:2272] T ca1d8934020d46ad9b3a2a19c509cf56 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.521898   431 tablet_replica.cc:333] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.522898   431 raft_consensus.cc:2243] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.524152   431 raft_consensus.cc:2272] T 1260e07a0f924664b09fbb38ee72b0a1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.527979   431 tablet_replica.cc:333] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.528873   431 raft_consensus.cc:2243] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.529495   431 raft_consensus.cc:2272] T 7d73ff90c30e40eaa01d6254d379aa3a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.533635   431 tablet_replica.cc:333] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.534878   431 raft_consensus.cc:2243] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.535560   431 raft_consensus.cc:2272] T 5a8f7860dfb8447a8960a50b1e087806 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.537011 10761 consensus_peers.cc:597] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.538854   431 tablet_replica.cc:333] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.539901   431 raft_consensus.cc:2243] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.540561   431 raft_consensus.cc:2272] T 25f5b1ed27b34d46b55c0e684d98d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.543835   431 tablet_replica.cc:333] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.544695   431 raft_consensus.cc:2243] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.545840   431 raft_consensus.cc:2272] T cd614e94b3a24216a9ae872cda1709c0 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.548400 10835 consensus_peers.cc:597] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.548879   431 tablet_replica.cc:333] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.549870   431 raft_consensus.cc:2243] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.551007   431 raft_consensus.cc:2272] T 0681122286764dc3b7d97a1a9962e46e P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.554373   431 tablet_replica.cc:333] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.555332   431 raft_consensus.cc:2243] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.555912   431 raft_consensus.cc:2272] T de828056db2a4621ba3a4e4ec0cc5d53 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.559154   431 tablet_replica.cc:333] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.560038   431 raft_consensus.cc:2243] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:40.559976 10835 consensus_peers.cc:597] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.560953   431 raft_consensus.cc:2272] T d744005e4a084fd29059cb6140b6d935 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.564218   431 tablet_replica.cc:333] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.565079   431 raft_consensus.cc:2243] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.565706   431 raft_consensus.cc:2272] T 410064a793614346a815b98b7c6d5316 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.569118   431 tablet_replica.cc:333] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.569964   431 raft_consensus.cc:2243] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.570546   431 raft_consensus.cc:2272] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.573750   431 tablet_replica.cc:333] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.574560   431 raft_consensus.cc:2243] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.575109   431 raft_consensus.cc:2272] T 9a50b607435548d3a8c9a26b49489a9a P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.578306   431 tablet_replica.cc:333] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.579196   431 raft_consensus.cc:2243] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.579803   431 raft_consensus.cc:2272] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.583702   431 tablet_replica.cc:333] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.584923   431 raft_consensus.cc:2243] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.585736   431 raft_consensus.cc:2272] T a233da3c2e7d4fe795e5191bd32c8daf P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.589541   431 tablet_replica.cc:333] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.590530   431 raft_consensus.cc:2243] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.591200   431 raft_consensus.cc:2272] T a37f22f2791e488bb016469737ecef9d P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.594745   431 tablet_replica.cc:333] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.595633   431 raft_consensus.cc:2243] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.596319   431 raft_consensus.cc:2272] T fe860f4e3a2945b28a2c390567bc043d P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.599939   431 tablet_replica.cc:333] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.600876   431 raft_consensus.cc:2243] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.601612   431 raft_consensus.cc:2272] T c3f299da07404b66be3f547c4629f7b6 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.605792   431 tablet_replica.cc:333] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.606971   431 raft_consensus.cc:2243] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.608112   431 raft_consensus.cc:2272] T a375c777784941d4a0945680b7406ed1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.611397   431 tablet_replica.cc:333] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.612238   431 raft_consensus.cc:2243] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.613389   431 raft_consensus.cc:2272] T 76a046317986428fbc51a3609a2bf91e P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.616379   431 tablet_replica.cc:333] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.617210   431 raft_consensus.cc:2243] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.619163   431 raft_consensus.cc:2272] T a95dffb6d91746b6bc3fd74d12cbdccc P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.622035   431 tablet_replica.cc:333] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.622745   431 raft_consensus.cc:2243] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.623368   431 raft_consensus.cc:2272] T 69a32458d0e54f08a475b354e480c787 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.626570   431 tablet_replica.cc:333] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.627481   431 raft_consensus.cc:2243] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.628119   431 raft_consensus.cc:2272] T fcce4acfbd7240178278200df749235a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.630280 10761 consensus_peers.cc:597] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.631076   431 tablet_replica.cc:333] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.631804   431 raft_consensus.cc:2243] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.632515   431 raft_consensus.cc:2272] T 193ca6746d1344849958720afeb79325 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.635310   431 tablet_replica.cc:333] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.635895   431 raft_consensus.cc:2243] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.636787   431 raft_consensus.cc:2272] T c394fa4e36384c6eb804d29258eb7ec1 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.639433   431 tablet_replica.cc:333] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.640035   431 raft_consensus.cc:2243] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.641144   431 raft_consensus.cc:2272] T 80f861846a004804a0c1cca12f60514e P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.644497   431 tablet_replica.cc:333] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.645298   431 raft_consensus.cc:2243] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.645831   431 raft_consensus.cc:2272] T 017730403fb44573ae22679b6d9d461f P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.648489   431 tablet_replica.cc:333] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.649333   431 raft_consensus.cc:2243] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.650035   431 raft_consensus.cc:2272] T d50a168e65034266bc912284a133cb88 P 39f0ed0891e14f20bb25882450033da4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.653376   431 tablet_replica.cc:333] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.654034   431 raft_consensus.cc:2243] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.654422   431 raft_consensus.cc:2272] T d52e964d865840ee885c5cb005982d82 P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.656535   431 tablet_replica.cc:333] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.657127   431 raft_consensus.cc:2243] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.657683   431 raft_consensus.cc:2272] T 97767de992584ed3962c38a716e7122a P 39f0ed0891e14f20bb25882450033da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.659790   431 tablet_replica.cc:333] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.660432   431 raft_consensus.cc:2243] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.660882   431 raft_consensus.cc:2272] T 77087779392747febcc9e60b2f23f96f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.663182   431 tablet_replica.cc:333] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.663717   431 raft_consensus.cc:2243] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.664171   431 raft_consensus.cc:2272] T 58ce1e65607f4e02b082593d502be0e9 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.666388   431 tablet_replica.cc:333] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.666945   431 raft_consensus.cc:2243] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.667399   431 raft_consensus.cc:2272] T 8b781a7ee3874351bfa70a5ab1452a6f P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.669417   431 tablet_replica.cc:333] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4: stopping tablet replica
I20260709 13:06:40.669963   431 raft_consensus.cc:2243] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.670826   431 raft_consensus.cc:2272] T 912716eee3434e46aead233d6f5a7c83 P 39f0ed0891e14f20bb25882450033da4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.703064 10761 consensus_peers.cc:597] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.742780 10761 consensus_peers.cc:597] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.747192 10761 consensus_peers.cc:597] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:40.747272 10835 consensus_peers.cc:597] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Couldn't send request to peer 39f0ed0891e14f20bb25882450033da4. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:40.766927   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 13:06:40.876613   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
I20260709 13:06:40.888773 11888 raft_consensus.cc:493] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:40.889565 11888 raft_consensus.cc:515] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
W20260709 13:06:40.900530 10761 leader_election.cc:336] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:40.899150 11888 leader_election.cc:290] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:40.907732 11890 raft_consensus.cc:493] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:40.908527 11890 raft_consensus.cc:515] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:40.911666 11890 leader_election.cc:290] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:40.913478 10760 leader_election.cc:336] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:40.914120 10760 leader_election.cc:304] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:40.916153 11856 raft_consensus.cc:2749] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 13:06:40.919667 10835 leader_election.cc:336] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
W20260709 13:06:40.920938 10834 leader_election.cc:336] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:40.921463 10834 leader_election.cc:304] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:40.922252 11890 raft_consensus.cc:2749] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:40.953850   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:40.955262   431 tablet_replica.cc:333] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.956405   431 raft_consensus.cc:2243] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.957626   431 raft_consensus.cc:2272] T c66cc57cc4ac4587a267846214205210 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.960014 11888 raft_consensus.cc:493] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:40.960651 11888 raft_consensus.cc:515] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:40.961671   431 tablet_replica.cc:333] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.962622   431 raft_consensus.cc:2243] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.963865   431 raft_consensus.cc:2272] T 77087779392747febcc9e60b2f23f96f P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.964018 11888 leader_election.cc:290] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:40.968243   431 tablet_replica.cc:333] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
W20260709 13:06:40.968922 10760 leader_election.cc:336] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:40.969228   431 raft_consensus.cc:2243] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.969877   431 raft_consensus.cc:2272] T 1260e07a0f924664b09fbb38ee72b0a1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:40.971217 10761 leader_election.cc:336] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:40.971738 10761 leader_election.cc:304] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:40.972493 11888 raft_consensus.cc:2749] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:40.973595   431 tablet_replica.cc:333] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.974407   431 raft_consensus.cc:2243] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.974996   431 raft_consensus.cc:2272] T 7cf735aa97ef4935a1f8ac140c341ceb P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.978031   431 tablet_replica.cc:333] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.979008   431 raft_consensus.cc:2243] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.979615   431 raft_consensus.cc:2272] T ec1ed15631604b35b414fe86a4c3083c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.981997   431 tablet_replica.cc:333] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.982627   431 raft_consensus.cc:2243] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.983121   431 raft_consensus.cc:2272] T 37b295fd9eb746c78bae881d54b57351 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.986914   431 tablet_replica.cc:333] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.988165   431 raft_consensus.cc:2243] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:40.988857   431 raft_consensus.cc:2272] T 3ebe7292f3cf4695813d0c4e643fb632 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.992547   431 tablet_replica.cc:333] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.993683   431 raft_consensus.cc:2243] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.994769   431 raft_consensus.cc:2272] T 5f424821b5444c69b981f137e64f96fc P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:40.997893   431 tablet_replica.cc:333] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:40.998914   431 raft_consensus.cc:2243] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:40.999840   431 raft_consensus.cc:2272] T d61cf6b014a34a9f8758702b48e6f5e0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.002803   431 tablet_replica.cc:333] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.003747   431 raft_consensus.cc:2243] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:41.003964 10832 consensus_peers.cc:597] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.004529   431 raft_consensus.cc:2272] T 6aa9d4273100419bb29366421f9a294c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.007009   431 tablet_replica.cc:333] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.007675   431 raft_consensus.cc:2243] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.008903   431 raft_consensus.cc:2272] T 1a0288d9025e4e1fa0e2269184186f0c P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.011001 10832 consensus_peers.cc:597] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.012522   431 tablet_replica.cc:333] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.013438   431 raft_consensus.cc:2243] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.014086   431 raft_consensus.cc:2272] T ca1d8934020d46ad9b3a2a19c509cf56 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.017664   431 tablet_replica.cc:333] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.018848   431 raft_consensus.cc:2243] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.019532   431 raft_consensus.cc:2272] T 96da319bd6aa4afb90b067caaedacf3e P 7133813a08fe45908399f19d1e019ca6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.023028   431 tablet_replica.cc:333] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.024034   431 raft_consensus.cc:2243] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.025204   431 raft_consensus.cc:2272] T cd8c76caaf4443508347a95b9085bf9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.028558   431 tablet_replica.cc:333] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.029611   431 raft_consensus.cc:2243] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.030519   431 raft_consensus.cc:2272] T cd614e94b3a24216a9ae872cda1709c0 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.033787   431 tablet_replica.cc:333] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.034164 11888 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.034786 11888 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:41.034683   431 raft_consensus.cc:2243] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.036477   431 raft_consensus.cc:2272] T d43a3401e55d4d8687de1b5dad8edc84 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.037596 11888 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 148a4048db5a444c9daaa8cbc7b7e9d7 (127.0.107.196:36819), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
I20260709 13:06:41.038901 10868 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bddfc21009464cbd8334356be1d409fb" candidate_uuid: "7133813a08fe45908399f19d1e019ca6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" is_pre_election: true
I20260709 13:06:41.040041   431 tablet_replica.cc:333] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.041079   431 raft_consensus.cc:2243] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.041780   431 raft_consensus.cc:2272] T 0681122286764dc3b7d97a1a9962e46e P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.042330 10760 leader_election.cc:336] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.042843 10760 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 148a4048db5a444c9daaa8cbc7b7e9d7, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.043601 11888 raft_consensus.cc:2749] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:41.044853   431 tablet_replica.cc:333] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.045764   431 raft_consensus.cc:2243] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.046789   431 raft_consensus.cc:2272] T 8d56437e06784282b4cb42f93fa4094a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.049789   431 tablet_replica.cc:333] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.050424   431 raft_consensus.cc:2243] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.051311   431 raft_consensus.cc:2272] T de828056db2a4621ba3a4e4ec0cc5d53 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.052342 11896 raft_consensus.cc:493] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.052812 11896 raft_consensus.cc:515] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } } peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } }
I20260709 13:06:41.053977   431 tablet_replica.cc:333] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.054898 11896 leader_election.cc:290] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:41.055044   431 raft_consensus.cc:2243] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.055860   431 raft_consensus.cc:2272] T 38f642f247be413085d971feb65e94be P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.059134   431 tablet_replica.cc:333] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
W20260709 13:06:41.059806 10761 leader_election.cc:336] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.060029   431 raft_consensus.cc:2243] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:41.060492 10760 leader_election.cc:336] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.061129 10760 leader_election.cc:304] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7133813a08fe45908399f19d1e019ca6; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.061107   431 raft_consensus.cc:2272] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.062145 11896 raft_consensus.cc:2706] T 25f5b1ed27b34d46b55c0e684d98d935 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Received pre-election callback for term 3 while not running: Illegal state: RaftConsensus is not running: State = Stopped
I20260709 13:06:41.065896   431 tablet_replica.cc:333] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.066870   431 raft_consensus.cc:2243] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.068542   431 raft_consensus.cc:2272] T 588969ac5b174336819392cffb0cd9e7 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.071208   431 tablet_replica.cc:333] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.071869   431 raft_consensus.cc:2243] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.072383   431 raft_consensus.cc:2272] T c2e822ba7018446f98e776bbd1c9e8a3 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.074802   431 tablet_replica.cc:333] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.075443   431 raft_consensus.cc:2243] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.076337   431 raft_consensus.cc:2272] T a37f22f2791e488bb016469737ecef9d P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.078765   431 tablet_replica.cc:333] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.079383   431 raft_consensus.cc:2243] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.079841   431 raft_consensus.cc:2272] T 5ce3e6e278894017aa528fe91907d31c P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.082845   431 tablet_replica.cc:333] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.083519   431 raft_consensus.cc:2243] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.084455   431 raft_consensus.cc:2272] T fe860f4e3a2945b28a2c390567bc043d P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.087078   431 tablet_replica.cc:333] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.087814   431 raft_consensus.cc:2243] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.088754   431 raft_consensus.cc:2272] T 95a19fc069014b2db397a4c45961acda P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.091403   431 tablet_replica.cc:333] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.092123   431 raft_consensus.cc:2243] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.092679   431 raft_consensus.cc:2272] T a375c777784941d4a0945680b7406ed1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.095387   431 tablet_replica.cc:333] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.096062   431 raft_consensus.cc:2243] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.096688   431 raft_consensus.cc:2272] T 76a046317986428fbc51a3609a2bf91e P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.098991   431 tablet_replica.cc:333] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.099610   431 raft_consensus.cc:2243] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.100080   431 raft_consensus.cc:2272] T 97767de992584ed3962c38a716e7122a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.101912 11890 raft_consensus.cc:493] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.102654 11890 raft_consensus.cc:515] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.103001   431 tablet_replica.cc:333] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.103998   431 raft_consensus.cc:2243] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.104643   431 raft_consensus.cc:2272] T bddfc21009464cbd8334356be1d409fb P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.104809 11890 leader_election.cc:290] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:41.107921   431 tablet_replica.cc:333] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.109663   431 raft_consensus.cc:2243] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:41.110273 10834 leader_election.cc:336] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:41.111076 10835 leader_election.cc:336] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.111091   431 raft_consensus.cc:2272] T fcce4acfbd7240178278200df749235a P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.111626 10835 leader_election.cc:304] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.112440 11890 raft_consensus.cc:2749] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:41.114387   431 tablet_replica.cc:333] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.115211   431 raft_consensus.cc:2243] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.116070   431 raft_consensus.cc:2272] T 193ca6746d1344849958720afeb79325 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.119663   431 tablet_replica.cc:333] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.120401   431 raft_consensus.cc:2243] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 13:06:41.120784 10832 consensus_peers.cc:597] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.121121   431 raft_consensus.cc:2272] T c394fa4e36384c6eb804d29258eb7ec1 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.125196   431 tablet_replica.cc:333] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.126368   431 raft_consensus.cc:2243] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.127586   431 raft_consensus.cc:2272] T a233da3c2e7d4fe795e5191bd32c8daf P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.131034   431 tablet_replica.cc:333] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.131620   431 raft_consensus.cc:2243] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.132658   431 raft_consensus.cc:2272] T 9a50b607435548d3a8c9a26b49489a9a P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.135192   431 tablet_replica.cc:333] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.135779   431 raft_consensus.cc:2243] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.136240   431 raft_consensus.cc:2272] T 912716eee3434e46aead233d6f5a7c83 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.136915 11890 raft_consensus.cc:493] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.137539 11890 raft_consensus.cc:515] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.139156   431 tablet_replica.cc:333] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.139636 11890 leader_election.cc:290] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:41.140435   431 raft_consensus.cc:2243] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.141111   431 raft_consensus.cc:2272] T 79aeb4ec11254f438869b1aae326f5c6 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.144116   431 tablet_replica.cc:333] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
W20260709 13:06:41.144160 10835 leader_election.cc:336] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.144788   431 raft_consensus.cc:2243] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.145711   431 raft_consensus.cc:2272] T d52e964d865840ee885c5cb005982d82 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.144817 10834 leader_election.cc:336] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.146551 10834 leader_election.cc:304] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.147469 11809 raft_consensus.cc:2749] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:41.148929   431 tablet_replica.cc:333] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
W20260709 13:06:41.149798 10832 consensus_peers.cc:597] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.149897   431 raft_consensus.cc:2243] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.150553   431 raft_consensus.cc:2272] T 4ff22da804864a31992dc8a587b224f2 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.153332   431 tablet_replica.cc:333] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.153707 11809 raft_consensus.cc:493] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.153985   431 raft_consensus.cc:2243] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.154260 11809 raft_consensus.cc:515] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.154870   431 raft_consensus.cc:2272] T e46c52fd7feb4776839cc022f4ef27e0 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.156481 11809 leader_election.cc:290] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
I20260709 13:06:41.157415   431 tablet_replica.cc:333] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.158017   431 raft_consensus.cc:2243] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.158514   431 raft_consensus.cc:2272] T a95dffb6d91746b6bc3fd74d12cbdccc P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.160640 10835 leader_election.cc:336] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.161289   431 tablet_replica.cc:333] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.162339   431 raft_consensus.cc:2243] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.163020   431 raft_consensus.cc:2272] T 69a32458d0e54f08a475b354e480c787 P 7133813a08fe45908399f19d1e019ca6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.162936 10834 leader_election.cc:336] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.163887 10834 leader_election.cc:304] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.164590 11809 raft_consensus.cc:2749] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 13:06:41.166708   431 tablet_replica.cc:333] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6: stopping tablet replica
I20260709 13:06:41.167340   431 raft_consensus.cc:2243] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.167826   431 raft_consensus.cc:2272] T 410064a793614346a815b98b7c6d5316 P 7133813a08fe45908399f19d1e019ca6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.175058 10832 consensus_peers.cc:597] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:41.183037 10832 consensus_peers.cc:597] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.183473 11809 raft_consensus.cc:493] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.184137 11809 raft_consensus.cc:515] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7133813a08fe45908399f19d1e019ca6" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33001 } } peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } }
I20260709 13:06:41.186861 11809 leader_election.cc:290] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001), bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307)
W20260709 13:06:41.189476 10832 leader_election.cc:336] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 13:06:41.191767 10834 leader_election.cc:336] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.192203 10834 leader_election.cc:304] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 7133813a08fe45908399f19d1e019ca6, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.192920 11809 raft_consensus.cc:2749] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:06:41.195981 10832 consensus_peers.cc:597] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:41.205916 10832 consensus_peers.cc:597] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.249699 11809 raft_consensus.cc:493] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.250440 11809 raft_consensus.cc:515] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.252964 11809 leader_election.cc:290] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:41.257359 10834 leader_election.cc:336] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:41.257934 10835 leader_election.cc:336] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.258601 10835 leader_election.cc:304] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.259706   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:06:41.260727 11809 raft_consensus.cc:2749] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:41.277721 11809 raft_consensus.cc:493] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.278469 11809 raft_consensus.cc:515] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.281410 11809 leader_election.cc:290] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:41.287340 10834 leader_election.cc:336] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:41.288484 10835 leader_election.cc:336] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.288918 10835 leader_election.cc:304] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.289855 11809 raft_consensus.cc:2749] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:41.299381 11809 raft_consensus.cc:493] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.300050 11809 raft_consensus.cc:515] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.302583 11809 leader_election.cc:290] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:41.307288 10834 leader_election.cc:336] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
W20260709 13:06:41.307945 10835 leader_election.cc:336] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
I20260709 13:06:41.308305 10835 leader_election.cc:304] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.308948 11809 raft_consensus.cc:2749] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 13:06:41.356062 10832 consensus_peers.cc:597] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.357846   431 tablet_server.cc:179] TabletServer@127.0.107.196:0 shutting down...
I20260709 13:06:41.364302 11809 raft_consensus.cc:493] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader bf9fc3264f844470bc0ea04312ff2dd6)
I20260709 13:06:41.364936 11809 raft_consensus.cc:515] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "148a4048db5a444c9daaa8cbc7b7e9d7" member_type: VOTER last_known_addr { host: "127.0.107.196" port: 36819 } } peers { permanent_uuid: "bf9fc3264f844470bc0ea04312ff2dd6" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 46307 } } peers { permanent_uuid: "39f0ed0891e14f20bb25882450033da4" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 45783 } }
I20260709 13:06:41.366786 11809 leader_election.cc:290] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307), 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783)
W20260709 13:06:41.371526 10835 leader_election.cc:336] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 39f0ed0891e14f20bb25882450033da4 (127.0.107.194:45783): Network error: Client connection negotiation failed: client connection to 127.0.107.194:45783: connect: Connection refused (error 111)
W20260709 13:06:41.372308 10834 leader_election.cc:336] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer bf9fc3264f844470bc0ea04312ff2dd6 (127.0.107.193:46307): Network error: Client connection negotiation failed: client connection to 127.0.107.193:46307: connect: Connection refused (error 111)
I20260709 13:06:41.372808 10834 leader_election.cc:304] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 148a4048db5a444c9daaa8cbc7b7e9d7; no voters: 39f0ed0891e14f20bb25882450033da4, bf9fc3264f844470bc0ea04312ff2dd6
I20260709 13:06:41.373590 11809 raft_consensus.cc:2749] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 13:06:41.415915   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:41.417261   431 tablet_replica.cc:333] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.418448   431 raft_consensus.cc:2243] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.419251   431 raft_consensus.cc:2272] T 3077ac317c854af19123e5a37389d01b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.422272   431 tablet_replica.cc:333] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.422894   431 raft_consensus.cc:2243] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.423323   431 raft_consensus.cc:2272] T c66cc57cc4ac4587a267846214205210 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.425669   431 tablet_replica.cc:333] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.426247   431 raft_consensus.cc:2243] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.427083   431 raft_consensus.cc:2272] T ec1ed15631604b35b414fe86a4c3083c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.429153   431 tablet_replica.cc:333] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.429774   431 raft_consensus.cc:2243] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.430179   431 raft_consensus.cc:2272] T a32195885f27470e85f07731d06d85d6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.432515   431 tablet_replica.cc:333] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.433214   431 raft_consensus.cc:2243] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.433841   431 raft_consensus.cc:2272] T d61cf6b014a34a9f8758702b48e6f5e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.436465   431 tablet_replica.cc:333] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.437152   431 raft_consensus.cc:2243] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.437709   431 raft_consensus.cc:2272] T 5f424821b5444c69b981f137e64f96fc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.439811   431 tablet_replica.cc:333] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.440411   431 raft_consensus.cc:2243] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.440912   431 raft_consensus.cc:2272] T 3cbb2820ebd1414e98070a8a401c92a0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.443521   431 tablet_replica.cc:333] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.444207   431 raft_consensus.cc:2243] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.444677   431 raft_consensus.cc:2272] T 37b295fd9eb746c78bae881d54b57351 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.446933   431 tablet_replica.cc:333] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.447525   431 raft_consensus.cc:2243] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.447978   431 raft_consensus.cc:2272] T 97d4e12c4db14c3397986584840b233b P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.450546   431 tablet_replica.cc:333] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.451269   431 raft_consensus.cc:2243] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.451833   431 raft_consensus.cc:2272] T 6aa9d4273100419bb29366421f9a294c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.454824   431 tablet_replica.cc:333] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.455590   431 raft_consensus.cc:2243] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.456116   431 raft_consensus.cc:2272] T a37f22f2791e488bb016469737ecef9d P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.458626   431 tablet_replica.cc:333] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.459285   431 raft_consensus.cc:2243] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.460214   431 raft_consensus.cc:2272] T 5ce3e6e278894017aa528fe91907d31c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.462646   431 tablet_replica.cc:333] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.463348   431 raft_consensus.cc:2243] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.463861   431 raft_consensus.cc:2272] T 1a0288d9025e4e1fa0e2269184186f0c P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.466123   431 tablet_replica.cc:333] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.466773   431 raft_consensus.cc:2243] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.467250   431 raft_consensus.cc:2272] T ca1d8934020d46ad9b3a2a19c509cf56 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.469779   431 tablet_replica.cc:333] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.470675   431 raft_consensus.cc:2243] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.471684   431 raft_consensus.cc:2272] T 7d73ff90c30e40eaa01d6254d379aa3a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.474838   431 tablet_replica.cc:333] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.475755   431 raft_consensus.cc:2243] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.476292   431 raft_consensus.cc:2272] T bddfc21009464cbd8334356be1d409fb P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.478790   431 tablet_replica.cc:333] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.479466   431 raft_consensus.cc:2243] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.480391   431 raft_consensus.cc:2272] T d43a3401e55d4d8687de1b5dad8edc84 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.482836   431 tablet_replica.cc:333] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.483449   431 raft_consensus.cc:2243] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.483908   431 raft_consensus.cc:2272] T 0681122286764dc3b7d97a1a9962e46e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.486275   431 tablet_replica.cc:333] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.486891   431 raft_consensus.cc:2243] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.487800   431 raft_consensus.cc:2272] T 38f642f247be413085d971feb65e94be P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.490825   431 tablet_replica.cc:333] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.491864   431 raft_consensus.cc:2243] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.492516   431 raft_consensus.cc:2272] T d744005e4a084fd29059cb6140b6d935 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:41.494149 10832 consensus_peers.cc:597] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 -> Peer 7133813a08fe45908399f19d1e019ca6 (127.0.107.195:33001): Couldn't send request to peer 7133813a08fe45908399f19d1e019ca6. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.195:33001: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:41.495656   431 tablet_replica.cc:333] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.496394   431 raft_consensus.cc:2243] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.497490   431 raft_consensus.cc:2272] T 410064a793614346a815b98b7c6d5316 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.499935   431 tablet_replica.cc:333] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.500629   431 raft_consensus.cc:2243] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.501637   431 raft_consensus.cc:2272] T 4cf8b7376abf4fc68a82af6386b6a6b7 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.504051   431 tablet_replica.cc:333] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.504705   431 raft_consensus.cc:2243] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.505683   431 raft_consensus.cc:2272] T c2e822ba7018446f98e776bbd1c9e8a3 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.508136   431 tablet_replica.cc:333] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.508800   431 raft_consensus.cc:2243] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.509327   431 raft_consensus.cc:2272] T c3f299da07404b66be3f547c4629f7b6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.512176   431 tablet_replica.cc:333] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.512861   431 raft_consensus.cc:2243] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.513363   431 raft_consensus.cc:2272] T 95a19fc069014b2db397a4c45961acda P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.517135   431 tablet_replica.cc:333] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.517908   431 raft_consensus.cc:2243] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.518580   431 raft_consensus.cc:2272] T 76a046317986428fbc51a3609a2bf91e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.521459   431 tablet_replica.cc:333] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.522218   431 raft_consensus.cc:2243] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.522743   431 raft_consensus.cc:2272] T cd8c76caaf4443508347a95b9085bf9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.525164   431 tablet_replica.cc:333] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.525904   431 raft_consensus.cc:2243] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.526628   431 raft_consensus.cc:2272] T 80f861846a004804a0c1cca12f60514e P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.529289   431 tablet_replica.cc:333] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.530028   431 raft_consensus.cc:2243] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.530578   431 raft_consensus.cc:2272] T 017730403fb44573ae22679b6d9d461f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.532956   431 tablet_replica.cc:333] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.533705   431 raft_consensus.cc:2243] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.534698   431 raft_consensus.cc:2272] T d50a168e65034266bc912284a133cb88 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.537397   431 tablet_replica.cc:333] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.538101   431 raft_consensus.cc:2243] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.538606   431 raft_consensus.cc:2272] T e46c52fd7feb4776839cc022f4ef27e0 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.540902   431 tablet_replica.cc:333] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.541581   431 raft_consensus.cc:2243] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.542106   431 raft_consensus.cc:2272] T a95dffb6d91746b6bc3fd74d12cbdccc P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.544523   431 tablet_replica.cc:333] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.545240   431 raft_consensus.cc:2243] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.545830   431 raft_consensus.cc:2272] T 8d56437e06784282b4cb42f93fa4094a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.548575   431 tablet_replica.cc:333] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.549237   431 raft_consensus.cc:2243] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.550390   431 raft_consensus.cc:2272] T 69a32458d0e54f08a475b354e480c787 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.553162   431 tablet_replica.cc:333] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.553877   431 raft_consensus.cc:2243] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.554574   431 raft_consensus.cc:2272] T f3f27b1950c54af2b8d48d3e4c7cb7e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.556952   431 tablet_replica.cc:333] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.557717   431 raft_consensus.cc:2243] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.558241   431 raft_consensus.cc:2272] T 9a50b607435548d3a8c9a26b49489a9a P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.560822   431 tablet_replica.cc:333] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.561786   431 raft_consensus.cc:2243] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.562425   431 raft_consensus.cc:2272] T 912716eee3434e46aead233d6f5a7c83 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.564860   431 tablet_replica.cc:333] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.565488   431 raft_consensus.cc:2243] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.566958   431 raft_consensus.cc:2272] T 79aeb4ec11254f438869b1aae326f5c6 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.570219   431 tablet_replica.cc:333] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.571134   431 raft_consensus.cc:2243] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.571998   431 raft_consensus.cc:2272] T d52e964d865840ee885c5cb005982d82 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.574558   431 tablet_replica.cc:333] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.575132   431 raft_consensus.cc:2243] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.576310   431 raft_consensus.cc:2272] T 4ff22da804864a31992dc8a587b224f2 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.578944   431 tablet_replica.cc:333] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.579546   431 raft_consensus.cc:2243] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.580003   431 raft_consensus.cc:2272] T 5a8f7860dfb8447a8960a50b1e087806 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.582422   431 tablet_replica.cc:333] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.583009   431 raft_consensus.cc:2243] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:41.583526   431 raft_consensus.cc:2272] T 77087779392747febcc9e60b2f23f96f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.585821   431 tablet_replica.cc:333] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.586385   431 raft_consensus.cc:2243] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.587278   431 raft_consensus.cc:2272] T 58ce1e65607f4e02b082593d502be0e9 P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.589603   431 tablet_replica.cc:333] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7: stopping tablet replica
I20260709 13:06:41.590106   431 raft_consensus.cc:2243] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.590919   431 raft_consensus.cc:2272] T 8b781a7ee3874351bfa70a5ab1452a6f P 148a4048db5a444c9daaa8cbc7b7e9d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.678125   431 tablet_server.cc:196] TabletServer@127.0.107.196:0 shutdown complete.
I20260709 13:06:41.757690   431 master.cc:562] Master@127.0.107.254:39861 shutting down...
I20260709 13:06:41.793881   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:41.794631   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:41.794996   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2c2db47040184744a819b7db4d1a589a: stopping tablet replica
I20260709 13:06:41.852787   431 master.cc:584] Master@127.0.107.254:39861 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (38926 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 13:06:41.970438   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:45061
I20260709 13:06:41.972375   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:41.979558 11901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:41.979598 11902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:41.984623 11904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:41.987329   431 server_base.cc:1061] running on GCE node
I20260709 13:06:41.988773   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:41.988989   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:41.989112   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602401989100 us; error 0 us; skew 500 ppm
I20260709 13:06:41.989827   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:41.993669   431 webserver.cc:533] Webserver started at http://127.0.107.254:35823/ using document root <none> and password file <none>
I20260709 13:06:41.994374   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:41.994635   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:41.995064   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:41.996392   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "466dfef8aeae4d6e901e9baf91d0a31f"
format_stamp: "Formatted at 2026-07-09 13:06:41 on dist-test-slave-p8rg"
I20260709 13:06:42.001493   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:06:42.007066 11909 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.008244   431 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.003s
I20260709 13:06:42.008721   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "466dfef8aeae4d6e901e9baf91d0a31f"
format_stamp: "Formatted at 2026-07-09 13:06:41 on dist-test-slave-p8rg"
I20260709 13:06:42.009060   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:42.027966   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:42.029618   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:42.077348   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:45061
I20260709 13:06:42.077457 11960 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:45061 every 8 connection(s)
I20260709 13:06:42.081984 11961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.096063 11961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f: Bootstrap starting.
I20260709 13:06:42.102386 11961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:42.107885 11961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f: No bootstrap required, opened a new log
I20260709 13:06:42.110909 11961 raft_consensus.cc:359] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER }
I20260709 13:06:42.111354 11961 raft_consensus.cc:385] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:42.111644 11961 raft_consensus.cc:740] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466dfef8aeae4d6e901e9baf91d0a31f, State: Initialized, Role: FOLLOWER
I20260709 13:06:42.112353 11961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER }
I20260709 13:06:42.112809 11961 raft_consensus.cc:399] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:42.113039 11961 raft_consensus.cc:493] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:42.113720 11961 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:42.119372 11961 raft_consensus.cc:515] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER }
I20260709 13:06:42.119968 11961 leader_election.cc:304] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 466dfef8aeae4d6e901e9baf91d0a31f; no voters: 
I20260709 13:06:42.121260 11961 leader_election.cc:290] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:42.121793 11964 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:42.123660 11964 raft_consensus.cc:697] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 1 LEADER]: Becoming Leader. State: Replica: 466dfef8aeae4d6e901e9baf91d0a31f, State: Running, Role: LEADER
I20260709 13:06:42.124571 11964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER }
I20260709 13:06:42.125741 11961 sys_catalog.cc:565] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:42.137142 11966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 466dfef8aeae4d6e901e9baf91d0a31f. Latest consensus state: current_term: 1 leader_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER } }
I20260709 13:06:42.138166 11966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:42.140955 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "466dfef8aeae4d6e901e9baf91d0a31f" member_type: VOTER } }
I20260709 13:06:42.141903 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:42.148085 11972 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:42.156405 11972 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:42.157485   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:42.167796 11972 catalog_manager.cc:1383] Generated new cluster ID: cf8f74420e134906880aa442f80860a4
I20260709 13:06:42.168120 11972 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:42.185058 11972 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:42.186712 11972 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:42.201653 11972 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f: Generated new TSK 0
I20260709 13:06:42.202415 11972 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:42.224923   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:42.232404 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:42.233879 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:42.238173 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:42.241118   431 server_base.cc:1061] running on GCE node
I20260709 13:06:42.242741   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:42.242956   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:42.243104   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602402243091 us; error 0 us; skew 500 ppm
I20260709 13:06:42.243654   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:42.246544   431 webserver.cc:533] Webserver started at http://127.0.107.193:46439/ using document root <none> and password file <none>
I20260709 13:06:42.247064   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:42.247257   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:42.247507   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:42.248759   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "f45847889ca34a7f8924d82d3b68d389"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.254370   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:06:42.258082 11990 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.259063   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:06:42.259446   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "f45847889ca34a7f8924d82d3b68d389"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.259819   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:42.288637   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:42.290593   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:42.292516   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:42.295187   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:42.295434   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.295709   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:42.295905   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.350574   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:37721
I20260709 13:06:42.350684 12052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:37721 every 8 connection(s)
I20260709 13:06:42.356884   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:42.364797 12057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:42.366065 12058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:42.371759 12060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:42.372430   431 server_base.cc:1061] running on GCE node
I20260709 13:06:42.373852   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:42.374177   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:42.374402   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602402374379 us; error 0 us; skew 500 ppm
I20260709 13:06:42.375258   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:42.379879   431 webserver.cc:533] Webserver started at http://127.0.107.194:43519/ using document root <none> and password file <none>
I20260709 13:06:42.380931   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:42.381258   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:42.381464 12053 heartbeater.cc:344] Connected to a master server at 127.0.107.254:45061
I20260709 13:06:42.381748   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:42.382090 12053 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:42.383062 12053 heartbeater.cc:507] Master 127.0.107.254:45061 requested a full tablet report, sending...
I20260709 13:06:42.384284   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "a310834a81a3483a845f1c84c3bbacec"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.386447 11926 ts_manager.cc:194] Registered new tserver with Master: f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721)
I20260709 13:06:42.388979 11926 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43700
I20260709 13:06:42.392851   431 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20260709 13:06:42.398700 12065 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.399572   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.005s	sys 0.000s
I20260709 13:06:42.399940   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "a310834a81a3483a845f1c84c3bbacec"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.400288   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:42.425275   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:42.426821   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:42.428772   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:42.431463   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:42.431710   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.431981   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:42.432159   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.481427   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:39855
I20260709 13:06:42.481617 12127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:39855 every 8 connection(s)
I20260709 13:06:42.487880   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:42.498188 12131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:42.499264 12132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:42.504993 12128 heartbeater.cc:344] Connected to a master server at 127.0.107.254:45061
I20260709 13:06:42.506280 12128 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:42.507629 12128 heartbeater.cc:507] Master 127.0.107.254:45061 requested a full tablet report, sending...
I20260709 13:06:42.508687   431 server_base.cc:1061] running on GCE node
W20260709 13:06:42.508720 12134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:42.510389   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:06:42.510393 11926 ts_manager.cc:194] Registered new tserver with Master: a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
W20260709 13:06:42.511088   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:42.511344   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602402511318 us; error 0 us; skew 500 ppm
I20260709 13:06:42.512225   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:42.512846 11926 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43704
I20260709 13:06:42.515733   431 webserver.cc:533] Webserver started at http://127.0.107.195:38557/ using document root <none> and password file <none>
I20260709 13:06:42.516315   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:42.516485   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:42.516809   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:42.518090   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "512d5a7eec0941429538e0003322920c"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.523139   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:42.527056 12139 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.528287   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 13:06:42.528744   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "512d5a7eec0941429538e0003322920c"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:42.529119   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:42.542538   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:42.543895   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:42.545657   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:42.548614   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:42.548821   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.549180   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:42.549382   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:42.604759   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:36363
I20260709 13:06:42.604830 12201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:36363 every 8 connection(s)
I20260709 13:06:42.622221 12202 heartbeater.cc:344] Connected to a master server at 127.0.107.254:45061
I20260709 13:06:42.622736 12202 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:42.623553 12202 heartbeater.cc:507] Master 127.0.107.254:45061 requested a full tablet report, sending...
I20260709 13:06:42.626143 11926 ts_manager.cc:194] Registered new tserver with Master: 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:42.626372   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017691858s
I20260709 13:06:42.627892 11926 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43708
I20260709 13:06:42.659845 11926 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43718:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:42.663594 11926 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:42.746877 12166 tablet_service.cc:1511] Processing CreateTablet for tablet 541b86a1451b492182d1ea3ff69fefce (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:42.747588 12167 tablet_service.cc:1511] Processing CreateTablet for tablet 9ce6aca95b444ff3a96903e95b9a3d82 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:42.748868 12166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541b86a1451b492182d1ea3ff69fefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.746877 12165 tablet_service.cc:1511] Processing CreateTablet for tablet 066be728742c4959835d7e8ebf88c4b2 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:42.752853 12165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066be728742c4959835d7e8ebf88c4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.753616 12167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ce6aca95b444ff3a96903e95b9a3d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.756376 12016 tablet_service.cc:1511] Processing CreateTablet for tablet 066be728742c4959835d7e8ebf88c4b2 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:42.756655 12015 tablet_service.cc:1511] Processing CreateTablet for tablet 4788b7685eb1429887dfdceb6c878352 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:42.758168 12016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066be728742c4959835d7e8ebf88c4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.759720 12017 tablet_service.cc:1511] Processing CreateTablet for tablet 541b86a1451b492182d1ea3ff69fefce (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:42.763124 12018 tablet_service.cc:1511] Processing CreateTablet for tablet 9ce6aca95b444ff3a96903e95b9a3d82 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:42.765125 12017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541b86a1451b492182d1ea3ff69fefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.769991 12015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4788b7685eb1429887dfdceb6c878352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.772257 12018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ce6aca95b444ff3a96903e95b9a3d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.775259 12164 tablet_service.cc:1511] Processing CreateTablet for tablet 4788b7685eb1429887dfdceb6c878352 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:42.777254 12164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4788b7685eb1429887dfdceb6c878352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.787003 12014 tablet_service.cc:1511] Processing CreateTablet for tablet 631f6e0702e44ae2b74044809ac1f626 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:42.789067 12014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631f6e0702e44ae2b74044809ac1f626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.789842 12163 tablet_service.cc:1511] Processing CreateTablet for tablet 631f6e0702e44ae2b74044809ac1f626 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:42.791383 12163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631f6e0702e44ae2b74044809ac1f626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.791759 12092 tablet_service.cc:1511] Processing CreateTablet for tablet 541b86a1451b492182d1ea3ff69fefce (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 13:06:42.793226 12092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 541b86a1451b492182d1ea3ff69fefce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.793460 12093 tablet_service.cc:1511] Processing CreateTablet for tablet 9ce6aca95b444ff3a96903e95b9a3d82 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 13:06:42.794737 12091 tablet_service.cc:1511] Processing CreateTablet for tablet 066be728742c4959835d7e8ebf88c4b2 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 13:06:42.797024 12090 tablet_service.cc:1511] Processing CreateTablet for tablet 4788b7685eb1429887dfdceb6c878352 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 13:06:42.800015 12089 tablet_service.cc:1511] Processing CreateTablet for tablet 631f6e0702e44ae2b74044809ac1f626 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 13:06:42.802018 12093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ce6aca95b444ff3a96903e95b9a3d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.803857 12091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 066be728742c4959835d7e8ebf88c4b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.806813 12013 tablet_service.cc:1511] Processing CreateTablet for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:42.808413 12013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.810181 12088 tablet_service.cc:1511] Processing CreateTablet for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:42.810665 12162 tablet_service.cc:1511] Processing CreateTablet for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 13:06:42.811956 12088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.812513 12162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94f38a83ab9540849b2d0e4cdf7ca8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.814042 12089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 631f6e0702e44ae2b74044809ac1f626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.816085 12012 tablet_service.cc:1511] Processing CreateTablet for tablet df6d4c1235b947369c3bd6066011ad45 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:42.817795 12090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4788b7685eb1429887dfdceb6c878352. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.820330 12087 tablet_service.cc:1511] Processing CreateTablet for tablet df6d4c1235b947369c3bd6066011ad45 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:42.820793 12167 tablet_service.cc:1511] Processing CreateTablet for tablet df6d4c1235b947369c3bd6066011ad45 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 13:06:42.822190 12087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6d4c1235b947369c3bd6066011ad45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.822377 12167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6d4c1235b947369c3bd6066011ad45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.817831 12012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df6d4c1235b947369c3bd6066011ad45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.833024 12011 tablet_service.cc:1511] Processing CreateTablet for tablet cd365fd2efb64564872c881e039ab8a5 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:42.834690 12011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd365fd2efb64564872c881e039ab8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.839736 12166 tablet_service.cc:1511] Processing CreateTablet for tablet cd365fd2efb64564872c881e039ab8a5 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:42.841209 12166 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd365fd2efb64564872c881e039ab8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.847477 12086 tablet_service.cc:1511] Processing CreateTablet for tablet cd365fd2efb64564872c881e039ab8a5 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 13:06:42.847781 12085 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3f7d158934be8a96735744924a3f9 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:42.849345 12085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3f7d158934be8a96735744924a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.850037 12010 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3f7d158934be8a96735744924a3f9 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:42.851538 12010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3f7d158934be8a96735744924a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.854216 12086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd365fd2efb64564872c881e039ab8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.876058 12161 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3f7d158934be8a96735744924a3f9 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 13:06:42.876680 12009 tablet_service.cc:1511] Processing CreateTablet for tablet 46000a969df248a2be708bd03e39b5b0 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:42.880759 12217 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:42.882536 12084 tablet_service.cc:1511] Processing CreateTablet for tablet 46000a969df248a2be708bd03e39b5b0 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:42.896591 12009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46000a969df248a2be708bd03e39b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.897300 12161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3f7d158934be8a96735744924a3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.905557 12084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46000a969df248a2be708bd03e39b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.938231 12217 tablet_bootstrap.cc:654] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:42.942292 12216 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:42.958752 12163 tablet_service.cc:1511] Processing CreateTablet for tablet 46000a969df248a2be708bd03e39b5b0 (DEFAULT_TABLE table=test-workload [id=e023534f9c1f46259ec77f506e5efd59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 13:06:42.960346 12163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46000a969df248a2be708bd03e39b5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:42.967124 12216 tablet_bootstrap.cc:654] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:42.969630 12217 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:42.984179 12217 ts_tablet_manager.cc:1403] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.104s	user 0.015s	sys 0.024s
I20260709 13:06:42.987735 12217 raft_consensus.cc:359] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:42.988646 12217 raft_consensus.cc:385] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:42.989017 12217 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:42.990058 12217 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:42.993647 12053 heartbeater.cc:499] Master 127.0.107.254:45061 was elected leader, sending a full tablet report...
I20260709 13:06:42.993974 12219 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:42.994176 12217 ts_tablet_manager.cc:1434] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 13:06:42.995633 12217 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.004566 12216 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.005363 12216 ts_tablet_manager.cc:1403] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.063s	user 0.011s	sys 0.013s
I20260709 13:06:43.009331 12217 tablet_bootstrap.cc:654] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.010179 12219 tablet_bootstrap.cc:654] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.013370 12216 raft_consensus.cc:359] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.014379 12216 raft_consensus.cc:385] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.014741 12216 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.016088 12216 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.032238 12202 heartbeater.cc:499] Master 127.0.107.254:45061 was elected leader, sending a full tablet report...
I20260709 13:06:43.033345 12217 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.033493 12219 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.033957 12216 ts_tablet_manager.cc:1434] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.028s	user 0.013s	sys 0.010s
I20260709 13:06:43.034075 12217 ts_tablet_manager.cc:1403] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.039s	user 0.024s	sys 0.007s
I20260709 13:06:43.034166 12219 ts_tablet_manager.cc:1403] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.052s	user 0.019s	sys 0.015s
I20260709 13:06:43.035444 12216 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.037806 12217 raft_consensus.cc:359] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.038558 12217 raft_consensus.cc:385] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.038102 12219 raft_consensus.cc:359] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.038911 12217 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.039119 12219 raft_consensus.cc:385] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.039588 12219 raft_consensus.cc:740] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.040702 12219 consensus_queue.cc:260] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.039822 12217 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.043329 12217 ts_tablet_manager.cc:1434] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 13:06:43.044281 12216 tablet_bootstrap.cc:654] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.046834 12217 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.051276 12128 heartbeater.cc:499] Master 127.0.107.254:45061 was elected leader, sending a full tablet report...
I20260709 13:06:43.051783 12219 ts_tablet_manager.cc:1434] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.009s
I20260709 13:06:43.053200 12219 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.057125 12217 tablet_bootstrap.cc:654] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.061139 12219 tablet_bootstrap.cc:654] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.067041 12216 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.067679 12216 ts_tablet_manager.cc:1403] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.004s
I20260709 13:06:43.070570 12217 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.070770 12216 raft_consensus.cc:359] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.071353 12217 ts_tablet_manager.cc:1403] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.012s
I20260709 13:06:43.071651 12216 raft_consensus.cc:385] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.072057 12216 raft_consensus.cc:740] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.073083 12219 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.072925 12216 consensus_queue.cc:260] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.073804 12219 ts_tablet_manager.cc:1403] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.006s
I20260709 13:06:43.075394 12216 ts_tablet_manager.cc:1434] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 13:06:43.076530 12216 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.076200 12217 raft_consensus.cc:359] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.077203 12217 raft_consensus.cc:385] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.077665 12217 raft_consensus.cc:740] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.077354 12219 raft_consensus.cc:359] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.078256 12219 raft_consensus.cc:385] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.078617 12219 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.078855 12217 consensus_queue.cc:260] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.079757 12219 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.081815 12217 ts_tablet_manager.cc:1434] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 13:06:43.082247 12219 ts_tablet_manager.cc:1434] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 13:06:43.083299 12217 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.083396 12219 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.084132 12216 tablet_bootstrap.cc:654] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.091948 12219 tablet_bootstrap.cc:654] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.091951 12217 tablet_bootstrap.cc:654] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.102454 12224 raft_consensus.cc:493] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.103411 12224 raft_consensus.cc:515] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
W20260709 13:06:43.105939 12054 tablet.cc:2406] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:06:43.107026 12217 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.107028 12219 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.107964 12219 ts_tablet_manager.cc:1403] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.025s	user 0.006s	sys 0.015s
I20260709 13:06:43.107964 12217 ts_tablet_manager.cc:1403] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.012s
I20260709 13:06:43.112018 12217 raft_consensus.cc:359] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.112516 12219 raft_consensus.cc:359] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.113049 12217 raft_consensus.cc:385] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.113436 12219 raft_consensus.cc:385] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.113608 12217 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.113935 12219 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.114540 12224 leader_election.cc:290] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.114964 12219 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.114964 12217 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.119009 12217 ts_tablet_manager.cc:1434] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.003s
I20260709 13:06:43.120756 12217 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.125756 12219 ts_tablet_manager.cc:1434] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.017s	user 0.012s	sys 0.004s
I20260709 13:06:43.127169 12219 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.129631 12217 tablet_bootstrap.cc:654] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.135293 12219 tablet_bootstrap.cc:654] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.144558 12217 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.145489 12217 ts_tablet_manager.cc:1403] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.013s
I20260709 13:06:43.148928 12219 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.149627 12219 ts_tablet_manager.cc:1403] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.023s	user 0.018s	sys 0.003s
I20260709 13:06:43.152340 12217 raft_consensus.cc:359] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.154062 12217 raft_consensus.cc:385] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.154621 12217 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.154115 12219 raft_consensus.cc:359] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.155231 12219 raft_consensus.cc:385] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.155579 12219 raft_consensus.cc:740] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.155776 12217 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.156607 12219 consensus_queue.cc:260] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.158974 12217 ts_tablet_manager.cc:1434] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 13:06:43.159091 12219 ts_tablet_manager.cc:1434] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:43.160298 12217 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.160303 12219 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.166028 12028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.169072 12219 tablet_bootstrap.cc:654] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.169072 12217 tablet_bootstrap.cc:654] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
W20260709 13:06:43.171833 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 13:06:43.169078 12140 leader_election.cc:343] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.183465 12217 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.183465 12219 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.184376 12217 ts_tablet_manager.cc:1403] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.024s	user 0.017s	sys 0.006s
I20260709 13:06:43.184425 12219 ts_tablet_manager.cc:1403] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.012s
I20260709 13:06:43.188159 12219 raft_consensus.cc:359] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.188308 12217 raft_consensus.cc:359] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.189167 12217 raft_consensus.cc:385] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.189167 12219 raft_consensus.cc:385] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.189711 12217 raft_consensus.cc:740] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.189810 12219 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.190939 12217 consensus_queue.cc:260] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.191073 12219 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.194346 12217 ts_tablet_manager.cc:1434] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20260709 13:06:43.194346 12219 ts_tablet_manager.cc:1434] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 13:06:43.195783 12217 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.195811 12219 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.200954 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.202301 12221 raft_consensus.cc:493] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.203065 12221 raft_consensus.cc:515] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.204928 12219 tablet_bootstrap.cc:654] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.202248 12103 raft_consensus.cc:2468] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 0.
I20260709 13:06:43.212482 12221 leader_election.cc:290] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.210039 12140 leader_election.cc:304] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.214159 12224 raft_consensus.cc:2804] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.214689 12224 raft_consensus.cc:493] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.215114 12224 raft_consensus.cc:3060] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.229257 12216 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.229947 12216 ts_tablet_manager.cc:1403] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.154s	user 0.046s	sys 0.098s
I20260709 13:06:43.230760 12217 tablet_bootstrap.cc:654] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.233547 12216 raft_consensus.cc:359] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.234437 12216 raft_consensus.cc:385] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.234722 12216 raft_consensus.cc:740] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.235670 12216 consensus_queue.cc:260] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.238799 12224 raft_consensus.cc:515] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.241482 12224 leader_election.cc:290] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.242690 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.247871 12103 raft_consensus.cc:3060] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.246662 12028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
W20260709 13:06:43.249846 12140 leader_election.cc:343] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.251386 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
W20260709 13:06:43.252687 11991 leader_election.cc:343] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.246590 12219 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.254230 12219 ts_tablet_manager.cc:1403] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.059s	user 0.022s	sys 0.003s
I20260709 13:06:43.254949 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
W20260709 13:06:43.256742 11994 leader_election.cc:343] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.257207 11994 leader_election.cc:304] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:43.258440 12221 raft_consensus.cc:2749] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.258735 12103 raft_consensus.cc:2468] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 1.
I20260709 13:06:43.258612 12219 raft_consensus.cc:359] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.259466 12219 raft_consensus.cc:385] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.259805 12219 raft_consensus.cc:740] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.260937 12217 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.261274 12221 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.260612 12219 consensus_queue.cc:260] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.262460 12140 leader_election.cc:304] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.263036 12232 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.263314 12216 ts_tablet_manager.cc:1434] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.033s	user 0.014s	sys 0.004s
I20260709 13:06:43.263581 12232 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.264470 12216 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.261562 12217 ts_tablet_manager.cc:1403] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.066s	user 0.007s	sys 0.009s
I20260709 13:06:43.266270 12219 ts_tablet_manager.cc:1434] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 13:06:43.267452 12219 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.269423 12217 raft_consensus.cc:359] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.270262 12217 raft_consensus.cc:385] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.270550 12217 raft_consensus.cc:740] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.271776 12216 tablet_bootstrap.cc:654] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.271368 12217 consensus_queue.cc:260] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.273140 12241 raft_consensus.cc:2804] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.273696 12232 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.273823 12241 raft_consensus.cc:697] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 1 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:43.275027 12219 tablet_bootstrap.cc:654] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.275928 12241 consensus_queue.cc:237] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.278282 12244 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.278817 12244 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.284077 12217 ts_tablet_manager.cc:1434] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 13:06:43.286513 12244 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.262019 12221 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.289461 12216 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.290162 12216 ts_tablet_manager.cc:1403] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.007s
I20260709 13:06:43.291122 12217 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.292905 12221 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.293591 12216 raft_consensus.cc:359] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.294533 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.295073 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.294520 12216 raft_consensus.cc:385] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.295795 12176 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
I20260709 13:06:43.296002 12216 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
W20260709 13:06:43.297112 11994 leader_election.cc:343] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.296872 12216 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.298164 11994 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:43.299160 12221 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.299579 12221 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.300099 12221 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.307610 12216 ts_tablet_manager.cc:1434] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 13:06:43.309158 12216 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.314086 12219 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.313163 11924 catalog_manager.cc:5697] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 0 to 1, leader changed from <none> to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 1 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:43.314728 12219 ts_tablet_manager.cc:1403] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.032s
I20260709 13:06:43.316243 12217 tablet_bootstrap.cc:654] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.318676 12219 raft_consensus.cc:359] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.319607 12219 raft_consensus.cc:385] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.320060 12219 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.321475 12219 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.321060 12221 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.323805 12216 tablet_bootstrap.cc:654] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.325635 12219 ts_tablet_manager.cc:1434] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.002s
I20260709 13:06:43.327268 12219 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.327905 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:43.327661 12221 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.328716 12176 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.336118 12219 tablet_bootstrap.cc:654] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.339243 12176 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.341310 11994 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:43.342672 12221 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.343389 12221 raft_consensus.cc:697] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:43.345343 12217 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.344482 12221 consensus_queue.cc:237] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.346202 12217 ts_tablet_manager.cc:1403] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.055s	user 0.017s	sys 0.007s
I20260709 13:06:43.351383 12217 raft_consensus.cc:359] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.352506 12217 raft_consensus.cc:385] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.352887 12217 raft_consensus.cc:740] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.357278 12216 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.360626 12232 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.361402 12232 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.364238 12232 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.363740 12217 consensus_queue.cc:260] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.364957 12216 ts_tablet_manager.cc:1403] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.056s	user 0.011s	sys 0.016s
I20260709 13:06:43.371012 12241 raft_consensus.cc:493] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.370855 12216 raft_consensus.cc:359] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.372115 12216 raft_consensus.cc:385] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.371793 12241 raft_consensus.cc:515] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.372578 12216 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.373011 12219 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.373710 12219 ts_tablet_manager.cc:1403] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.047s	user 0.014s	sys 0.019s
I20260709 13:06:43.373842 12217 ts_tablet_manager.cc:1434] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.003s
I20260709 13:06:43.373857 12216 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.375114 12217 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.375562 12028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.377053 12216 ts_tablet_manager.cc:1434] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:06:43.378430 12216 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.378731 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.379451 12103 raft_consensus.cc:2468] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 0.
I20260709 13:06:43.381094 12241 leader_election.cc:290] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.381880 12140 leader_election.cc:304] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
W20260709 13:06:43.383414 12140 leader_election.cc:343] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.376919 12219 raft_consensus.cc:359] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.384152 12219 raft_consensus.cc:385] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.384536 12241 raft_consensus.cc:2804] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.385022 12241 raft_consensus.cc:493] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.385380 12241 raft_consensus.cc:3060] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.384528 12219 raft_consensus.cc:740] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.386389 12216 tablet_bootstrap.cc:654] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.386539 12219 consensus_queue.cc:260] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.388499 12217 tablet_bootstrap.cc:654] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.391484 12219 ts_tablet_manager.cc:1434] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20260709 13:06:43.392114 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.392742 12219 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.393023 12176 raft_consensus.cc:2393] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a310834a81a3483a845f1c84c3bbacec in current term 1: Already voted for candidate f45847889ca34a7f8924d82d3b68d389 in this term.
I20260709 13:06:43.395516 12232 raft_consensus.cc:493] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.396219 12232 raft_consensus.cc:515] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.396822 12243 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.397687 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.397606 12243 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.398522 12103 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
I20260709 13:06:43.398940 12232 leader_election.cc:290] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.401065 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.403682 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.404330 12176 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 0.
I20260709 13:06:43.405275 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.405550 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.405618 12243 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.406219 12103 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
W20260709 13:06:43.406896 11994 leader_election.cc:343] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:06:43.412650 12069 leader_election.cc:343] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.414001 12069 leader_election.cc:304] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:43.414407 12243 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.415344 12232 raft_consensus.cc:2804] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.415259 12243 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.416049 12232 raft_consensus.cc:493] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.416553 12232 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.423794 12243 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.425140 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
W20260709 13:06:43.426851 11994 leader_election.cc:343] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.426934 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.427627 12216 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.427804 12243 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.427915 12103 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.428270 12216 ts_tablet_manager.cc:1403] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.050s	user 0.011s	sys 0.009s
I20260709 13:06:43.428331 12243 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.432597 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.432492 12216 raft_consensus.cc:359] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.433490 12216 raft_consensus.cc:385] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.433887 12216 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
W20260709 13:06:43.434221 11994 leader_election.cc:343] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.434823 12243 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.436031 12219 tablet_bootstrap.cc:654] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.434927 12216 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.421048 12241 raft_consensus.cc:515] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.441177 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.442476 12102 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
I20260709 13:06:43.443553 12216 ts_tablet_manager.cc:1434] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 13:06:43.443953 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.444610 12102 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
I20260709 13:06:43.444862 12216 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.445796 12028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:43.446888 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.447929 12102 raft_consensus.cc:2468] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
W20260709 13:06:43.448133 12140 leader_election.cc:343] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.451006 12217 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.451581 12217 ts_tablet_manager.cc:1403] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.077s	user 0.007s	sys 0.009s
I20260709 13:06:43.452338 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.453877 11991 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.454063 12102 raft_consensus.cc:3060] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.454712 12241 leader_election.cc:290] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.456538 12027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.457535 12028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.458204 12027 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 0.
I20260709 13:06:43.458316 12217 raft_consensus.cc:359] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.459105 12217 raft_consensus.cc:385] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.459390 12217 raft_consensus.cc:740] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.459411 12066 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec; no voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.456911 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.460443 12217 consensus_queue.cc:260] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.456024 11991 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.461081 12066 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.462466 12221 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.462677 12244 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.462952 12221 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.463162 12244 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.463423 12221 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.463591 12244 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.463950 12103 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.464538 12225 raft_consensus.cc:2749] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.456014 12216 tablet_bootstrap.cc:654] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.460489 12026 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 0.
I20260709 13:06:43.468221 12225 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.468631 12225 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.469862 12217 ts_tablet_manager.cc:1434] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 13:06:43.470836 12217 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:43.468991 12225 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.474211 11991 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.476146 12243 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.476648 12243 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.476735 11991 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.477085 12243 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.488270 12241 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.489190 12241 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.492538 12241 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.494062 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
W20260709 13:06:43.496765 12140 leader_election.cc:343] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.509279 12217 tablet_bootstrap.cc:654] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.501292 12243 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.511507 12259 raft_consensus.cc:493] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.512167 12259 raft_consensus.cc:515] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.513080 12260 raft_consensus.cc:2804] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.514523 12260 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.514912 12260 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.515193 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
W20260709 13:06:43.516906 11994 leader_election.cc:343] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.478169 12232 raft_consensus.cc:515] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.518666 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.519199 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.520022 12176 raft_consensus.cc:2393] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f45847889ca34a7f8924d82d3b68d389 in current term 1: Already voted for candidate 512d5a7eec0941429538e0003322920c in this term.
I20260709 13:06:43.521188 12232 leader_election.cc:290] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
W20260709 13:06:43.521592 12140 leader_election.cc:343] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.522159 12140 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.523051 12241 raft_consensus.cc:2749] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.525024 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:43.525717 12176 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.513867 12243 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.527688 12260 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.531225 12260 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.538654 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
W20260709 13:06:43.541118 11994 leader_election.cc:343] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.542034 12260 raft_consensus.cc:493] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.542657 12260 raft_consensus.cc:515] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.542351 12259 leader_election.cc:290] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 512d5a7eec0941429538e0003322920c (127.0.107.195:36363), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.546260 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
W20260709 13:06:43.547930 11994 leader_election.cc:343] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.548609 12260 leader_election.cc:290] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.552495 12244 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.555759 12244 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.556510 12219 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.557317 12219 ts_tablet_manager.cc:1403] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.165s	user 0.023s	sys 0.023s
I20260709 13:06:43.557577 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:43.558341 12221 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
W20260709 13:06:43.559509 11994 leader_election.cc:343] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.542404 12102 raft_consensus.cc:2468] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 1.
I20260709 13:06:43.548990 11923 catalog_manager.cc:5697] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:43.560914 12066 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:37721, user_credentials={real_user=slave}} blocked reactor thread for 99916.3us
I20260709 13:06:43.563864 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:43.564606 12177 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.562515 12219 raft_consensus.cc:359] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.566123 12219 raft_consensus.cc:385] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.566449 12219 raft_consensus.cc:740] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.566457 12221 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.557647 12176 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:43.568491 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:43.569192 12026 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.553053 12225 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.571782 12219 consensus_queue.cc:260] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.572775 12140 leader_election.cc:304] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.574379 12219 ts_tablet_manager.cc:1434] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 13:06:43.577610 12216 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.577744 12241 raft_consensus.cc:2804] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.578325 12216 ts_tablet_manager.cc:1403] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.134s	user 0.013s	sys 0.012s
I20260709 13:06:43.578404 12241 raft_consensus.cc:697] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:43.579823 12219 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: Bootstrap starting.
I20260709 13:06:43.578424 12069 leader_election.cc:304] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:43.582664 12225 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.579360 12241 consensus_queue.cc:237] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:43.583688 12225 raft_consensus.cc:2804] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.584535 12225 raft_consensus.cc:697] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:43.585445 12225 consensus_queue.cc:237] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.586452 12176 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
W20260709 13:06:43.587922 12069 leader_election.cc:343] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 512d5a7eec0941429538e0003322920c (127.0.107.195:36363): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:06:43.588622 12216 raft_consensus.cc:359] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.591437 12102 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.591753 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.592846 12100 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.599298 12102 raft_consensus.cc:2393] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f45847889ca34a7f8924d82d3b68d389 in current term 1: Already voted for candidate a310834a81a3483a845f1c84c3bbacec in this term.
I20260709 13:06:43.599351 12103 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.599546 12100 raft_consensus.cc:2468] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 0.
I20260709 13:06:43.597223 12219 tablet_bootstrap.cc:654] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.593544 12098 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.601697 12098 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.589493 12216 raft_consensus.cc:385] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.603286 12216 raft_consensus.cc:740] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.592353 12101 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9ce6aca95b444ff3a96903e95b9a3d82" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.593549 12099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.609570 11991 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:43.610610 12099 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.610438 12027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:43.612540 11991 leader_election.cc:304] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.613888 12244 raft_consensus.cc:2804] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:43.614449 12244 raft_consensus.cc:493] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:43.614909 12244 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.623652 12027 raft_consensus.cc:2393] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a310834a81a3483a845f1c84c3bbacec in current term 1: Already voted for candidate f45847889ca34a7f8924d82d3b68d389 in this term.
I20260709 13:06:43.624295 12221 raft_consensus.cc:2749] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.625600 12101 raft_consensus.cc:2393] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f45847889ca34a7f8924d82d3b68d389 in current term 1: Already voted for candidate 512d5a7eec0941429538e0003322920c in this term.
I20260709 13:06:43.624678 11923 catalog_manager.cc:5697] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 0 to 1, leader changed from <none> to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 1 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.627110 12066 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec; no voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.628032 12262 raft_consensus.cc:2749] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.630475 11991 leader_election.cc:304] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:43.619109 12026 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:43.604636 12216 consensus_queue.cc:260] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.638173 12221 raft_consensus.cc:3060] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.639717 12103 raft_consensus.cc:2468] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.640154 12098 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.643082 11991 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:43.643891 12099 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.644232 12260 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.646530 11991 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.647516 12259 raft_consensus.cc:2804] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.647913 12219 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: No bootstrap required, opened a new log
I20260709 13:06:43.647971 12260 raft_consensus.cc:697] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:43.648576 12219 ts_tablet_manager.cc:1403] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: Time spent bootstrapping tablet: real 0.073s	user 0.012s	sys 0.004s
I20260709 13:06:43.649187 12260 consensus_queue.cc:237] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.651572 12219 raft_consensus.cc:359] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.652311 12219 raft_consensus.cc:385] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.652549 12219 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.655799 12219 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.657922 12177 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.658612 12219 ts_tablet_manager.cc:1434] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.002s
I20260709 13:06:43.659228 12244 raft_consensus.cc:515] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.660894 11923 catalog_manager.cc:5697] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 0 to 1, leader changed from <none> to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 1 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.661684 12244 leader_election.cc:290] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.662904 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:43.663587 12177 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.653441 12259 raft_consensus.cc:697] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:43.664023 11991 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c
I20260709 13:06:43.665045 12259 consensus_queue.cc:237] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.666090 12099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:43.667192 12099 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.668447 12244 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.671010 12244 raft_consensus.cc:697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:43.680341 12216 ts_tablet_manager.cc:1434] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.102s	user 0.006s	sys 0.004s
I20260709 13:06:43.681720 12216 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.683624 12221 raft_consensus.cc:2749] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.688388 12217 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: No bootstrap required, opened a new log
I20260709 13:06:43.689605 12217 ts_tablet_manager.cc:1403] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.219s	user 0.022s	sys 0.015s
I20260709 13:06:43.691489 12099 raft_consensus.cc:2468] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.693099 11991 leader_election.cc:304] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:43.692626 11926 catalog_manager.cc:5697] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.695609 12260 raft_consensus.cc:2804] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:43.696273 12260 raft_consensus.cc:697] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:43.697072 12216 tablet_bootstrap.cc:654] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.692914 12217 raft_consensus.cc:359] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.697520 12260 consensus_queue.cc:237] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.698163 12217 raft_consensus.cc:385] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.698585 12217 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.687760 12177 raft_consensus.cc:2468] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:43.694844 11926 catalog_manager.cc:5697] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.699726 12217 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.703408 12217 ts_tablet_manager.cc:1434] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 13:06:43.686570 12244 consensus_queue.cc:237] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.712252 12259 raft_consensus.cc:493] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.712873 12259 raft_consensus.cc:515] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.715458 12259 leader_election.cc:290] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:43.716559 12099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.716763 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "541b86a1451b492182d1ea3ff69fefce" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:43.717751 12099 raft_consensus.cc:2393] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f45847889ca34a7f8924d82d3b68d389 in current term 1: Already voted for candidate 512d5a7eec0941429538e0003322920c in this term.
I20260709 13:06:43.719195 12216 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.719168 11991 leader_election.cc:304] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f45847889ca34a7f8924d82d3b68d389; no voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:43.720084 12216 ts_tablet_manager.cc:1403] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.039s	user 0.020s	sys 0.001s
I20260709 13:06:43.720690 12259 raft_consensus.cc:3060] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.723599 12216 raft_consensus.cc:359] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.724819 12216 raft_consensus.cc:385] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.725193 12216 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.726068 12216 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.729367 12216 ts_tablet_manager.cc:1434] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:06:43.731114 12216 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.730790 11926 catalog_manager.cc:5697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.733244 11926 catalog_manager.cc:5697] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 1, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:43.734197 12259 raft_consensus.cc:2749] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.737273 12216 tablet_bootstrap.cc:654] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.742373 12216 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.742913 12216 ts_tablet_manager.cc:1403] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.012s	user 0.005s	sys 0.006s
I20260709 13:06:43.745627 12216 raft_consensus.cc:359] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.746332 12216 raft_consensus.cc:385] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.746588 12216 raft_consensus.cc:740] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.747231 12216 consensus_queue.cc:260] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.749076 12216 ts_tablet_manager.cc:1434] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:06:43.750003 12216 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: Bootstrap starting.
I20260709 13:06:43.752552 12241 consensus_queue.cc:1048] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:43.753567 12224 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.754199 12224 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.755996 12216 tablet_bootstrap.cc:654] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:43.756309 12224 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.758915 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.761132 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.762009 12103 raft_consensus.cc:2393] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 512d5a7eec0941429538e0003322920c in current term 1: Already voted for candidate f45847889ca34a7f8924d82d3b68d389 in this term.
I20260709 13:06:43.762018 12216 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: No bootstrap required, opened a new log
I20260709 13:06:43.762733 12216 ts_tablet_manager.cc:1403] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 13:06:43.764086 12140 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.765190 12224 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.767264 12216 raft_consensus.cc:359] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.768272 12216 raft_consensus.cc:385] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:43.768612 12216 raft_consensus.cc:740] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Initialized, Role: FOLLOWER
I20260709 13:06:43.769726 12216 consensus_queue.cc:260] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.772044 12216 ts_tablet_manager.cc:1434] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:43.779212 12224 raft_consensus.cc:2749] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.789860 12276 consensus_queue.cc:1048] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:43.903004 12259 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:43.917860 12260 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:43.960254 12276 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:43.960889 12276 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:43.963135 12276 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:43.964143 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:43.964437 12099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:43.965273 12099 raft_consensus.cc:2393] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 512d5a7eec0941429538e0003322920c in current term 1: Already voted for candidate f45847889ca34a7f8924d82d3b68d389 in this term.
I20260709 13:06:43.966276 12140 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:43.967021 12276 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:43.974191 12276 raft_consensus.cc:2749] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:43.975589 12276 consensus_queue.cc:1048] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:43.990123 12276 consensus_queue.cc:1048] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.042354 12276 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:44.042955 12276 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:44.044962 12276 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:44.045320 12225 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:44.045976 12225 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:44.046511 12099 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:44.046511 12026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:44.047750 12099 raft_consensus.cc:2393] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 512d5a7eec0941429538e0003322920c in current term 1: Already voted for candidate a310834a81a3483a845f1c84c3bbacec in this term.
I20260709 13:06:44.047770 12026 raft_consensus.cc:2393] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 512d5a7eec0941429538e0003322920c in current term 1: Already voted for candidate f45847889ca34a7f8924d82d3b68d389 in this term.
I20260709 13:06:44.048934 12027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389" is_pre_election: true
I20260709 13:06:44.049300 12225 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:44.049719 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:44.049981 12027 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 0.
I20260709 13:06:44.049890 12140 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:44.050663 12177 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 0.
I20260709 13:06:44.051208 12276 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:44.051667 12066 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:44.052713 12225 raft_consensus.cc:2804] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:44.053084 12225 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:44.053629 12225 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:44.060261 12276 raft_consensus.cc:2749] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:06:44.061920 12225 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:44.063633 12225 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:44.064745 12027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:44.065119 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:44.065351 12027 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:44.065757 12177 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:44.071563 12177 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:44.071751 12027 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:44.073079 12069 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:44.074350 12225 raft_consensus.cc:2804] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:44.074914 12225 raft_consensus.cc:697] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:44.075841 12225 consensus_queue.cc:237] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:44.085139 11924 catalog_manager.cc:5697] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 0 to 1, leader changed from <none> to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 1 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:44.089226 12260 consensus_queue.cc:1048] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.102236 12260 consensus_queue.cc:1048] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.106437 12260 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:44.120724 12221 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.143266 12225 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:44.155915 12262 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:44.158465 12259 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.164950 12221 consensus_queue.cc:1048] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:06:44.184517 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:44.190811 12221 consensus_queue.cc:1048] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.192930 12244 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:44.463979 12262 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:44.476337 12292 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 13:06:45.192292 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:45.784183 12221 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:45.784806 12221 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:45.786654 12221 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:45.787870 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:45.787873 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:45.789044 12103 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:45.789004 12177 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 1.
I20260709 13:06:45.790709 11991 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:45.791508 12221 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:45.791960 12221 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:45.792405 12221 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:45.798671 12221 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:45.800716 12221 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 2 election: Requested vote from peers a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:45.801859 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:45.801712 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "f45847889ca34a7f8924d82d3b68d389" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:45.802699 12177 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:45.803114 12103 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:45.811471 12103 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 2.
I20260709 13:06:45.811612 12177 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f45847889ca34a7f8924d82d3b68d389 in term 2.
I20260709 13:06:45.812734 11991 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:45.813551 12221 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:45.814514 12221 raft_consensus.cc:697] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 LEADER]: Becoming Leader. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Running, Role: LEADER
I20260709 13:06:45.815312 12221 consensus_queue.cc:237] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:45.823540 11924 catalog_manager.cc:5697] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 reported cstate change: term changed from 0 to 2, leader changed from <none> to f45847889ca34a7f8924d82d3b68d389 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "f45847889ca34a7f8924d82d3b68d389" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:46.119364   431 ts_manager.cc:295] Set tserver state for f45847889ca34a7f8924d82d3b68d389 to MAINTENANCE_MODE
I20260709 13:06:46.120445   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 13:06:46.148926   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:46.149677   431 tablet_replica.cc:333] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.150367   431 raft_consensus.cc:2243] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:46.150872   431 raft_consensus.cc:2272] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.153090   431 tablet_replica.cc:333] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.153730   431 raft_consensus.cc:2243] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:46.154153   431 raft_consensus.cc:2272] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.156139   431 tablet_replica.cc:333] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.156726   431 raft_consensus.cc:2243] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.157341   431 pending_rounds.cc:70] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Trying to abort 1 pending ops.
I20260709 13:06:46.157542   431 pending_rounds.cc:77] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7305635454227243008 op_type: NO_OP noop_request { }
I20260709 13:06:46.157873   431 raft_consensus.cc:2889] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:06:46.158169   431 raft_consensus.cc:2272] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.160315   431 tablet_replica.cc:333] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.160885   431 raft_consensus.cc:2243] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.161899   431 raft_consensus.cc:2272] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.163864   431 tablet_replica.cc:333] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.164431   431 raft_consensus.cc:2243] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.165225   431 raft_consensus.cc:2272] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.167173   431 tablet_replica.cc:333] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.167724   431 raft_consensus.cc:2243] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:46.168154   431 raft_consensus.cc:2272] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.170084   431 tablet_replica.cc:333] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.170622   431 raft_consensus.cc:2243] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.171427   431 raft_consensus.cc:2272] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.173334   431 tablet_replica.cc:333] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.173920   431 raft_consensus.cc:2243] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.174739   431 raft_consensus.cc:2272] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.176604   431 tablet_replica.cc:333] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.177112   431 raft_consensus.cc:2243] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:46.177927   431 raft_consensus.cc:2272] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:46.179857   431 tablet_replica.cc:333] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: stopping tablet replica
I20260709 13:06:46.180354   431 raft_consensus.cc:2243] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:46.180773   431 raft_consensus.cc:2272] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:46.193343 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:46.212286   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:06:46.313316 12066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111) [suppressed 158 similar messages]
W20260709 13:06:46.316948 12066 consensus_peers.cc:597] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:46.334192 12140 consensus_peers.cc:597] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:46.367676 12140 consensus_peers.cc:597] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:46.515441 12066 consensus_peers.cc:597] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:47.075187 12313 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.075837 12313 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.077824 12313 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.079733 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
W20260709 13:06:47.082276 12140 leader_election.cc:336] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.082721 12140 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.083504 12313 raft_consensus.cc:2749] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 13:06:47.194990 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:47.196579 12313 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.197206 12313 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.199115 12313 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.201215 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
W20260709 13:06:47.203481 12140 leader_election.cc:336] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.203858 12140 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.204504 12313 raft_consensus.cc:2749] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:47.219547 12311 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.220357 12311 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.222921 12311 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.224411 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:47.224701 12313 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.225293 12177 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:47.225277 12313 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.226920 12069 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
W20260709 13:06:47.227552 12066 leader_election.cc:336] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.227792 12313 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.228049 12311 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:47.228485 12311 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.228935 12311 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.229387 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
W20260709 13:06:47.232246 12140 leader_election.cc:336] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.232697 12140 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.233464 12313 raft_consensus.cc:2749] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:47.236352 12311 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.238217 12311 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.239393 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:47.239954 12177 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:47.241865 12066 leader_election.cc:336] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.245574 12177 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 2.
I20260709 13:06:47.246694 12069 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.247421 12311 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.247913 12311 raft_consensus.cc:697] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:47.248742 12311 consensus_queue.cc:237] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.255931 11924 catalog_manager.cc:5697] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 1 to 2, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 2 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.259680 12311 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.260263 12311 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.262393 12311 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.263748 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:47.264391 12313 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.264634 12177 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:47.264969 12313 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.266379 12069 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
W20260709 13:06:47.267258 12066 leader_election.cc:336] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.267665 12311 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:47.268296 12311 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.268860 12311 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.269176 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:47.267261 12313 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
W20260709 13:06:47.274936 12140 leader_election.cc:336] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.275460 12140 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512d5a7eec0941429538e0003322920c; no voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.276294 12313 raft_consensus.cc:2749] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:47.279778 12311 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.281590 12311 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.282742 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:47.283310 12177 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:47.285167 12066 leader_election.cc:336] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.289444 12177 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 2.
I20260709 13:06:47.290966 12069 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.291884 12311 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.292413 12311 raft_consensus.cc:697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:47.293159 12311 consensus_queue.cc:237] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.301932 11924 catalog_manager.cc:5697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 1 to 2, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 2 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.309525 12311 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.310146 12311 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.314985 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:47.316097 12177 raft_consensus.cc:2468] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:47.317466 12069 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:47.318748 12225 raft_consensus.cc:2804] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:47.319139 12225 raft_consensus.cc:493] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
W20260709 13:06:47.319478 12066 leader_election.cc:336] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.319528 12225 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.320482 12311 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.326261 12225 raft_consensus.cc:515] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.328368 12225 leader_election.cc:290] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.329713 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "46000a969df248a2be708bd03e39b5b0" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:47.330441 12177 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:47.333037 12066 leader_election.cc:336] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.337868 12177 raft_consensus.cc:2468] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 2.
I20260709 13:06:47.339076 12069 leader_election.cc:304] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.339936 12225 raft_consensus.cc:2804] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.340381 12225 raft_consensus.cc:697] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:47.341367 12225 consensus_queue.cc:237] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.349342 11924 catalog_manager.cc:5697] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 1 to 2, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 2 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.423071 12225 raft_consensus.cc:493] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.423723 12225 raft_consensus.cc:515] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.425629 12225 leader_election.cc:290] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.426913 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
W20260709 13:06:47.430262 12066 leader_election.cc:336] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.430794 12066 leader_election.cc:304] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec; no voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.431749 12225 raft_consensus.cc:2749] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 13:06:47.521004 12225 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.521756 12225 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.524036 12225 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.525254 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c" is_pre_election: true
I20260709 13:06:47.526126 12177 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 1.
I20260709 13:06:47.527233 12069 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
W20260709 13:06:47.528069 12066 leader_election.cc:336] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.528087 12225 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 13:06:47.528707 12225 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Starting leader election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.529050 12225 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.535828 12225 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.537772 12225 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:47.539016 12177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:47.539707 12177 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:47.541478 12066 leader_election.cc:336] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.545560 12177 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 2.
I20260709 13:06:47.546813 12069 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.547569 12225 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.548019 12225 raft_consensus.cc:697] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:47.548923 12225 consensus_queue.cc:237] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.557987 12313 raft_consensus.cc:493] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.558605 12313 raft_consensus.cc:515] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.561041 12313 leader_election.cc:290] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.562662 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:47.563449 12103 raft_consensus.cc:2468] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 1.
I20260709 13:06:47.564572 12140 leader_election.cc:304] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:47.565304 12313 raft_consensus.cc:2804] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 13:06:47.565361 12140 leader_election.cc:336] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.565809 12313 raft_consensus.cc:493] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f45847889ca34a7f8924d82d3b68d389)
I20260709 13:06:47.566272 12313 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:47.566080 11924 catalog_manager.cc:5697] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 1 to 2, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 2 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:47.573787 12313 raft_consensus.cc:515] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.576195 12313 leader_election.cc:290] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.577930 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "df6d4c1235b947369c3bd6066011ad45" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:47.578661 12103 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:47.580343 12140 leader_election.cc:336] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.585597 12103 raft_consensus.cc:2468] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:47.586728 12140 leader_election.cc:304] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.587456 12313 raft_consensus.cc:2804] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:47.588418 12313 raft_consensus.cc:697] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:47.589196 12313 consensus_queue.cc:237] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.597714 11924 catalog_manager.cc:5697] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 1 to 2, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 2 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:47.671728 12066 consensus_peers.cc:597] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:47.770375 12066 consensus_peers.cc:597] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:47.874599 12313 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:47.876437 12313 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.881223 12313 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.887844 12177 raft_consensus.cc:1275] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.889317 12176 raft_consensus.cc:1275] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.890343 12325 consensus_queue.cc:1048] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:47.892621 12311 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:47.893476 12140 leader_election.cc:336] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:47.893926 12177 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.902305 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec" is_pre_election: true
I20260709 13:06:47.903849 12103 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:47.906231 12140 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:47.907389 12313 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 13:06:47.907866 12313 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:47.908244 12313 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:47.907125 12311 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:47.962579 12066 consensus_peers.cc:597] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:47.968981 12313 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:47.971376 12175 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:47.976512 12313 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:47.978029 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:47.981177 12103 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:47.981261 12225 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 13:06:47.988725 12140 leader_election.cc:336] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111)
I20260709 13:06:48.004770 12103 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:48.008064 12140 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: f45847889ca34a7f8924d82d3b68d389
I20260709 13:06:48.010070 12313 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:48.010918 12313 raft_consensus.cc:697] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:48.012578 12313 consensus_queue.cc:237] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
W20260709 13:06:48.019805 12069 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:36363, user_credentials={real_user=slave}} blocked reactor thread for 43839.5us
W20260709 13:06:48.033553 12066 consensus_peers.cc:597] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:48.049134 11924 catalog_manager.cc:5697] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 2 to 3, leader changed from f45847889ca34a7f8924d82d3b68d389 (127.0.107.193) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 3 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:48.138311 12140 consensus_peers.cc:597] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:48.175338 12103 raft_consensus.cc:1275] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:48.176919 12313 consensus_queue.cc:1048] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 13:06:48.196496 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:48.439316 12103 raft_consensus.cc:1275] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 1. (index mismatch)
I20260709 13:06:48.441380 12313 consensus_queue.cc:1048] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 13:06:48.466609 12140 consensus_peers.cc:597] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:48.717330 12140 consensus_peers.cc:597] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:48.803920 12066 consensus_peers.cc:597] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:48.966878 12140 consensus_peers.cc:597] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:49.024179 12066 consensus_peers.cc:597] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:49.197706 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:50.199133 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 13:06:50.314277 12066 consensus_peers.cc:597] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:50.403968 12066 consensus_peers.cc:597] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:50.404382 12066 consensus_peers.cc:597] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:50.597573 12140 consensus_peers.cc:597] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:50.777899 12066 consensus_peers.cc:597] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:50.876156 12140 consensus_peers.cc:597] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 13:06:51.040843 12066 consensus_peers.cc:597] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:51.045008 12066 consensus_peers.cc:597] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 13:06:51.200309 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:51.247727   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:51.256623 12342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:51.257347 12343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:51.258637 12345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:51.262892   431 server_base.cc:1061] running on GCE node
I20260709 13:06:51.264153   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:51.264389   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:51.264624   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602411264602 us; error 0 us; skew 500 ppm
I20260709 13:06:51.265278   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:51.268601   431 webserver.cc:533] Webserver started at http://127.0.107.193:46439/ using document root <none> and password file <none>
I20260709 13:06:51.269198   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:51.269393   431 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 13:06:51.276204   431 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:06:51.281894 12350 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:51.282929   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 13:06:51.283316   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "f45847889ca34a7f8924d82d3b68d389"
format_stamp: "Formatted at 2026-07-09 13:06:42 on dist-test-slave-p8rg"
I20260709 13:06:51.283764   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:51.301906   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:51.303769   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:51.305783   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:51.310382 12357 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 13:06:51.340854 12066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111) [suppressed 99 similar messages]
I20260709 13:06:51.373116   431 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 13:06:51.373687   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.064s	user 0.001s	sys 0.003s
I20260709 13:06:51.374066   431 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 13:06:51.381633 12357 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
W20260709 13:06:51.392988 12140 consensus_peers.cc:597] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:51.402853 12357 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.404068 12357 tablet_bootstrap.cc:492] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.404894 12357 ts_tablet_manager.cc:1403] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.011s
I20260709 13:06:51.408241 12357 raft_consensus.cc:359] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.409039 12357 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.409912 12357 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.413270 12357 ts_tablet_manager.cc:1434] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 13:06:51.414214 12357 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.434900 12357 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.436007 12357 tablet_bootstrap.cc:492] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.436755 12357 ts_tablet_manager.cc:1403] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.006s
I20260709 13:06:51.439832 12357 raft_consensus.cc:359] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:51.440587 12357 raft_consensus.cc:740] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.441366 12357 consensus_queue.cc:260] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } } peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } }
I20260709 13:06:51.443432 12357 ts_tablet_manager.cc:1434] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 13:06:51.444418 12357 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.451575   431 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 13:06:51.452032   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.078s	user 0.075s	sys 0.000s
I20260709 13:06:51.465039 12357 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.466589 12357 tablet_bootstrap.cc:492] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.467859 12357 ts_tablet_manager.cc:1403] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.008s
I20260709 13:06:51.471869 12357 raft_consensus.cc:359] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.473984 12357 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.475364 12357 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.479238 12357 ts_tablet_manager.cc:1434] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.004s
I20260709 13:06:51.481114 12357 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.502599 12357 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.503968 12357 tablet_bootstrap.cc:492] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.504858 12357 ts_tablet_manager.cc:1403] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.024s	user 0.009s	sys 0.010s
I20260709 13:06:51.508545 12357 raft_consensus.cc:359] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.509433 12357 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.510267 12357 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.513990 12357 ts_tablet_manager.cc:1434] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.009s	user 0.010s	sys 0.000s
I20260709 13:06:51.515909 12357 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.533210 12357 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.534296 12357 tablet_bootstrap.cc:492] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.534935 12357 ts_tablet_manager.cc:1403] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20260709 13:06:51.537245 12357 raft_consensus.cc:359] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.537916 12357 raft_consensus.cc:740] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.538743 12357 consensus_queue.cc:260] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.539875   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:37721
I20260709 13:06:51.539996 12419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:37721 every 8 connection(s)
I20260709 13:06:51.540397 12357 ts_tablet_manager.cc:1434] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:51.541265 12357 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.571821 12357 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.576670 12357 tablet_bootstrap.cc:492] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.577929 12357 ts_tablet_manager.cc:1403] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.020s
I20260709 13:06:51.581619   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:51.582202 12357 raft_consensus.cc:359] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.583217 12357 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.584172 12357 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.586395 12357 ts_tablet_manager.cc:1434] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:06:51.587380 12357 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.595911 12420 heartbeater.cc:344] Connected to a master server at 127.0.107.254:45061
I20260709 13:06:51.596467 12420 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:51.597604 12420 heartbeater.cc:507] Master 127.0.107.254:45061 requested a full tablet report, sending...
I20260709 13:06:51.610409 11926 ts_manager.cc:194] Re-registered known tserver with Master: f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721)
I20260709 13:06:51.629798 11926 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40734
I20260709 13:06:51.635435 12420 heartbeater.cc:499] Master 127.0.107.254:45061 was elected leader, sending a full tablet report...
I20260709 13:06:51.635766 12357 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 13:06:51.637207 12357 tablet_bootstrap.cc:492] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.638216 12357 ts_tablet_manager.cc:1403] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.051s	user 0.029s	sys 0.019s
I20260709 13:06:51.641695 12357 raft_consensus.cc:359] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.642671 12357 raft_consensus.cc:740] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.643590 12357 consensus_queue.cc:260] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.645777 12357 ts_tablet_manager.cc:1434] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 13:06:51.646824 12357 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.646808 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:51.647666 12103 raft_consensus.cc:606] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:51.651923 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:51.653031 12103 raft_consensus.cc:606] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:51.654953   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:06:51.655757   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:51.656709   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:51.665259 12357 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.666218 12357 tablet_bootstrap.cc:492] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.666851 12357 ts_tablet_manager.cc:1403] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.008s
I20260709 13:06:51.669407 12357 raft_consensus.cc:359] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.670099 12357 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.670797 12357 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.673087 12357 ts_tablet_manager.cc:1434] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 13:06:51.674067 12357 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.692507 12357 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.693571 12357 tablet_bootstrap.cc:492] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.694224 12357 ts_tablet_manager.cc:1403] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.005s
W20260709 13:06:51.695536 12140 consensus_peers.cc:597] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 13:06:51.696424 12357 raft_consensus.cc:359] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.697029 12357 raft_consensus.cc:740] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.697726 12357 consensus_queue.cc:260] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.699646 12357 ts_tablet_manager.cc:1434] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 13:06:51.700830 12357 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Bootstrap starting.
I20260709 13:06:51.715469 12357 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 13:06:51.716326 12357 tablet_bootstrap.cc:492] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Bootstrap complete.
I20260709 13:06:51.717000 12357 ts_tablet_manager.cc:1403] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Time spent bootstrapping tablet: real 0.016s	user 0.008s	sys 0.005s
I20260709 13:06:51.719221 12357 raft_consensus.cc:359] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.719754 12357 raft_consensus.cc:740] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45847889ca34a7f8924d82d3b68d389, State: Initialized, Role: FOLLOWER
I20260709 13:06:51.720417 12357 consensus_queue.cc:260] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.722004 12357 ts_tablet_manager.cc:1434] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.006s
I20260709 13:06:51.768445 12393 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:51.778854 12393 pending_rounds.cc:85] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Aborting all ops after (but not including) 0
I20260709 13:06:51.779219 12393 pending_rounds.cc:107] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20260709 13:06:51.779533 12393 raft_consensus.cc:2889] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 13:06:51.786908 12393 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.880672 12437 raft_consensus.cc:993] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:51.881460 12436 raft_consensus.cc:1081] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:51.882408 12393 raft_consensus.cc:3060] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.883780 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:51.884517 12175 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:51.885037 12175 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:51.895645 12175 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.897869 12175 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:51.898911 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:51.899152 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:51.899731 12393 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 3
I20260709 13:06:51.899813 12103 raft_consensus.cc:3055] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:51.900269 12103 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:51.901258 12103 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.902550 12103 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:51.907542 12393 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:51.908891 12140 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:51.910044 12330 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:51.910769 12103 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:51.911353 12330 raft_consensus.cc:697] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:51.912642 12330 consensus_queue.cc:237] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:51.923810 11926 catalog_manager.cc:5697] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 2 to 3, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 3 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:51.962451 12393 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:51.989219 12437 raft_consensus.cc:993] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:51.989919 12436 raft_consensus.cc:1081] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:51.992072 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:51.992594 12175 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:51.992868 12175 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:52.000412 12175 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.002542 12175 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:52.003636 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:52.003705 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:52.004400 12393 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:52.004460 12103 raft_consensus.cc:3055] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:52.004802 12103 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:52.005752 12103 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.006826 12103 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:52.011687 12393 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:52.013206 12140 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:52.014093 12330 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:52.014668 12330 raft_consensus.cc:697] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:52.016070 12103 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:52.015694 12330 consensus_queue.cc:237] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.024083 11926 catalog_manager.cc:5697] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 2 to 3, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 3 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:52.137542 12393 raft_consensus.cc:3060] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
W20260709 13:06:52.201691 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:52.408641 12393 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:52.410279 12448 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:52.422744 12103 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:52.424872 12448 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:52.454833 12103 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:52.456233 12448 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:52.465914 12394 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:52.467751 12330 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:52.667366   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:52.687844 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:52.688632 12175 raft_consensus.cc:606] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:52.692444 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:52.693071 12175 raft_consensus.cc:606] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:52.695875 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:52.696405 12103 raft_consensus.cc:606] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:52.697710   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:06:52.698341   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:52.699014   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:52.888319 12439 raft_consensus.cc:993] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:52.889006 12330 raft_consensus.cc:1081] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:52.891086 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:52.891705 12103 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:52.891979 12103 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:52.898773 12103 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.900393 12103 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:52.901618 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:52.901790 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:52.902412 12175 raft_consensus.cc:3055] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:52.902537 12393 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:52.902768 12175 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:52.903714 12175 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.904778 12175 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:52.911857 12393 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:52.913115 12175 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:52.913462 12066 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:52.914374 12436 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:52.914808 12436 raft_consensus.cc:697] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:52.915581 12436 consensus_queue.cc:237] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:52.924332 11926 catalog_manager.cc:5697] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 3 to 4, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 4 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:53.023207 12439 raft_consensus.cc:993] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:53.023758 12330 raft_consensus.cc:1081] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:53.025810 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:53.026379 12103 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:53.026705 12103 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:53.033758 12103 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.036060 12103 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:53.037266 12393 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:53.037307 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:53.038102 12393 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:53.038141 12175 raft_consensus.cc:3055] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:53.038616 12175 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:53.039507 12175 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.040553 12175 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:53.044793 12393 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:53.046258 12066 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:53.046871 12175 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:53.047258 12436 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:53.047907 12436 raft_consensus.cc:697] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:53.049124 12436 consensus_queue.cc:237] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.056555 11926 catalog_manager.cc:5697] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 3 to 4, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 4 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:53.105013 12341 raft_consensus.cc:993] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:53.105666 12436 raft_consensus.cc:1081] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:53.107527 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:53.108108 12175 raft_consensus.cc:493] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:53.108424 12175 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:53.115854 12175 raft_consensus.cc:515] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.118363 12175 leader_election.cc:290] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:53.118813 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:53.119657 12394 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:53.121193 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:53.121897 12103 raft_consensus.cc:3055] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:53.122227 12103 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:53.123077 12103 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.124466 12103 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:53.127157 12394 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:53.128563 12140 leader_election.cc:304] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:53.129323 12330 raft_consensus.cc:2804] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:53.129806 12330 raft_consensus.cc:697] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:53.130164 12103 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:53.130983 12330 consensus_queue.cc:237] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.139837 11926 catalog_manager.cc:5697] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 1 to 2, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 2 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:53.203039 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:53.368903 12175 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:53.370405 12436 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:53.381848 12394 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:53.384739 12341 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:53.568102 12394 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:53.569679 12341 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:06:53.580550 12175 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:53.582955 12461 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 13:06:53.657142 12103 raft_consensus.cc:1275] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:53.658921 12330 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:53.674065 12394 raft_consensus.cc:1275] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:53.675849 12468 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:53.707202   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:53.715337 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:53.716044 12103 raft_consensus.cc:606] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:53.719206 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:53.719731 12103 raft_consensus.cc:606] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:53.722687 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:53.723446 12175 raft_consensus.cc:606] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:53.724663   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:06:53.725236   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:53.725860   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:53.867391 12461 raft_consensus.cc:993] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:53.868057 12436 raft_consensus.cc:1081] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:53.869761 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:53.870322 12175 raft_consensus.cc:493] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:53.870678 12175 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:53.878041 12175 raft_consensus.cc:515] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.879946 12175 leader_election.cc:290] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 5 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:53.880970 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:53.880782 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "631f6e0702e44ae2b74044809ac1f626" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:53.881702 12394 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:53.881709 12103 raft_consensus.cc:3055] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:06:53.882263 12103 raft_consensus.cc:740] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:53.883248 12103 consensus_queue.cc:260] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.884889 12103 raft_consensus.cc:3060] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:53.888366 12394 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 5.
I20260709 13:06:53.889627 12140 leader_election.cc:304] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:53.890352 12468 raft_consensus.cc:2804] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:06:53.890786 12468 raft_consensus.cc:697] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 5 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:53.891641 12468 consensus_queue.cc:237] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:53.891935 12103 raft_consensus.cc:2468] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 5.
I20260709 13:06:53.898118 11924 catalog_manager.cc:5697] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 4 to 5, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 5 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:54.148988 12461 raft_consensus.cc:993] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:54.149631 12436 raft_consensus.cc:1081] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:54.151199 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:54.151832 12175 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:54.152245 12175 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:54.159921 12175 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.161720 12175 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:54.162642 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:54.162848 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:54.163296 12394 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:54.163415 12103 raft_consensus.cc:3055] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:54.163889 12103 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:54.164705 12103 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.165871 12103 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:54.169440 12394 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:54.170800 12140 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:54.171794 12468 raft_consensus.cc:2804] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:54.172286 12468 raft_consensus.cc:697] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:54.173337 12103 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 2.
I20260709 13:06:54.173375 12468 consensus_queue.cc:237] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.181721 11924 catalog_manager.cc:5697] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 1 to 2, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 2 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
W20260709 13:06:54.204178 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:54.205916 12439 raft_consensus.cc:993] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:54.206580 12439 raft_consensus.cc:1081] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:54.208178 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:54.208726 12103 raft_consensus.cc:493] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:54.209143 12103 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.216861 12103 raft_consensus.cc:515] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.218713 12103 leader_election.cc:290] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:54.219851 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:54.219897 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cd365fd2efb64564872c881e039ab8a5" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:54.220580 12394 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.220691 12175 raft_consensus.cc:3055] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:54.221134 12175 raft_consensus.cc:740] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:54.221904 12175 consensus_queue.cc:260] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.222980 12175 raft_consensus.cc:3060] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.227249 12394 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 3.
I20260709 13:06:54.228792 12066 leader_election.cc:304] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:54.229429 12175 raft_consensus.cc:2468] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 3.
I20260709 13:06:54.229830 12436 raft_consensus.cc:2804] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:54.230471 12436 raft_consensus.cc:697] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:54.231462 12436 consensus_queue.cc:237] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.238646 11924 catalog_manager.cc:5697] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 2 to 3, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 3 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:54.406914 12394 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:54.408447 12439 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:06:54.420275 12103 raft_consensus.cc:1275] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:54.422544 12439 consensus_queue.cc:1048] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:06:54.580420 12103 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:54.582613 12439 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:54.598621 12394 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:06:54.601572 12439 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:06:54.664526 12394 raft_consensus.cc:1275] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:54.666441 12436 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:54.678232 12175 raft_consensus.cc:1275] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:54.681203 12436 consensus_queue.cc:1048] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:54.733120   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:54.741197 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:54.741801 12175 raft_consensus.cc:606] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:54.744309 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:54.744774 12175 raft_consensus.cc:606] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:54.747205 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:54.747623 12103 raft_consensus.cc:606] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:54.748675   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:06:54.749205   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:54.749876   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:54.857101 12468 raft_consensus.cc:993] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:54.857622 12439 raft_consensus.cc:1081] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:54.859006 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:54.859505 12103 raft_consensus.cc:493] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:54.859797 12103 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:54.864997 12103 raft_consensus.cc:515] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.866714 12103 leader_election.cc:290] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:54.867578 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:54.867923 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3f7d158934be8a96735744924a3f9" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 1 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:54.868294 12394 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:54.868525 12175 raft_consensus.cc:3055] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:54.868876 12175 raft_consensus.cc:740] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:54.869724 12175 consensus_queue.cc:260] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.870802 12175 raft_consensus.cc:3060] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:54.875149 12394 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:54.876331 12066 leader_election.cc:304] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:54.876909 12175 raft_consensus.cc:2468] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:54.877013 12436 raft_consensus.cc:2804] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:54.877696 12436 raft_consensus.cc:697] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:54.878659 12436 consensus_queue.cc:237] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.885634 11926 catalog_manager.cc:5697] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 3 to 4, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 4 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:54.935532 12473 raft_consensus.cc:993] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:54.936266 12436 raft_consensus.cc:1081] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:54.937935 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:54.938436 12175 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:54.938716 12175 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.944070 12175 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.945597 12175 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:54.946676 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:54.946959 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:54.947455 12394 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.947667 12103 raft_consensus.cc:3055] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:54.948081 12103 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:54.948808 12103 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.949990 12103 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:54.954524 12394 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:54.955852 12140 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:54.956591 12439 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:54.957293 12439 raft_consensus.cc:697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:54.957494 12103 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 3.
I20260709 13:06:54.958596 12439 consensus_queue.cc:237] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:54.966233 11926 catalog_manager.cc:5697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 2 to 3, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 3 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:55.121639 12468 raft_consensus.cc:993] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:55.122159 12439 raft_consensus.cc:1081] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:55.123776 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:55.124322 12103 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:55.124657 12103 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.131104 12103 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:55.132956 12103 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 3 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:55.134029 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:55.134068 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:55.134696 12175 raft_consensus.cc:3055] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:06:55.134889 12394 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.135051 12175 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:55.135816 12175 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:55.136849 12175 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:06:55.142000 12175 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 3.
I20260709 13:06:55.142813 12394 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 3.
I20260709 13:06:55.143283 12069 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, a310834a81a3483a845f1c84c3bbacec; no voters: 
I20260709 13:06:55.144225 12436 raft_consensus.cc:2804] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:06:55.144666 12436 raft_consensus.cc:697] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:55.145470 12436 consensus_queue.cc:237] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:55.152045 11926 catalog_manager.cc:5697] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 2 to 3, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 3 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:55.205281 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:55.280798 12394 raft_consensus.cc:1275] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:06:55.282392 12436 consensus_queue.cc:1048] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:55.286319 12175 raft_consensus.cc:1275] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 13:06:55.287951 12436 consensus_queue.cc:1048] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:06:55.363431 12103 raft_consensus.cc:1275] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.364899 12439 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:55.374682 12394 raft_consensus.cc:1275] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.376324 12468 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:55.605388 12175 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.606890 12436 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:06:55.616998 12394 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:06:55.619032 12436 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:06:55.756764   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:55.765308 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:55.765978 12103 raft_consensus.cc:606] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:55.768544 12103 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
mode: GRACEFUL
new_leader_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:49876
I20260709 13:06:55.769007 12103 raft_consensus.cc:606] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Received request to transfer leadership to TS 512d5a7eec0941429538e0003322920c
I20260709 13:06:55.771469 12175 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc"
dest_uuid: "512d5a7eec0941429538e0003322920c"
mode: GRACEFUL
new_leader_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:48184
I20260709 13:06:55.771950 12175 raft_consensus.cc:606] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Received request to transfer leadership to TS a310834a81a3483a845f1c84c3bbacec
I20260709 13:06:55.773262   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 13:06:55.773885   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:06:55.774471   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:56.119045 12436 raft_consensus.cc:993] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:56.119663 12503 raft_consensus.cc:1081] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:56.121574 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:56.122145 12175 raft_consensus.cc:493] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:56.122541 12175 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.129769 12175 raft_consensus.cc:515] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.131651 12175 leader_election.cc:290] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 4 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:56.132689 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:56.132505 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4788b7685eb1429887dfdceb6c878352" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:56.133320 12103 raft_consensus.cc:3055] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:56.133549 12394 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.133692 12103 raft_consensus.cc:740] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:56.134596 12103 consensus_queue.cc:260] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.135571 12103 raft_consensus.cc:3060] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.141804 12394 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 4.
I20260709 13:06:56.141913 12103 raft_consensus.cc:2468] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 4.
I20260709 13:06:56.143036 12140 leader_election.cc:304] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:56.143738 12501 raft_consensus.cc:2804] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:56.144213 12501 raft_consensus.cc:697] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 4 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:56.145008 12501 consensus_queue.cc:237] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.151820 11924 catalog_manager.cc:5697] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 3 to 4, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 4 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.189695 12436 raft_consensus.cc:993] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: : Instructing follower 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:56.190240 12503 raft_consensus.cc:1081] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Signalling peer 512d5a7eec0941429538e0003322920c to start an election
I20260709 13:06:56.191809 12175 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2"
dest_uuid: "512d5a7eec0941429538e0003322920c"
 from {username='slave'} at 127.0.0.1:40034
I20260709 13:06:56.192473 12175 raft_consensus.cc:493] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:56.192828 12175 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:56.199288 12175 raft_consensus.cc:515] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.200883 12175 leader_election.cc:290] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 5 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855)
I20260709 13:06:56.201813 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:56.202159 12103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "066be728742c4959835d7e8ebf88c4b2" candidate_uuid: "512d5a7eec0941429538e0003322920c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
I20260709 13:06:56.202610 12394 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Advancing to term 5
I20260709 13:06:56.202798 12103 raft_consensus.cc:3055] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Stepping down as leader of term 4
I20260709 13:06:56.203223 12103 raft_consensus.cc:740] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:56.203897 12103 consensus_queue.cc:260] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.204926 12103 raft_consensus.cc:3060] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Advancing to term 5
W20260709 13:06:56.206642 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:56.209424 12394 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 5.
I20260709 13:06:56.210831 12140 leader_election.cc:304] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512d5a7eec0941429538e0003322920c, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:56.211102 12103 raft_consensus.cc:2468] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 512d5a7eec0941429538e0003322920c in term 5.
I20260709 13:06:56.211609 12501 raft_consensus.cc:2804] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Leader election won for term 5
I20260709 13:06:56.212211 12501 raft_consensus.cc:697] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 5 LEADER]: Becoming Leader. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:56.213174 12501 consensus_queue.cc:237] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.220957 11924 catalog_manager.cc:5697] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c reported cstate change: term changed from 4 to 5, leader changed from a310834a81a3483a845f1c84c3bbacec (127.0.107.194) to 512d5a7eec0941429538e0003322920c (127.0.107.195). New cstate: current_term: 5 leader_uuid: "512d5a7eec0941429538e0003322920c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:56.292513 12468 raft_consensus.cc:993] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: : Instructing follower a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:56.293022 12501 raft_consensus.cc:1081] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Signalling peer a310834a81a3483a845f1c84c3bbacec to start an election
I20260709 13:06:56.294783 12103 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc"
dest_uuid: "a310834a81a3483a845f1c84c3bbacec"
 from {username='slave'} at 127.0.0.1:36978
I20260709 13:06:56.295261 12103 raft_consensus.cc:493] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:56.295503 12103 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.300930 12103 raft_consensus.cc:515] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.303140 12103 leader_election.cc:290] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Requested vote from peers f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721), 512d5a7eec0941429538e0003322920c (127.0.107.195:36363)
I20260709 13:06:56.303920 12394 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f45847889ca34a7f8924d82d3b68d389"
I20260709 13:06:56.304162 12175 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94f38a83ab9540849b2d0e4cdf7ca8cc" candidate_uuid: "a310834a81a3483a845f1c84c3bbacec" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "512d5a7eec0941429538e0003322920c"
I20260709 13:06:56.304654 12394 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.304715 12175 raft_consensus.cc:3055] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Stepping down as leader of term 3
I20260709 13:06:56.305137 12175 raft_consensus.cc:740] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 512d5a7eec0941429538e0003322920c, State: Running, Role: LEADER
I20260709 13:06:56.306005 12175 consensus_queue.cc:260] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.307261 12175 raft_consensus.cc:3060] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Advancing to term 4
I20260709 13:06:56.312837 12394 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:56.314425 12066 leader_election.cc:304] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a310834a81a3483a845f1c84c3bbacec, f45847889ca34a7f8924d82d3b68d389; no voters: 
I20260709 13:06:56.314971 12175 raft_consensus.cc:2468] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a310834a81a3483a845f1c84c3bbacec in term 4.
I20260709 13:06:56.315196 12436 raft_consensus.cc:2804] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Leader election won for term 4
I20260709 13:06:56.315820 12436 raft_consensus.cc:697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Becoming Leader. State: Replica: a310834a81a3483a845f1c84c3bbacec, State: Running, Role: LEADER
I20260709 13:06:56.316712 12436 consensus_queue.cc:237] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } }
I20260709 13:06:56.323598 11924 catalog_manager.cc:5697] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec reported cstate change: term changed from 3 to 4, leader changed from 512d5a7eec0941429538e0003322920c (127.0.107.195) to a310834a81a3483a845f1c84c3bbacec (127.0.107.194). New cstate: current_term: 4 leader_uuid: "a310834a81a3483a845f1c84c3bbacec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:56.635250 12394 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:56.636581 12501 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:56.651093 12103 raft_consensus.cc:1275] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:56.652889 12501 consensus_queue.cc:1048] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:06:56.695652 12103 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:56.696816 12501 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a310834a81a3483a845f1c84c3bbacec" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 13:06:56.707734 12394 raft_consensus.cc:1275] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Refusing update from remote peer 512d5a7eec0941429538e0003322920c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 13:06:56.710180 12511 consensus_queue.cc:1048] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f45847889ca34a7f8924d82d3b68d389" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 37721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 13:06:56.780783   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:06:56.816989   431 tablet_server.cc:179] TabletServer@127.0.107.193:37721 shutting down...
I20260709 13:06:56.835119 12175 raft_consensus.cc:1275] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Refusing update from remote peer a310834a81a3483a845f1c84c3bbacec: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 13:06:56.837445 12436 consensus_queue.cc:1048] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [LEADER]: Connected to new peer: Peer: permanent_uuid: "512d5a7eec0941429538e0003322920c" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 36363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 13:06:56.862262 12066 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
I20260709 13:06:56.866307   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:56.868363   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.869549   431 raft_consensus.cc:2243] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.870350   431 raft_consensus.cc:2272] T 66d3f7d158934be8a96735744924a3f9 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:56.870838 12066 consensus_peers.cc:597] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.875860   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.876754   431 raft_consensus.cc:2243] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.877358   431 raft_consensus.cc:2272] T cd365fd2efb64564872c881e039ab8a5 P f45847889ca34a7f8924d82d3b68d389 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.879860   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.880460   431 raft_consensus.cc:2243] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.880919   431 raft_consensus.cc:2272] T 4788b7685eb1429887dfdceb6c878352 P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.883982   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.884584   431 raft_consensus.cc:2243] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.885036   431 raft_consensus.cc:2272] T df6d4c1235b947369c3bd6066011ad45 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.887334   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.888060   431 raft_consensus.cc:2243] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.888737   431 raft_consensus.cc:2272] T 631f6e0702e44ae2b74044809ac1f626 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.890798   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.891355   431 raft_consensus.cc:2243] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.891747   431 raft_consensus.cc:2272] T 9ce6aca95b444ff3a96903e95b9a3d82 P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.894091   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.894676   431 raft_consensus.cc:2243] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.895356   431 raft_consensus.cc:2272] T 541b86a1451b492182d1ea3ff69fefce P f45847889ca34a7f8924d82d3b68d389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.897979   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.898519   431 raft_consensus.cc:2243] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.899089   431 raft_consensus.cc:2272] T 94f38a83ab9540849b2d0e4cdf7ca8cc P f45847889ca34a7f8924d82d3b68d389 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.901144   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.901706   431 raft_consensus.cc:2243] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.902196   431 raft_consensus.cc:2272] T 46000a969df248a2be708bd03e39b5b0 P f45847889ca34a7f8924d82d3b68d389 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.904382   431 tablet_replica.cc:333] stopping tablet replica
I20260709 13:06:56.905058   431 raft_consensus.cc:2243] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:56.905566   431 raft_consensus.cc:2272] T 066be728742c4959835d7e8ebf88c4b2 P f45847889ca34a7f8924d82d3b68d389 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:56.938073   431 tablet_server.cc:196] TabletServer@127.0.107.193:37721 shutdown complete.
I20260709 13:06:56.964563   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:06:56.977015 12140 consensus_peers.cc:597] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:56.995906   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:56.996876   431 tablet_replica.cc:333] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:56.997788   431 raft_consensus.cc:2243] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:06:56.998773   431 raft_consensus.cc:2272] T cd365fd2efb64564872c881e039ab8a5 P a310834a81a3483a845f1c84c3bbacec [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.001756   431 tablet_replica.cc:333] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.002524   431 raft_consensus.cc:2243] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.003088   431 raft_consensus.cc:2272] T df6d4c1235b947369c3bd6066011ad45 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.006387   431 tablet_replica.cc:333] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
W20260709 13:06:57.006958 12140 consensus_peers.cc:597] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c -> Peer a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855): Couldn't send request to peer a310834a81a3483a845f1c84c3bbacec. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:57.007098   431 raft_consensus.cc:2243] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.007705   431 raft_consensus.cc:2272] T 631f6e0702e44ae2b74044809ac1f626 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.010267   431 tablet_replica.cc:333] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.011341   431 raft_consensus.cc:2243] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.012427   431 raft_consensus.cc:2272] T 46000a969df248a2be708bd03e39b5b0 P a310834a81a3483a845f1c84c3bbacec [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.015326   431 tablet_replica.cc:333] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.016037   431 raft_consensus.cc:2243] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.016614   431 raft_consensus.cc:2272] T 541b86a1451b492182d1ea3ff69fefce P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.019830   431 tablet_replica.cc:333] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.020543   431 raft_consensus.cc:2243] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.021345   431 raft_consensus.cc:2272] T 066be728742c4959835d7e8ebf88c4b2 P a310834a81a3483a845f1c84c3bbacec [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.024907   431 tablet_replica.cc:333] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.025740   431 raft_consensus.cc:2243] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.026736   431 raft_consensus.cc:2272] T 94f38a83ab9540849b2d0e4cdf7ca8cc P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.030345   431 tablet_replica.cc:333] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.031136   431 raft_consensus.cc:2243] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.031649   431 raft_consensus.cc:2272] T 4788b7685eb1429887dfdceb6c878352 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.034657   431 tablet_replica.cc:333] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.035208   431 raft_consensus.cc:2243] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.035630   431 raft_consensus.cc:2272] T 9ce6aca95b444ff3a96903e95b9a3d82 P a310834a81a3483a845f1c84c3bbacec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.038024   431 tablet_replica.cc:333] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec: stopping tablet replica
I20260709 13:06:57.038504   431 raft_consensus.cc:2243] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.039393   431 raft_consensus.cc:2272] T 66d3f7d158934be8a96735744924a3f9 P a310834a81a3483a845f1c84c3bbacec [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:57.059491 12140 consensus_peers.cc:597] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:57.077011   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
W20260709 13:06:57.094146 12140 consensus_peers.cc:597] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c -> Peer a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855): Couldn't send request to peer a310834a81a3483a845f1c84c3bbacec. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:39855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:57.095175 12140 consensus_peers.cc:597] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:06:57.114493 12140 consensus_peers.cc:597] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c -> Peer f45847889ca34a7f8924d82d3b68d389 (127.0.107.193:37721): Couldn't send request to peer f45847889ca34a7f8924d82d3b68d389. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:57.118427   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
I20260709 13:06:57.155833   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:06:57.156615   431 tablet_replica.cc:333] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.157457   431 raft_consensus.cc:2243] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.158468   431 raft_consensus.cc:2272] T 66d3f7d158934be8a96735744924a3f9 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.161244   431 tablet_replica.cc:333] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.162065   431 raft_consensus.cc:2243] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 4 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.163357   431 raft_consensus.cc:2272] T 4788b7685eb1429887dfdceb6c878352 P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.167590   431 tablet_replica.cc:333] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
W20260709 13:06:57.167632 12140 consensus_peers.cc:597] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c -> Peer a310834a81a3483a845f1c84c3bbacec (127.0.107.194:39855): Couldn't send request to peer a310834a81a3483a845f1c84c3bbacec. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:39855: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:06:57.168613   431 raft_consensus.cc:2243] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.169772   431 raft_consensus.cc:2272] T df6d4c1235b947369c3bd6066011ad45 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.173290   431 tablet_replica.cc:333] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.174070   431 raft_consensus.cc:2243] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.175202   431 raft_consensus.cc:2272] T 631f6e0702e44ae2b74044809ac1f626 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.177776   431 tablet_replica.cc:333] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.178445   431 raft_consensus.cc:2243] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.178912   431 raft_consensus.cc:2272] T cd365fd2efb64564872c881e039ab8a5 P 512d5a7eec0941429538e0003322920c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.181363   431 tablet_replica.cc:333] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.181981   431 raft_consensus.cc:2243] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.182865   431 raft_consensus.cc:2272] T 9ce6aca95b444ff3a96903e95b9a3d82 P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.185245   431 tablet_replica.cc:333] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.185860   431 raft_consensus.cc:2243] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.186337   431 raft_consensus.cc:2272] T 46000a969df248a2be708bd03e39b5b0 P 512d5a7eec0941429538e0003322920c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.188678   431 tablet_replica.cc:333] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.189272   431 raft_consensus.cc:2243] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 13:06:57.189751   431 raft_consensus.cc:2272] T 94f38a83ab9540849b2d0e4cdf7ca8cc P 512d5a7eec0941429538e0003322920c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.193255   431 tablet_replica.cc:333] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.193867   431 raft_consensus.cc:2243] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.194782   431 raft_consensus.cc:2272] T 541b86a1451b492182d1ea3ff69fefce P 512d5a7eec0941429538e0003322920c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.197117   431 tablet_replica.cc:333] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c: stopping tablet replica
I20260709 13:06:57.197666   431 raft_consensus.cc:2243] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 5 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.198814   431 raft_consensus.cc:2272] T 066be728742c4959835d7e8ebf88c4b2 P 512d5a7eec0941429538e0003322920c [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 13:06:57.208168 11974 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 13:06:57.235734   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:06:57.263514   431 master.cc:562] Master@127.0.107.254:45061 shutting down...
I20260709 13:06:57.298772   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:06:57.299451   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:06:57.299777   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 466dfef8aeae4d6e901e9baf91d0a31f: stopping tablet replica
I20260709 13:06:57.323848   431 master.cc:584] Master@127.0.107.254:45061 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (15402 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 13:06:57.373379   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:37625
I20260709 13:06:57.374908   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:57.380662 12526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.381228 12527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.382436 12529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:57.383647   431 server_base.cc:1061] running on GCE node
I20260709 13:06:57.384784   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:57.384971   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:57.385157   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602417385141 us; error 0 us; skew 500 ppm
I20260709 13:06:57.386190   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:57.388842   431 webserver.cc:533] Webserver started at http://127.0.107.254:46693/ using document root <none> and password file <none>
I20260709 13:06:57.389403   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:57.389662   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:57.389961   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:57.391170   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "2ed37b04e9944d74ba43014e36c4791c"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.396186   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:06:57.400110 12534 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.400964   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260709 13:06:57.401368   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "2ed37b04e9944d74ba43014e36c4791c"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.401717   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:57.419412   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:57.420919   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:57.466601   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:37625
I20260709 13:06:57.466717 12585 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:37625 every 8 connection(s)
I20260709 13:06:57.471307 12586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:57.485363 12586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c: Bootstrap starting.
I20260709 13:06:57.491452 12586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:57.496376 12586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c: No bootstrap required, opened a new log
I20260709 13:06:57.498970 12586 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER }
I20260709 13:06:57.499571 12586 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:57.499917 12586 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ed37b04e9944d74ba43014e36c4791c, State: Initialized, Role: FOLLOWER
I20260709 13:06:57.501245 12586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER }
I20260709 13:06:57.501807 12586 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:06:57.502018 12586 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:06:57.502238 12586 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:57.508373 12586 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER }
I20260709 13:06:57.508939 12586 leader_election.cc:304] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ed37b04e9944d74ba43014e36c4791c; no voters: 
I20260709 13:06:57.510145 12586 leader_election.cc:290] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:06:57.510673 12589 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:57.512216 12589 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 1 LEADER]: Becoming Leader. State: Replica: 2ed37b04e9944d74ba43014e36c4791c, State: Running, Role: LEADER
I20260709 13:06:57.512948 12589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER }
I20260709 13:06:57.513360 12586 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:06:57.517616 12591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ed37b04e9944d74ba43014e36c4791c. Latest consensus state: current_term: 1 leader_uuid: "2ed37b04e9944d74ba43014e36c4791c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER } }
I20260709 13:06:57.518381 12591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:57.519738 12590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ed37b04e9944d74ba43014e36c4791c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ed37b04e9944d74ba43014e36c4791c" member_type: VOTER } }
I20260709 13:06:57.520370 12590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [sys.catalog]: This master's current role is: LEADER
I20260709 13:06:57.521896 12597 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:06:57.530905 12597 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:06:57.532101   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:06:57.540843 12597 catalog_manager.cc:1383] Generated new cluster ID: e5733573dfeb47dcbb49c1482f3f1ac4
I20260709 13:06:57.541182 12597 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:06:57.565604 12597 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:06:57.568271 12597 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:06:57.590698 12597 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c: Generated new TSK 0
I20260709 13:06:57.591461 12597 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:06:57.599215   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:57.606771 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.608316 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.609419 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:57.610380   431 server_base.cc:1061] running on GCE node
I20260709 13:06:57.611442   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:57.611673   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:57.611838   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602417611818 us; error 0 us; skew 500 ppm
I20260709 13:06:57.612404   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:57.615284   431 webserver.cc:533] Webserver started at http://127.0.107.193:38935/ using document root <none> and password file <none>
I20260709 13:06:57.615813   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:57.616043   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:57.616330   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:57.617616   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "caa1fb15352347d6a05abe42b40f93d7"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.622640   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 13:06:57.626453 12615 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.627271   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.004s	sys 0.000s
I20260709 13:06:57.627575   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "caa1fb15352347d6a05abe42b40f93d7"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.627893   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:57.639578   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:57.640936   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:57.642642   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:57.645077   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:57.645313   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.645606   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:57.645777   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.696540   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:40893
I20260709 13:06:57.696673 12677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:40893 every 8 connection(s)
I20260709 13:06:57.703029   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:57.710978 12682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.712160 12683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:57.715071   431 server_base.cc:1061] running on GCE node
W20260709 13:06:57.714843 12685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:57.716710   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:06:57.716924   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:57.717165   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602417717141 us; error 0 us; skew 500 ppm
I20260709 13:06:57.717455 12678 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37625
I20260709 13:06:57.717900   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:57.717960 12678 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:57.719099 12678 heartbeater.cc:507] Master 127.0.107.254:37625 requested a full tablet report, sending...
I20260709 13:06:57.721393 12551 ts_manager.cc:194] Registered new tserver with Master: caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893)
I20260709 13:06:57.721886   431 webserver.cc:533] Webserver started at http://127.0.107.194:36345/ using document root <none> and password file <none>
I20260709 13:06:57.722627   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:57.722858   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:57.723316   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:57.723534 12551 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45690
I20260709 13:06:57.724788   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "9e501bea9dbd4518a29e3458792e415f"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.729986   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.003s
I20260709 13:06:57.733780 12690 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.734782   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 13:06:57.735145   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "9e501bea9dbd4518a29e3458792e415f"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.735471   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:57.757053   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:57.758556   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:57.760191   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:57.763163   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:57.763355   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.763689   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:57.763849   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.816963   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:41255
I20260709 13:06:57.817109 12752 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:41255 every 8 connection(s)
I20260709 13:06:57.822966   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:06:57.831463 12756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.832513 12757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:06:57.836588 12759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:06:57.837231   431 server_base.cc:1061] running on GCE node
I20260709 13:06:57.838255 12753 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37625
I20260709 13:06:57.838430   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:06:57.838704 12753 heartbeater.cc:461] Registering TS with master...
W20260709 13:06:57.838774   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:06:57.839162   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602417839137 us; error 0 us; skew 500 ppm
I20260709 13:06:57.839691 12753 heartbeater.cc:507] Master 127.0.107.254:37625 requested a full tablet report, sending...
I20260709 13:06:57.839813   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:06:57.842036 12551 ts_manager.cc:194] Registered new tserver with Master: 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255)
I20260709 13:06:57.842851   431 webserver.cc:533] Webserver started at http://127.0.107.195:38577/ using document root <none> and password file <none>
I20260709 13:06:57.843444   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:06:57.843662   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:06:57.843698 12551 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45700
I20260709 13:06:57.843950   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:06:57.845534   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "6a9b492e68cd43abbd59dac5b2051823"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.850396   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:06:57.854027 12764 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.854803   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 13:06:57.855137   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "6a9b492e68cd43abbd59dac5b2051823"
format_stamp: "Formatted at 2026-07-09 13:06:57 on dist-test-slave-p8rg"
I20260709 13:06:57.855475   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:06:57.883406   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:06:57.885052   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:06:57.886890   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:06:57.890023   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:06:57.927731   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.037s	user 0.001s	sys 0.000s
I20260709 13:06:57.931882   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:06:57.932186   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:06:57.984485   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:39927
I20260709 13:06:57.984840 12827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:39927 every 8 connection(s)
I20260709 13:06:58.006253 12828 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37625
I20260709 13:06:58.006875 12828 heartbeater.cc:461] Registering TS with master...
I20260709 13:06:58.007862 12828 heartbeater.cc:507] Master 127.0.107.254:37625 requested a full tablet report, sending...
I20260709 13:06:58.010561 12551 ts_manager.cc:194] Registered new tserver with Master: 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.010962   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.021697221s
I20260709 13:06:58.013636 12551 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45704
I20260709 13:06:58.046571 12551 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45712:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:58.049333 12551 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:58.109042 12717 tablet_service.cc:1511] Processing CreateTablet for tablet 505fa1d73bc24a20a260a206de0826f5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.109128 12718 tablet_service.cc:1511] Processing CreateTablet for tablet 89f7ef86b6fb4f9192416f8a24b136e5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.111711 12717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505fa1d73bc24a20a260a206de0826f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.113473 12718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f7ef86b6fb4f9192416f8a24b136e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.122979 12643 tablet_service.cc:1511] Processing CreateTablet for tablet 89f7ef86b6fb4f9192416f8a24b136e5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.123679 12792 tablet_service.cc:1511] Processing CreateTablet for tablet 505fa1d73bc24a20a260a206de0826f5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.124706 12642 tablet_service.cc:1511] Processing CreateTablet for tablet 505fa1d73bc24a20a260a206de0826f5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.125576 12792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505fa1d73bc24a20a260a206de0826f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.126243 12643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f7ef86b6fb4f9192416f8a24b136e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.127635 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 89f7ef86b6fb4f9192416f8a24b136e5 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=df394b7ad9d847a382504a040656bd10]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.129369 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f7ef86b6fb4f9192416f8a24b136e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.132006 12642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 505fa1d73bc24a20a260a206de0826f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.151721 12842 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: Bootstrap starting.
I20260709 13:06:58.157315 12843 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: Bootstrap starting.
I20260709 13:06:58.162905 12842 tablet_bootstrap.cc:654] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.166260 12843 tablet_bootstrap.cc:654] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.169791 12844 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: Bootstrap starting.
I20260709 13:06:58.184633 12842 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: No bootstrap required, opened a new log
I20260709 13:06:58.185323 12842 ts_tablet_manager.cc:1403] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.009s
I20260709 13:06:58.188637 12842 raft_consensus.cc:359] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.189206 12843 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: No bootstrap required, opened a new log
I20260709 13:06:58.189566 12842 raft_consensus.cc:385] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.189967 12842 raft_consensus.cc:740] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.190016 12843 ts_tablet_manager.cc:1403] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.015s
I20260709 13:06:58.190239 12844 tablet_bootstrap.cc:654] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.190915 12842 consensus_queue.cc:260] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.193900 12753 heartbeater.cc:499] Master 127.0.107.254:37625 was elected leader, sending a full tablet report...
I20260709 13:06:58.194334 12842 ts_tablet_manager.cc:1434] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:06:58.194578 12843 raft_consensus.cc:359] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.195447 12842 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: Bootstrap starting.
I20260709 13:06:58.195828 12843 raft_consensus.cc:385] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.196334 12843 raft_consensus.cc:740] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.197641 12843 consensus_queue.cc:260] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.205848 12842 tablet_bootstrap.cc:654] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.207358 12844 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: No bootstrap required, opened a new log
I20260709 13:06:58.208050 12844 ts_tablet_manager.cc:1403] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: Time spent bootstrapping tablet: real 0.039s	user 0.011s	sys 0.015s
I20260709 13:06:58.211740 12844 raft_consensus.cc:359] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.212695 12844 raft_consensus.cc:385] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.213071 12844 raft_consensus.cc:740] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9b492e68cd43abbd59dac5b2051823, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.214148 12844 consensus_queue.cc:260] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.224272 12844 ts_tablet_manager.cc:1434] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.007s
I20260709 13:06:58.225211 12828 heartbeater.cc:499] Master 127.0.107.254:37625 was elected leader, sending a full tablet report...
I20260709 13:06:58.226127 12852 raft_consensus.cc:493] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:58.226761 12852 raft_consensus.cc:515] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.225391 12678 heartbeater.cc:499] Master 127.0.107.254:37625 was elected leader, sending a full tablet report...
I20260709 13:06:58.237087 12843 ts_tablet_manager.cc:1434] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: Time spent starting tablet: real 0.047s	user 0.022s	sys 0.021s
I20260709 13:06:58.242762 12843 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: Bootstrap starting.
I20260709 13:06:58.244146 12844 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: Bootstrap starting.
I20260709 13:06:58.252535 12842 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: No bootstrap required, opened a new log
I20260709 13:06:58.253104 12844 tablet_bootstrap.cc:654] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.253278 12842 ts_tablet_manager.cc:1403] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: Time spent bootstrapping tablet: real 0.058s	user 0.032s	sys 0.016s
I20260709 13:06:58.254361 12843 tablet_bootstrap.cc:654] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.260960 12852 leader_election.cc:290] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.261018 12842 raft_consensus.cc:359] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.262576 12842 raft_consensus.cc:385] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.263015 12842 raft_consensus.cc:740] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.264250 12842 consensus_queue.cc:260] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.267688 12842 ts_tablet_manager.cc:1434] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.003s
I20260709 13:06:58.272758 12843 tablet_bootstrap.cc:492] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: No bootstrap required, opened a new log
I20260709 13:06:58.273536 12843 ts_tablet_manager.cc:1403] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: Time spent bootstrapping tablet: real 0.031s	user 0.009s	sys 0.016s
I20260709 13:06:58.277099 12843 raft_consensus.cc:359] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.277961 12843 raft_consensus.cc:385] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.278330 12843 raft_consensus.cc:740] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.279153 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89f7ef86b6fb4f9192416f8a24b136e5" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e501bea9dbd4518a29e3458792e415f" is_pre_election: true
I20260709 13:06:58.279354 12843 consensus_queue.cc:260] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.280146 12728 raft_consensus.cc:2468] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 0.
I20260709 13:06:58.280699 12844 tablet_bootstrap.cc:492] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: No bootstrap required, opened a new log
I20260709 13:06:58.281396 12844 ts_tablet_manager.cc:1403] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: Time spent bootstrapping tablet: real 0.038s	user 0.016s	sys 0.006s
I20260709 13:06:58.282127 12843 ts_tablet_manager.cc:1434] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:06:58.282003 12619 leader_election.cc:304] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e501bea9dbd4518a29e3458792e415f, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:58.283267 12852 raft_consensus.cc:2804] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:58.283700 12852 raft_consensus.cc:493] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:58.284088 12852 raft_consensus.cc:3060] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.285251 12844 raft_consensus.cc:359] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.286234 12844 raft_consensus.cc:385] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.286588 12844 raft_consensus.cc:740] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9b492e68cd43abbd59dac5b2051823, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.287776 12844 consensus_queue.cc:260] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.290326 12844 ts_tablet_manager.cc:1434] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 13:06:58.294163 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89f7ef86b6fb4f9192416f8a24b136e5" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823" is_pre_election: true
I20260709 13:06:58.294932 12803 raft_consensus.cc:2468] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 0.
I20260709 13:06:58.295768 12852 raft_consensus.cc:515] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.298038 12852 leader_election.cc:290] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 election: Requested vote from peers 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.299014 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89f7ef86b6fb4f9192416f8a24b136e5" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
I20260709 13:06:58.299074 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89f7ef86b6fb4f9192416f8a24b136e5" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:06:58.299849 12728 raft_consensus.cc:3060] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.299906 12803 raft_consensus.cc:3060] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.309613 12728 raft_consensus.cc:2468] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 1.
I20260709 13:06:58.309602 12803 raft_consensus.cc:2468] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 1.
I20260709 13:06:58.311079 12619 leader_election.cc:304] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e501bea9dbd4518a29e3458792e415f, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:58.311987 12852 raft_consensus.cc:2804] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:58.312497 12852 raft_consensus.cc:697] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Becoming Leader. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Running, Role: LEADER
I20260709 13:06:58.313441 12852 consensus_queue.cc:237] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.322288 12551 catalog_manager.cc:5697] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "caa1fb15352347d6a05abe42b40f93d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
W20260709 13:06:58.526799 12599 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:06:58.547744 12847 raft_consensus.cc:493] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:58.548403 12847 raft_consensus.cc:515] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.550866 12847 leader_election.cc:290] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.563000 12653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa1fb15352347d6a05abe42b40f93d7" is_pre_election: true
I20260709 13:06:58.564014 12653 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 0.
I20260709 13:06:58.565636 12691 leader_election.cc:304] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e501bea9dbd4518a29e3458792e415f, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:58.566108 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823" is_pre_election: true
I20260709 13:06:58.566681 12847 raft_consensus.cc:2804] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:58.566877 12803 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 0.
I20260709 13:06:58.567171 12847 raft_consensus.cc:493] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:58.567554 12847 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.575350 12847 raft_consensus.cc:515] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.577134 12847 leader_election.cc:290] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 election: Requested vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.578106 12653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
I20260709 13:06:58.578188 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:06:58.578780 12653 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.578856 12803 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.584595 12803 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 1.
I20260709 13:06:58.585484 12653 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 1.
I20260709 13:06:58.585767 12691 leader_election.cc:304] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, 9e501bea9dbd4518a29e3458792e415f; no voters: 
I20260709 13:06:58.586647 12847 raft_consensus.cc:2804] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:58.587671 12847 raft_consensus.cc:697] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Becoming Leader. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Running, Role: LEADER
I20260709 13:06:58.588644 12847 consensus_queue.cc:237] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.597419 12551 catalog_manager.cc:5697] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e501bea9dbd4518a29e3458792e415f (127.0.107.194). New cstate: current_term: 1 leader_uuid: "9e501bea9dbd4518a29e3458792e415f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:58.641935 12551 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45720:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:06:58.644040 12551 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:06:58.669308 12642 tablet_service.cc:1511] Processing CreateTablet for tablet 8016c2d6ebc44893b9c41564bebc9672 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.670833 12642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8016c2d6ebc44893b9c41564bebc9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.671332 12717 tablet_service.cc:1511] Processing CreateTablet for tablet 8016c2d6ebc44893b9c41564bebc9672 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.673012 12717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8016c2d6ebc44893b9c41564bebc9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.673461 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 8016c2d6ebc44893b9c41564bebc9672 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 13:06:58.674872 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8016c2d6ebc44893b9c41564bebc9672. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.674988 12643 tablet_service.cc:1511] Processing CreateTablet for tablet 8d4535d5f63e4618bf8559c9395c3d3f (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.676272 12643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4535d5f63e4618bf8559c9395c3d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.692380 12843 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: Bootstrap starting.
I20260709 13:06:58.693943 12718 tablet_service.cc:1511] Processing CreateTablet for tablet 8d4535d5f63e4618bf8559c9395c3d3f (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.695482 12718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4535d5f63e4618bf8559c9395c3d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.699563 12844 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: Bootstrap starting.
I20260709 13:06:58.706097 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 8d4535d5f63e4618bf8559c9395c3d3f (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=42a6624d3780427cbc0f661e5e2b8687]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 13:06:58.711221 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d4535d5f63e4618bf8559c9395c3d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:06:58.708144 12843 tablet_bootstrap.cc:654] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.709586 12842 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Bootstrap starting.
I20260709 13:06:58.709789 12844 tablet_bootstrap.cc:654] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.721446 12842 tablet_bootstrap.cc:654] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.733106 12844 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: No bootstrap required, opened a new log
I20260709 13:06:58.733318 12843 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: No bootstrap required, opened a new log
I20260709 13:06:58.733922 12843 ts_tablet_manager.cc:1403] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.004s
I20260709 13:06:58.733850 12842 tablet_bootstrap.cc:492] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: No bootstrap required, opened a new log
I20260709 13:06:58.733862 12844 ts_tablet_manager.cc:1403] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: Time spent bootstrapping tablet: real 0.035s	user 0.012s	sys 0.009s
I20260709 13:06:58.734969 12842 ts_tablet_manager.cc:1403] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.007s
I20260709 13:06:58.738229 12843 raft_consensus.cc:359] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.738605 12842 raft_consensus.cc:359] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.739414 12842 raft_consensus.cc:385] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.739231 12844 raft_consensus.cc:359] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.739745 12842 raft_consensus.cc:740] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.740059 12844 raft_consensus.cc:385] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.740439 12844 raft_consensus.cc:740] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9b492e68cd43abbd59dac5b2051823, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.739151 12843 raft_consensus.cc:385] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.740705 12842 consensus_queue.cc:260] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.741400 12843 raft_consensus.cc:740] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.741426 12844 consensus_queue.cc:260] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.742550 12843 consensus_queue.cc:260] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.744565 12842 ts_tablet_manager.cc:1434] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:58.745355 12843 ts_tablet_manager.cc:1434] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:06:58.745846 12842 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: Bootstrap starting.
I20260709 13:06:58.746557 12843 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: Bootstrap starting.
I20260709 13:06:58.746564 12844 ts_tablet_manager.cc:1434] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 13:06:58.748039 12844 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: Bootstrap starting.
I20260709 13:06:58.754096 12842 tablet_bootstrap.cc:654] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.758363 12844 tablet_bootstrap.cc:654] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.767534 12844 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: No bootstrap required, opened a new log
I20260709 13:06:58.767448 12843 tablet_bootstrap.cc:654] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: Neither blocks nor log segments found. Creating new log.
I20260709 13:06:58.768380 12844 ts_tablet_manager.cc:1403] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.003s
I20260709 13:06:58.771634 12844 raft_consensus.cc:359] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.772289 12844 raft_consensus.cc:385] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.772539 12844 raft_consensus.cc:740] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9b492e68cd43abbd59dac5b2051823, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.772596 12842 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: No bootstrap required, opened a new log
I20260709 13:06:58.773378 12842 ts_tablet_manager.cc:1403] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.007s
I20260709 13:06:58.773327 12844 consensus_queue.cc:260] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.775498 12844 ts_tablet_manager.cc:1434] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 13:06:58.777351 12842 raft_consensus.cc:359] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.778255 12842 raft_consensus.cc:385] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.778578 12843 tablet_bootstrap.cc:492] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: No bootstrap required, opened a new log
I20260709 13:06:58.778594 12842 raft_consensus.cc:740] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.779238 12843 ts_tablet_manager.cc:1403] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: Time spent bootstrapping tablet: real 0.033s	user 0.008s	sys 0.010s
I20260709 13:06:58.780038 12842 consensus_queue.cc:260] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.782871 12842 ts_tablet_manager.cc:1434] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:06:58.782814 12843 raft_consensus.cc:359] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.785408 12843 raft_consensus.cc:385] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:06:58.785830 12843 raft_consensus.cc:740] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Initialized, Role: FOLLOWER
I20260709 13:06:58.786545 12843 consensus_queue.cc:260] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.789278 12843 ts_tablet_manager.cc:1434] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 13:06:58.826479 12852 consensus_queue.cc:1048] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:58.842038 12875 consensus_queue.cc:1048] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:58.947708 12847 raft_consensus.cc:493] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:58.948241 12847 raft_consensus.cc:515] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.950246 12847 leader_election.cc:290] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.951123 12653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa1fb15352347d6a05abe42b40f93d7" is_pre_election: true
I20260709 13:06:58.951275 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823" is_pre_election: true
I20260709 13:06:58.951907 12653 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 0.
I20260709 13:06:58.951952 12803 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 0.
I20260709 13:06:58.953199 12691 leader_election.cc:304] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e501bea9dbd4518a29e3458792e415f, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:58.954005 12847 raft_consensus.cc:2804] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:58.954325 12847 raft_consensus.cc:493] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:58.954644 12847 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.962386 12847 raft_consensus.cc:515] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.964262 12847 leader_election.cc:290] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 election: Requested vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:58.965265 12653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
I20260709 13:06:58.965927 12653 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.965739 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:06:58.966570 12803 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:58.972551 12847 consensus_queue.cc:1048] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:58.973261 12653 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 1.
I20260709 13:06:58.974092 12803 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 1.
I20260709 13:06:58.974802 12691 leader_election.cc:304] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e501bea9dbd4518a29e3458792e415f, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:58.975660 12847 raft_consensus.cc:2804] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:58.976243 12847 raft_consensus.cc:697] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Becoming Leader. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Running, Role: LEADER
I20260709 13:06:58.977116 12847 consensus_queue.cc:237] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:58.987736 12860 consensus_queue.cc:1048] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:58.990608 12549 catalog_manager.cc:5697] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e501bea9dbd4518a29e3458792e415f (127.0.107.194). New cstate: current_term: 1 leader_uuid: "9e501bea9dbd4518a29e3458792e415f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:59.049970 12875 raft_consensus.cc:493] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:06:59.050647 12875 raft_consensus.cc:515] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.052538 12875 leader_election.cc:290] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:59.053624 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e501bea9dbd4518a29e3458792e415f" is_pre_election: true
I20260709 13:06:59.053624 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823" is_pre_election: true
I20260709 13:06:59.054606 12728 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 0.
I20260709 13:06:59.054606 12803 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 0.
I20260709 13:06:59.056139 12616 leader_election.cc:304] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:59.056943 12875 raft_consensus.cc:2804] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:06:59.057343 12875 raft_consensus.cc:493] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:06:59.057700 12875 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:59.064507 12875 raft_consensus.cc:515] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.066277 12875 leader_election.cc:290] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 election: Requested vote from peers 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:59.067206 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
I20260709 13:06:59.067387 12803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:06:59.067830 12728 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:59.067986 12803 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:06:59.074501 12803 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 1.
I20260709 13:06:59.075098 12728 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 1.
I20260709 13:06:59.075690 12616 leader_election.cc:304] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:59.076627 12875 raft_consensus.cc:2804] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:06:59.077193 12875 raft_consensus.cc:697] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Becoming Leader. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Running, Role: LEADER
I20260709 13:06:59.078179 12875 consensus_queue.cc:237] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.085968 12547 catalog_manager.cc:5697] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "caa1fb15352347d6a05abe42b40f93d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:59.186178 12728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5"
dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
mode: GRACEFUL
new_leader_uuid: "6a9b492e68cd43abbd59dac5b2051823"
 from {username='slave'} at 127.0.0.1:38460
I20260709 13:06:59.186825 12728 raft_consensus.cc:606] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Received request to transfer leadership to TS 6a9b492e68cd43abbd59dac5b2051823
I20260709 13:06:59.195756 12728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f"
dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
mode: GRACEFUL
new_leader_uuid: "caa1fb15352347d6a05abe42b40f93d7"
 from {username='slave'} at 127.0.0.1:38460
I20260709 13:06:59.196259 12728 raft_consensus.cc:606] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Received request to transfer leadership to TS caa1fb15352347d6a05abe42b40f93d7
I20260709 13:06:59.530457 12888 consensus_queue.cc:1048] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:06:59.540613 12892 raft_consensus.cc:993] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: : Instructing follower 6a9b492e68cd43abbd59dac5b2051823 to start an election
I20260709 13:06:59.541253 12892 raft_consensus.cc:1081] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Signalling peer 6a9b492e68cd43abbd59dac5b2051823 to start an election
I20260709 13:06:59.543401 12803 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5"
dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
 from {username='slave'} at 127.0.0.1:46240
I20260709 13:06:59.544097 12803 raft_consensus.cc:493] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:59.544484 12803 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.550534 12888 consensus_queue.cc:1048] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:59.554761 12803 raft_consensus.cc:515] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.559104 12803 leader_election.cc:290] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [CANDIDATE]: Term 2 election: Requested vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255)
I20260709 13:06:59.575235 12889 consensus_queue.cc:1048] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:59.582880 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "6a9b492e68cd43abbd59dac5b2051823" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
I20260709 13:06:59.583467 12728 raft_consensus.cc:3055] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:59.583693 12728 raft_consensus.cc:740] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Running, Role: LEADER
I20260709 13:06:59.584426 12728 consensus_queue.cc:260] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.585569 12728 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.589860 12653 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "505fa1d73bc24a20a260a206de0826f5" candidate_uuid: "6a9b492e68cd43abbd59dac5b2051823" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
I20260709 13:06:59.590641 12653 raft_consensus.cc:3060] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.592783 12728 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9b492e68cd43abbd59dac5b2051823 in term 2.
I20260709 13:06:59.597376 12768 leader_election.cc:304] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, 9e501bea9dbd4518a29e3458792e415f; no voters: 
I20260709 13:06:59.598342 12890 raft_consensus.cc:2804] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:59.598384 12653 raft_consensus.cc:2468] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a9b492e68cd43abbd59dac5b2051823 in term 2.
I20260709 13:06:59.602535 12890 raft_consensus.cc:697] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 LEADER]: Becoming Leader. State: Replica: 6a9b492e68cd43abbd59dac5b2051823, State: Running, Role: LEADER
I20260709 13:06:59.603655 12892 consensus_queue.cc:1048] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:06:59.603971 12890 consensus_queue.cc:237] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.613945 12895 raft_consensus.cc:993] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: : Instructing follower caa1fb15352347d6a05abe42b40f93d7 to start an election
I20260709 13:06:59.614547 12892 raft_consensus.cc:1081] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Signalling peer caa1fb15352347d6a05abe42b40f93d7 to start an election
I20260709 13:06:59.615726 12549 catalog_manager.cc:5697] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 reported cstate change: term changed from 1 to 2, leader changed from 9e501bea9dbd4518a29e3458792e415f (127.0.107.194) to 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "6a9b492e68cd43abbd59dac5b2051823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: HEALTHY } } }
I20260709 13:06:59.616562 12652 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f"
dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
 from {username='slave'} at 127.0.0.1:60232
I20260709 13:06:59.619243 12652 raft_consensus.cc:493] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:06:59.619771 12652 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.630347 12652 raft_consensus.cc:515] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.633009 12652 leader_election.cc:290] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 2 election: Requested vote from peers 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:06:59.633306 12728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
I20260709 13:06:59.634153 12728 raft_consensus.cc:3055] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:06:59.634277 12802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d4535d5f63e4618bf8559c9395c3d3f" candidate_uuid: "caa1fb15352347d6a05abe42b40f93d7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:06:59.634657 12728 raft_consensus.cc:740] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Running, Role: LEADER
I20260709 13:06:59.635044 12802 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.635684 12728 consensus_queue.cc:260] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.637130 12728 raft_consensus.cc:3060] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:06:59.644697 12802 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 2.
I20260709 13:06:59.646039 12728 raft_consensus.cc:2468] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate caa1fb15352347d6a05abe42b40f93d7 in term 2.
I20260709 13:06:59.647131 12616 leader_election.cc:304] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, caa1fb15352347d6a05abe42b40f93d7; no voters: 
I20260709 13:06:59.648028 12888 raft_consensus.cc:2804] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:06:59.648589 12888 raft_consensus.cc:697] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 2 LEADER]: Becoming Leader. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Running, Role: LEADER
I20260709 13:06:59.649555 12888 consensus_queue.cc:237] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:06:59.657683 12549 catalog_manager.cc:5697] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 reported cstate change: term changed from 1 to 2, leader changed from 9e501bea9dbd4518a29e3458792e415f (127.0.107.194) to caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "caa1fb15352347d6a05abe42b40f93d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260709 13:06:59.935155   431 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 13:06:59.938062   431 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 13:06:59.938789   431 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 13:06:59.955323 12652 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672"
dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
mode: GRACEFUL
new_leader_uuid: "9e501bea9dbd4518a29e3458792e415f"
 from {username='slave'} at 127.0.0.1:60256
I20260709 13:06:59.956197 12652 raft_consensus.cc:606] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Received request to transfer leadership to TS 9e501bea9dbd4518a29e3458792e415f
I20260709 13:06:59.957837   431 auto_leader_rebalancer.cc:452] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 13:07:00.062176 12728 raft_consensus.cc:1275] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Refusing update from remote peer 6a9b492e68cd43abbd59dac5b2051823: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:00.063761 12890 consensus_queue.cc:1048] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:00.075116 12652 raft_consensus.cc:1275] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Refusing update from remote peer 6a9b492e68cd43abbd59dac5b2051823: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:00.078219 12902 consensus_queue.cc:1048] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:00.182819 12911 raft_consensus.cc:993] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: : Instructing follower 9e501bea9dbd4518a29e3458792e415f to start an election
I20260709 13:07:00.183383 12912 raft_consensus.cc:1081] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Signalling peer 9e501bea9dbd4518a29e3458792e415f to start an election
I20260709 13:07:00.185051 12728 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672"
dest_uuid: "9e501bea9dbd4518a29e3458792e415f"
 from {username='slave'} at 127.0.0.1:38444
I20260709 13:07:00.185648 12728 raft_consensus.cc:493] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:00.185984 12728 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:00.194070 12728 raft_consensus.cc:515] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:07:00.195513 12727 raft_consensus.cc:1275] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Refusing update from remote peer caa1fb15352347d6a05abe42b40f93d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:00.196017 12728 leader_election.cc:290] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 2 election: Requested vote from peers caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193:40893), 6a9b492e68cd43abbd59dac5b2051823 (127.0.107.195:39927)
I20260709 13:07:00.196960 12652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "caa1fb15352347d6a05abe42b40f93d7"
I20260709 13:07:00.197373 12802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8016c2d6ebc44893b9c41564bebc9672" candidate_uuid: "9e501bea9dbd4518a29e3458792e415f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6a9b492e68cd43abbd59dac5b2051823"
I20260709 13:07:00.197590 12912 consensus_queue.cc:1048] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:00.197742 12652 raft_consensus.cc:3055] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:00.198068 12802 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:00.198333 12652 raft_consensus.cc:740] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: caa1fb15352347d6a05abe42b40f93d7, State: Running, Role: LEADER
I20260709 13:07:00.199565 12652 consensus_queue.cc:260] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:07:00.201005 12652 raft_consensus.cc:3060] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:00.206763 12802 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 2.
I20260709 13:07:00.208554 12803 raft_consensus.cc:1275] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Refusing update from remote peer caa1fb15352347d6a05abe42b40f93d7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:00.210860 12691 leader_election.cc:304] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a9b492e68cd43abbd59dac5b2051823, 9e501bea9dbd4518a29e3458792e415f; no voters: 
I20260709 13:07:00.212093 12892 raft_consensus.cc:2804] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:00.213749 12892 raft_consensus.cc:697] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 LEADER]: Becoming Leader. State: Replica: 9e501bea9dbd4518a29e3458792e415f, State: Running, Role: LEADER
I20260709 13:07:00.210877 12888 consensus_queue.cc:1048] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 13:07:00.215101 12892 consensus_queue.cc:237] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } }
I20260709 13:07:00.213625 12652 raft_consensus.cc:2468] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e501bea9dbd4518a29e3458792e415f in term 2.
I20260709 13:07:00.226069 12547 catalog_manager.cc:5697] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f reported cstate change: term changed from 1 to 2, leader changed from caa1fb15352347d6a05abe42b40f93d7 (127.0.107.193) to 9e501bea9dbd4518a29e3458792e415f (127.0.107.194). New cstate: current_term: 2 leader_uuid: "9e501bea9dbd4518a29e3458792e415f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "caa1fb15352347d6a05abe42b40f93d7" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e501bea9dbd4518a29e3458792e415f" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 41255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a9b492e68cd43abbd59dac5b2051823" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 39927 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:00.331660   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
I20260709 13:07:00.359799   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:00.361049   431 tablet_replica.cc:333] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7: stopping tablet replica
I20260709 13:07:00.362128   431 raft_consensus.cc:2243] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.362722   431 raft_consensus.cc:2272] T 8016c2d6ebc44893b9c41564bebc9672 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.365890   431 tablet_replica.cc:333] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7: stopping tablet replica
I20260709 13:07:00.366611   431 raft_consensus.cc:2243] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:00.367985   431 raft_consensus.cc:2272] T 8d4535d5f63e4618bf8559c9395c3d3f P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.371304   431 tablet_replica.cc:333] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7: stopping tablet replica
I20260709 13:07:00.371917   431 raft_consensus.cc:2243] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.372404   431 raft_consensus.cc:2272] T 505fa1d73bc24a20a260a206de0826f5 P caa1fb15352347d6a05abe42b40f93d7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.374959   431 tablet_replica.cc:333] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7: stopping tablet replica
I20260709 13:07:00.375497   431 raft_consensus.cc:2243] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:00.376367   431 raft_consensus.cc:2272] T 89f7ef86b6fb4f9192416f8a24b136e5 P caa1fb15352347d6a05abe42b40f93d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.403474   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
I20260709 13:07:00.420302   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
I20260709 13:07:00.448477   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:00.449349   431 tablet_replica.cc:333] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: stopping tablet replica
I20260709 13:07:00.450039   431 raft_consensus.cc:2243] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:00.450729   431 pending_rounds.cc:70] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Trying to abort 1 pending ops.
I20260709 13:07:00.450903   431 pending_rounds.cc:77] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305635513215225856 op_type: NO_OP noop_request { }
I20260709 13:07:00.451187   431 raft_consensus.cc:2889] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 13:07:00.451442   431 raft_consensus.cc:2272] T 8016c2d6ebc44893b9c41564bebc9672 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.453889   431 tablet_replica.cc:333] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f: stopping tablet replica
I20260709 13:07:00.454416   431 raft_consensus.cc:2243] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.454877   431 raft_consensus.cc:2272] T 8d4535d5f63e4618bf8559c9395c3d3f P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.457724   431 tablet_replica.cc:333] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f: stopping tablet replica
I20260709 13:07:00.458284   431 raft_consensus.cc:2243] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.458714   431 raft_consensus.cc:2272] T 505fa1d73bc24a20a260a206de0826f5 P 9e501bea9dbd4518a29e3458792e415f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.461007   431 tablet_replica.cc:333] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f: stopping tablet replica
I20260709 13:07:00.461611   431 raft_consensus.cc:2243] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.462033   431 raft_consensus.cc:2272] T 89f7ef86b6fb4f9192416f8a24b136e5 P 9e501bea9dbd4518a29e3458792e415f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:00.479458 12768 consensus_peers.cc:597] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 -> Peer 9e501bea9dbd4518a29e3458792e415f (127.0.107.194:41255): Couldn't send request to peer 9e501bea9dbd4518a29e3458792e415f. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:41255: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:00.487218   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 13:07:00.505578   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
I20260709 13:07:00.529742   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:00.530810   431 tablet_replica.cc:333] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823: stopping tablet replica
I20260709 13:07:00.531546   431 raft_consensus.cc:2243] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.532115   431 raft_consensus.cc:2272] T 8016c2d6ebc44893b9c41564bebc9672 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.535035   431 tablet_replica.cc:333] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823: stopping tablet replica
I20260709 13:07:00.535755   431 raft_consensus.cc:2243] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.536264   431 raft_consensus.cc:2272] T 89f7ef86b6fb4f9192416f8a24b136e5 P 6a9b492e68cd43abbd59dac5b2051823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.538617   431 tablet_replica.cc:333] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823: stopping tablet replica
I20260709 13:07:00.539247   431 raft_consensus.cc:2243] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:00.539764   431 raft_consensus.cc:2272] T 8d4535d5f63e4618bf8559c9395c3d3f P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.542701   431 tablet_replica.cc:333] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823: stopping tablet replica
I20260709 13:07:00.543543   431 raft_consensus.cc:2243] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:00.544531   431 raft_consensus.cc:2272] T 505fa1d73bc24a20a260a206de0826f5 P 6a9b492e68cd43abbd59dac5b2051823 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.560123   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:07:00.577288   431 master.cc:562] Master@127.0.107.254:37625 shutting down...
I20260709 13:07:00.599275   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:00.600040   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:00.600445   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2ed37b04e9944d74ba43014e36c4791c: stopping tablet replica
I20260709 13:07:00.616067   431 master.cc:584] Master@127.0.107.254:37625 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (3285 ms)
[----------] 6 tests from LeaderRebalancerTest (409959 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 13:07:00.658447   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:40339
I20260709 13:07:00.659725   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:00.666669 12920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:00.666785 12919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:00.667301 12922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:00.671101   431 server_base.cc:1061] running on GCE node
I20260709 13:07:00.672158   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:00.672358   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:00.672492   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602420672474 us; error 0 us; skew 500 ppm
I20260709 13:07:00.673025   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:00.675896   431 webserver.cc:533] Webserver started at http://127.0.107.254:40703/ using document root <none> and password file <none>
I20260709 13:07:00.676411   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:00.676575   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:00.676837   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:00.678151   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "90f12b937c774e499518ad9714a34cfe"
format_stamp: "Formatted at 2026-07-09 13:07:00 on dist-test-slave-p8rg"
I20260709 13:07:00.683341   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 13:07:00.687525 12927 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:00.688498   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:07:00.688863   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "90f12b937c774e499518ad9714a34cfe"
format_stamp: "Formatted at 2026-07-09 13:07:00 on dist-test-slave-p8rg"
I20260709 13:07:00.689792   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:00.717011   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:00.718427   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:00.764748   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:40339
I20260709 13:07:00.764959 12978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:40339 every 8 connection(s)
I20260709 13:07:00.770164 12979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:00.783375 12979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: Bootstrap starting.
I20260709 13:07:00.789093 12979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:00.794521 12979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: No bootstrap required, opened a new log
I20260709 13:07:00.797364 12979 raft_consensus.cc:359] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER }
I20260709 13:07:00.797861 12979 raft_consensus.cc:385] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:00.798060 12979 raft_consensus.cc:740] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90f12b937c774e499518ad9714a34cfe, State: Initialized, Role: FOLLOWER
I20260709 13:07:00.798761 12979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER }
I20260709 13:07:00.799412 12979 raft_consensus.cc:399] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:07:00.799686 12979 raft_consensus.cc:493] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:07:00.799957 12979 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:00.806208 12979 raft_consensus.cc:515] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER }
I20260709 13:07:00.806874 12979 leader_election.cc:304] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90f12b937c774e499518ad9714a34cfe; no voters: 
I20260709 13:07:00.808368 12979 leader_election.cc:290] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:07:00.808823 12982 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:00.810631 12982 raft_consensus.cc:697] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 1 LEADER]: Becoming Leader. State: Replica: 90f12b937c774e499518ad9714a34cfe, State: Running, Role: LEADER
I20260709 13:07:00.811540 12979 sys_catalog.cc:565] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:07:00.811463 12982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER }
I20260709 13:07:00.815192 12984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90f12b937c774e499518ad9714a34cfe. Latest consensus state: current_term: 1 leader_uuid: "90f12b937c774e499518ad9714a34cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER } }
I20260709 13:07:00.815857 12984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:00.816000 12983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90f12b937c774e499518ad9714a34cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90f12b937c774e499518ad9714a34cfe" member_type: VOTER } }
I20260709 13:07:00.816798 12983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:00.820485 12988 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:07:00.829212 12988 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:07:00.830410   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:07:00.839323 12988 catalog_manager.cc:1383] Generated new cluster ID: a2610c941b7b4677b39c02598b559745
I20260709 13:07:00.839720 12988 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:07:00.853044 12988 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:07:00.854535 12988 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:07:00.869920 12988 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: Generated new TSK 0
I20260709 13:07:00.870641 12988 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:07:00.897584   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:00.904871 13000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:00.906476 13001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:00.908134 13003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:00.908639   431 server_base.cc:1061] running on GCE node
I20260709 13:07:00.910634   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:00.910874   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:00.911048   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602420911025 us; error 0 us; skew 500 ppm
I20260709 13:07:00.911641   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:00.915004   431 webserver.cc:533] Webserver started at http://127.0.107.193:43601/ using document root <none> and password file <none>
I20260709 13:07:00.915540   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:00.915756   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:00.916049   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:00.917300   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "3fb587cb63854c42abf5c4f501ee32b3"
format_stamp: "Formatted at 2026-07-09 13:07:00 on dist-test-slave-p8rg"
I20260709 13:07:00.922325   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.001s
I20260709 13:07:00.926856 13008 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:00.928004   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:07:00.928295   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "3fb587cb63854c42abf5c4f501ee32b3"
format_stamp: "Formatted at 2026-07-09 13:07:00 on dist-test-slave-p8rg"
I20260709 13:07:00.928601   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:00.955371   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:00.956661   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:00.958359   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:00.961181   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:00.961445   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:00.962059   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:00.962227   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.018065   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:43505
I20260709 13:07:01.018240 13070 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:43505 every 8 connection(s)
I20260709 13:07:01.023910   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:01.032085 13075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:01.033151 13076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:01.036123   431 server_base.cc:1061] running on GCE node
W20260709 13:07:01.036810 13078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:01.037647 13071 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40339
I20260709 13:07:01.038081 13071 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:01.038798   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:07:01.038977 13071 heartbeater.cc:507] Master 127.0.107.254:40339 requested a full tablet report, sending...
W20260709 13:07:01.039286   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:01.039750   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602421039719 us; error 0 us; skew 500 ppm
I20260709 13:07:01.040591   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:01.042877 12944 ts_manager.cc:194] Registered new tserver with Master: 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505)
I20260709 13:07:01.044385   431 webserver.cc:533] Webserver started at http://127.0.107.194:39149/ using document root <none> and password file <none>
I20260709 13:07:01.045019   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:01.045077 12944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54636
I20260709 13:07:01.045284   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:01.045744   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:01.046962   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "077c76b5bd474881905cb6cfe23c3124"
format_stamp: "Formatted at 2026-07-09 13:07:01 on dist-test-slave-p8rg"
I20260709 13:07:01.051700   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.003s
I20260709 13:07:01.055356 13083 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.056231   431 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 13:07:01.056568   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "077c76b5bd474881905cb6cfe23c3124"
format_stamp: "Formatted at 2026-07-09 13:07:01 on dist-test-slave-p8rg"
I20260709 13:07:01.056875   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:01.068862   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:01.070250   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:01.071905   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:01.074493   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:01.074746   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.075002   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:01.075184   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.124854   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:39583
I20260709 13:07:01.124961 13145 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:39583 every 8 connection(s)
I20260709 13:07:01.130812   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:01.139295 13149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:01.140647 13150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:01.142580 13152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:01.145532 13146 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40339
I20260709 13:07:01.146085 13146 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:01.146078   431 server_base.cc:1061] running on GCE node
I20260709 13:07:01.147207 13146 heartbeater.cc:507] Master 127.0.107.254:40339 requested a full tablet report, sending...
I20260709 13:07:01.147305   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:01.147646   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:01.147847   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602421147825 us; error 0 us; skew 500 ppm
I20260709 13:07:01.148730   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:01.149814 12944 ts_manager.cc:194] Registered new tserver with Master: 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:01.151780   431 webserver.cc:533] Webserver started at http://127.0.107.195:33737/ using document root <none> and password file <none>
I20260709 13:07:01.151978 12944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54646
I20260709 13:07:01.152599   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:01.152902   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:01.153395   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:01.154851   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "8a788c7ceff54a7683cb674b33080008"
format_stamp: "Formatted at 2026-07-09 13:07:01 on dist-test-slave-p8rg"
I20260709 13:07:01.160187   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 13:07:01.164140 13157 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.165086   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 13:07:01.165477   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "8a788c7ceff54a7683cb674b33080008"
format_stamp: "Formatted at 2026-07-09 13:07:01 on dist-test-slave-p8rg"
I20260709 13:07:01.165843   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:01.184755   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:01.186072   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:01.188331   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:01.190734   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:01.190991   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.191462   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:01.191702   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:01.241103   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:45349
I20260709 13:07:01.241255 13219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:45349 every 8 connection(s)
I20260709 13:07:01.256481 13220 heartbeater.cc:344] Connected to a master server at 127.0.107.254:40339
I20260709 13:07:01.256987 13220 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:01.257783 13220 heartbeater.cc:507] Master 127.0.107.254:40339 requested a full tablet report, sending...
I20260709 13:07:01.259996 12944 ts_manager.cc:194] Registered new tserver with Master: 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:01.260701   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015909329s
I20260709 13:07:01.261703 12944 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:54648
I20260709 13:07:01.292115 12944 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54654:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:01.295270 12944 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:01.382977 13036 tablet_service.cc:1511] Processing CreateTablet for tablet 5426131135564b57af11c0740aa0c357 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:01.383260 13109 tablet_service.cc:1511] Processing CreateTablet for tablet bce188e6a2944fb0abd11b7fba0b46a7 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:01.384688 13036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5426131135564b57af11c0740aa0c357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.384948 13109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce188e6a2944fb0abd11b7fba0b46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.383327 13111 tablet_service.cc:1511] Processing CreateTablet for tablet 5426131135564b57af11c0740aa0c357 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:01.387447 13108 tablet_service.cc:1511] Processing CreateTablet for tablet e60047ab21874e2086ee857dce35d837 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:01.384454 13110 tablet_service.cc:1511] Processing CreateTablet for tablet e31761a5de5e4572b87141de0d90cba8 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:01.387888 13111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5426131135564b57af11c0740aa0c357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.389402 13108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60047ab21874e2086ee857dce35d837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.390702 13035 tablet_service.cc:1511] Processing CreateTablet for tablet e31761a5de5e4572b87141de0d90cba8 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:01.392226 13035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31761a5de5e4572b87141de0d90cba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.394711 13034 tablet_service.cc:1511] Processing CreateTablet for tablet bce188e6a2944fb0abd11b7fba0b46a7 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:01.395260 13184 tablet_service.cc:1511] Processing CreateTablet for tablet e31761a5de5e4572b87141de0d90cba8 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:01.396828 13034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce188e6a2944fb0abd11b7fba0b46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.398569 13185 tablet_service.cc:1511] Processing CreateTablet for tablet 5426131135564b57af11c0740aa0c357 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:01.399744 13033 tablet_service.cc:1511] Processing CreateTablet for tablet e60047ab21874e2086ee857dce35d837 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:01.400863 13110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31761a5de5e4572b87141de0d90cba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.401707 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60047ab21874e2086ee857dce35d837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.397238 13184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e31761a5de5e4572b87141de0d90cba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.403261 13185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5426131135564b57af11c0740aa0c357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.417227 13183 tablet_service.cc:1511] Processing CreateTablet for tablet bce188e6a2944fb0abd11b7fba0b46a7 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:01.419802 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce188e6a2944fb0abd11b7fba0b46a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.440758 13182 tablet_service.cc:1511] Processing CreateTablet for tablet e60047ab21874e2086ee857dce35d837 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:01.442759 13182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60047ab21874e2086ee857dce35d837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.458581 13107 tablet_service.cc:1511] Processing CreateTablet for tablet 12a7ada4b5544ef8891f296e167a3a84 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:01.458420 13032 tablet_service.cc:1511] Processing CreateTablet for tablet 12a7ada4b5544ef8891f296e167a3a84 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:01.460414 13107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7ada4b5544ef8891f296e167a3a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.460902 13032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7ada4b5544ef8891f296e167a3a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.476641 13181 tablet_service.cc:1511] Processing CreateTablet for tablet 12a7ada4b5544ef8891f296e167a3a84 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:01.478531 13181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12a7ada4b5544ef8891f296e167a3a84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.490026 13031 tablet_service.cc:1511] Processing CreateTablet for tablet d72ffe8729f143098d6418807c1a940b (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:01.491820 13031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ffe8729f143098d6418807c1a940b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.506983 13235 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:01.512292 13108 tablet_service.cc:1511] Processing CreateTablet for tablet d72ffe8729f143098d6418807c1a940b (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:01.514261 13235 tablet_bootstrap.cc:654] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.523059 13030 tablet_service.cc:1511] Processing CreateTablet for tablet 62db6cb99508458e9640d9841aa1998d (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:01.531802 13030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62db6cb99508458e9640d9841aa1998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.526186 13236 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.528332 13237 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:01.545537 13237 tablet_bootstrap.cc:654] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.547117 13183 tablet_service.cc:1511] Processing CreateTablet for tablet d72ffe8729f143098d6418807c1a940b (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:01.552423 13236 tablet_bootstrap.cc:654] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.562070 13110 tablet_service.cc:1511] Processing CreateTablet for tablet 62db6cb99508458e9640d9841aa1998d (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:01.530591 13108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ffe8729f143098d6418807c1a940b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.563812 13110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62db6cb99508458e9640d9841aa1998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.549310 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d72ffe8729f143098d6418807c1a940b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.567759 13235 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:01.568380 13235 ts_tablet_manager.cc:1403] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.062s	user 0.015s	sys 0.020s
I20260709 13:07:01.572117 13235 raft_consensus.cc:359] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.573987 13184 tablet_service.cc:1511] Processing CreateTablet for tablet 62db6cb99508458e9640d9841aa1998d (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:01.575790 13236 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.576262 13184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62db6cb99508458e9640d9841aa1998d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.576457 13236 ts_tablet_manager.cc:1403] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.051s	user 0.011s	sys 0.003s
I20260709 13:07:01.580082 13236 raft_consensus.cc:359] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.581027 13236 raft_consensus.cc:385] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.584757 13033 tablet_service.cc:1511] Processing CreateTablet for tablet 9cbad72123464a59a5a17a82b5cabe04 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:01.586534 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbad72123464a59a5a17a82b5cabe04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.587356 13237 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:01.587996 13237 ts_tablet_manager.cc:1403] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.060s	user 0.012s	sys 0.000s
I20260709 13:07:01.592078 13237 raft_consensus.cc:359] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.593096 13237 raft_consensus.cc:385] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.593544 13235 raft_consensus.cc:385] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.594116 13235 raft_consensus.cc:740] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.594928 13235 consensus_queue.cc:260] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.595073 13109 tablet_service.cc:1511] Processing CreateTablet for tablet 9cbad72123464a59a5a17a82b5cabe04 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:01.597474 13109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbad72123464a59a5a17a82b5cabe04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.598474 13181 tablet_service.cc:1511] Processing CreateTablet for tablet 9cbad72123464a59a5a17a82b5cabe04 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:01.599980 13181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cbad72123464a59a5a17a82b5cabe04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.593463 13237 raft_consensus.cc:740] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.602378 13237 consensus_queue.cc:260] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.603449 13031 tablet_service.cc:1511] Processing CreateTablet for tablet 8809f7896b03429fa2253088c87b8a71 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:01.605183 13031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8809f7896b03429fa2253088c87b8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.581332 13236 raft_consensus.cc:740] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.617681 13236 consensus_queue.cc:260] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.650977 13110 tablet_service.cc:1511] Processing CreateTablet for tablet 8809f7896b03429fa2253088c87b8a71 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:01.655614 13071 heartbeater.cc:499] Master 127.0.107.254:40339 was elected leader, sending a full tablet report...
I20260709 13:07:01.655938 13110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8809f7896b03429fa2253088c87b8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.656558 13236 ts_tablet_manager.cc:1434] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.080s	user 0.005s	sys 0.007s
I20260709 13:07:01.657759 13236 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.657831 13220 heartbeater.cc:499] Master 127.0.107.254:40339 was elected leader, sending a full tablet report...
I20260709 13:07:01.665963 13236 tablet_bootstrap.cc:654] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.666265 13235 ts_tablet_manager.cc:1434] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.098s	user 0.014s	sys 0.031s
I20260709 13:07:01.667171 13146 heartbeater.cc:499] Master 127.0.107.254:40339 was elected leader, sending a full tablet report...
I20260709 13:07:01.667618 13235 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:01.675064 13237 ts_tablet_manager.cc:1434] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.087s	user 0.003s	sys 0.017s
I20260709 13:07:01.676882 13183 tablet_service.cc:1511] Processing CreateTablet for tablet 8809f7896b03429fa2253088c87b8a71 (DEFAULT_TABLE table=test-workload [id=de6ba135d8ba4fcbb30860c9633b2982]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:01.679500 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8809f7896b03429fa2253088c87b8a71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:01.681942 13243 raft_consensus.cc:493] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.683333 13236 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.683188 13243 raft_consensus.cc:515] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.683985 13236 ts_tablet_manager.cc:1403] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.008s
I20260709 13:07:01.684403 13235 tablet_bootstrap.cc:654] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.687974 13236 raft_consensus.cc:359] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.689193 13236 raft_consensus.cc:385] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.689649 13236 raft_consensus.cc:740] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.676862 13237 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:01.690821 13236 consensus_queue.cc:260] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.698474 13243 leader_election.cc:290] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:01.707693 13237 tablet_bootstrap.cc:654] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.710819 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:01.712944 13236 ts_tablet_manager.cc:1434] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.000s
W20260709 13:07:01.713198 13159 leader_election.cc:343] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:01.722712 13236 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.732501 13236 tablet_bootstrap.cc:654] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.753244 13236 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.754257 13236 ts_tablet_manager.cc:1403] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.008s
I20260709 13:07:01.754554 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
W20260709 13:07:01.757344 13160 leader_election.cc:343] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:01.758049 13160 leader_election.cc:304] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:01.759354 13243 raft_consensus.cc:2749] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:01.759289 13236 raft_consensus.cc:359] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.760985 13236 raft_consensus.cc:385] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.761416 13236 raft_consensus.cc:740] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.762490 13236 consensus_queue.cc:260] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.766762 13236 ts_tablet_manager.cc:1434] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.003s
I20260709 13:07:01.768549 13236 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.778028 13236 tablet_bootstrap.cc:654] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.795493 13236 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.796533 13236 ts_tablet_manager.cc:1403] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.028s	user 0.015s	sys 0.008s
I20260709 13:07:01.801165 13236 raft_consensus.cc:359] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.802542 13236 raft_consensus.cc:385] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.802968 13236 raft_consensus.cc:740] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.803947 13236 consensus_queue.cc:260] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.805028 13237 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:01.805769 13237 ts_tablet_manager.cc:1403] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.129s	user 0.047s	sys 0.048s
I20260709 13:07:01.805931 13235 tablet_bootstrap.cc:492] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:01.806701 13235 ts_tablet_manager.cc:1403] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.139s	user 0.049s	sys 0.077s
I20260709 13:07:01.806712 13236 ts_tablet_manager.cc:1434] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.004s
I20260709 13:07:01.807936 13236 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.810110 13237 raft_consensus.cc:359] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.810899 13237 raft_consensus.cc:385] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.810871 13235 raft_consensus.cc:359] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.812151 13235 raft_consensus.cc:385] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.812579 13235 raft_consensus.cc:740] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.813714 13235 consensus_queue.cc:260] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.815287 13236 tablet_bootstrap.cc:654] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.811532 13237 raft_consensus.cc:740] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.818183 13237 consensus_queue.cc:260] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.819547 13235 ts_tablet_manager.cc:1434] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 13:07:01.821213 13235 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
W20260709 13:07:01.824724 12992 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:01.827054 13237 ts_tablet_manager.cc:1434] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.003s
I20260709 13:07:01.827768 13236 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.828418 13236 ts_tablet_manager.cc:1403] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.007s
I20260709 13:07:01.828711 13237 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:01.830282 13235 tablet_bootstrap.cc:654] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.832540 13236 raft_consensus.cc:359] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.833482 13236 raft_consensus.cc:385] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.833915 13236 raft_consensus.cc:740] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.835031 13236 consensus_queue.cc:260] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.837028 13237 tablet_bootstrap.cc:654] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.839286 13236 ts_tablet_manager.cc:1434] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.010s	user 0.012s	sys 0.000s
I20260709 13:07:01.841787 13236 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.850347 13235 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:01.851140 13235 ts_tablet_manager.cc:1403] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.016s
I20260709 13:07:01.852499 13237 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:01.852957 13236 tablet_bootstrap.cc:654] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.853139 13237 ts_tablet_manager.cc:1403] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.001s
I20260709 13:07:01.854506 13235 raft_consensus.cc:359] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.855406 13235 raft_consensus.cc:385] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.855698 13235 raft_consensus.cc:740] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.856419 13237 raft_consensus.cc:359] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.857239 13237 raft_consensus.cc:385] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.857592 13237 raft_consensus.cc:740] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.856925 13235 consensus_queue.cc:260] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.858353 13237 consensus_queue.cc:260] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.860780 13237 ts_tablet_manager.cc:1434] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 13:07:01.861727 13235 ts_tablet_manager.cc:1434] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 13:07:01.861935 13237 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:01.863581 13235 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:01.869597 13243 raft_consensus.cc:493] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.870916 13237 tablet_bootstrap.cc:654] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.870496 13243 raft_consensus.cc:515] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.871619 13236 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.872272 13236 ts_tablet_manager.cc:1403] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.006s
I20260709 13:07:01.873646 13243 leader_election.cc:290] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:01.874991 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:01.875659 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:01.876379 13046 raft_consensus.cc:2468] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
W20260709 13:07:01.876667 13159 leader_election.cc:343] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:01.875629 13236 raft_consensus.cc:359] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.878609 13236 raft_consensus.cc:385] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.878988 13236 raft_consensus.cc:740] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.879830 13236 consensus_queue.cc:260] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.880300 13160 leader_election.cc:304] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:01.886368 13235 tablet_bootstrap.cc:654] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.886756 13243 raft_consensus.cc:2804] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:01.887221 13243 raft_consensus.cc:493] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:01.887322 13237 tablet_bootstrap.cc:492] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:01.887646 13243 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:01.888079 13237 ts_tablet_manager.cc:1403] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.006s
I20260709 13:07:01.892035 13237 raft_consensus.cc:359] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.892971 13237 raft_consensus.cc:385] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.893271 13237 raft_consensus.cc:740] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.883049 13236 ts_tablet_manager.cc:1434] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.003s
I20260709 13:07:01.894590 13235 tablet_bootstrap.cc:492] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:01.895370 13235 ts_tablet_manager.cc:1403] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.010s
I20260709 13:07:01.894646 13237 consensus_queue.cc:260] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.896072 13236 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.898694 13235 raft_consensus.cc:359] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.899528 13235 raft_consensus.cc:385] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.899839 13235 raft_consensus.cc:740] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.900781 13235 consensus_queue.cc:260] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.901876 13237 ts_tablet_manager.cc:1434] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.001s
I20260709 13:07:01.902804 13242 raft_consensus.cc:493] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.903329 13242 raft_consensus.cc:515] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.906083 13242 leader_election.cc:290] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:01.911052 13236 tablet_bootstrap.cc:654] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.911309 13235 ts_tablet_manager.cc:1434] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260709 13:07:01.912351 13243 raft_consensus.cc:515] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.919801 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:01.921077 13235 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
W20260709 13:07:01.921757 13159 leader_election.cc:343] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:01.922930 13243 leader_election.cc:290] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:01.923476 13237 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:01.941425 13235 tablet_bootstrap.cc:654] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.941365 13237 tablet_bootstrap.cc:654] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:01.951795 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:01.952062 13236 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:01.952581 13046 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:01.952968 13236 ts_tablet_manager.cc:1403] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.057s	user 0.021s	sys 0.018s
I20260709 13:07:01.954058 13241 raft_consensus.cc:493] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.954622 13241 raft_consensus.cc:515] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.956758 13236 raft_consensus.cc:359] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.957612 13236 raft_consensus.cc:385] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.957973 13236 raft_consensus.cc:740] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.959228 13236 consensus_queue.cc:260] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.963587 13046 raft_consensus.cc:2468] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:01.965067 13160 leader_election.cc:304] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:01.966820 13243 raft_consensus.cc:2804] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:01.967398 13243 raft_consensus.cc:697] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:01.968370 13243 consensus_queue.cc:237] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.969276 13236 ts_tablet_manager.cc:1434] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 13:07:01.970160 13241 leader_election.cc:290] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:01.970736 13235 tablet_bootstrap.cc:492] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:01.971323 13235 ts_tablet_manager.cc:1403] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.051s	user 0.014s	sys 0.001s
I20260709 13:07:01.975425 13235 raft_consensus.cc:359] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.972330 13236 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:01.976534 13235 raft_consensus.cc:385] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:01.980855 13235 raft_consensus.cc:740] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:01.981050 13241 raft_consensus.cc:493] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.982116 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:01.981752 13241 raft_consensus.cc:515] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.981781 13235 consensus_queue.cc:260] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.982808 13195 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:01.984414 13243 raft_consensus.cc:493] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:01.984779 13241 leader_election.cc:290] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:01.984681 13009 leader_election.cc:304] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:01.985181 13243 raft_consensus.cc:515] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:01.985947 13242 raft_consensus.cc:2804] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:01.986366 13242 raft_consensus.cc:493] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:01.986738 13242 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:01.987742 12940 catalog_manager.cc:5697] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:01.989159 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:01.989936 13121 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
I20260709 13:07:01.993273 13243 leader_election.cc:290] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:01.993180 13159 leader_election.cc:304] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:01.993147 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.000478 13046 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
I20260709 13:07:02.002889 13243 raft_consensus.cc:2804] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.003472 13243 raft_consensus.cc:493] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.004127 13243 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.005695 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.006537 13121 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.009291 13242 raft_consensus.cc:515] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.010993 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.012101 13045 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:02.013068 13242 leader_election.cc:290] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.014417 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.014882 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.015326 13195 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.016243 13237 tablet_bootstrap.cc:492] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:02.016847 13237 ts_tablet_manager.cc:1403] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.094s	user 0.006s	sys 0.010s
I20260709 13:07:02.017309 13236 tablet_bootstrap.cc:654] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.017825 13244 raft_consensus.cc:493] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.018373 13244 raft_consensus.cc:515] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.014496 13086 leader_election.cc:304] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
W20260709 13:07:02.022975 13086 leader_election.cc:343] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.023304 13241 raft_consensus.cc:2804] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.024528 13241 raft_consensus.cc:493] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.024893 13241 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.025130 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.026293 13243 raft_consensus.cc:515] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.027489 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.028940 13244 leader_election.cc:290] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
W20260709 13:07:02.029919 13160 leader_election.cc:343] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.030903 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.031191 13243 leader_election.cc:290] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.031791 13046 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:02.033623 13159 leader_election.cc:343] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.037009 13159 leader_election.cc:304] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.038233 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.039413 13121 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.042165 13243 raft_consensus.cc:2749] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:02.046725 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.049285 13195 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.050873 13009 leader_election.cc:304] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.052182 13242 raft_consensus.cc:2804] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.066139 13235 ts_tablet_manager.cc:1434] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.094s	user 0.017s	sys 0.015s
I20260709 13:07:02.067659 13235 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:02.067943 13121 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.070529 13046 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:02.072121 13242 raft_consensus.cc:697] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:02.073208 13241 raft_consensus.cc:515] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.075897 13120 raft_consensus.cc:2393] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a788c7ceff54a7683cb674b33080008 in current term 1: Already voted for candidate 077c76b5bd474881905cb6cfe23c3124 in this term.
I20260709 13:07:02.075711 13160 leader_election.cc:304] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.073378 13242 consensus_queue.cc:237] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.077621 13237 raft_consensus.cc:359] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.078513 13237 raft_consensus.cc:385] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.076891 13236 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:02.076946 13243 raft_consensus.cc:2804] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.078871 13237 raft_consensus.cc:740] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.079773 13236 ts_tablet_manager.cc:1403] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.109s	user 0.024s	sys 0.031s
I20260709 13:07:02.080724 13243 raft_consensus.cc:697] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:02.080525 13237 consensus_queue.cc:260] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.082034 13243 consensus_queue.cc:237] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.083676 13237 ts_tablet_manager.cc:1434] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.066s	user 0.010s	sys 0.001s
I20260709 13:07:02.083446 13236 raft_consensus.cc:359] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.084244 13236 raft_consensus.cc:385] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.084524 13235 tablet_bootstrap.cc:654] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.084899 13237 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:02.084534 13236 raft_consensus.cc:740] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.086071 13236 consensus_queue.cc:260] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.088302 13236 ts_tablet_manager.cc:1434] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 13:07:02.089414 13236 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:02.088086 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.090432 13195 raft_consensus.cc:2468] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:02.090904 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.091697 13046 raft_consensus.cc:2393] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 077c76b5bd474881905cb6cfe23c3124 in current term 1: Already voted for candidate 8a788c7ceff54a7683cb674b33080008 in this term.
I20260709 13:07:02.092612 13241 leader_election.cc:290] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.092783 13084 leader_election.cc:304] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.089176 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.095418 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.097067 13237 tablet_bootstrap.cc:654] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.098969 13084 leader_election.cc:304] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124; no voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:02.099968 13241 raft_consensus.cc:2804] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.097067 13236 tablet_bootstrap.cc:654] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.100443 13241 raft_consensus.cc:493] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.100312 13266 raft_consensus.cc:2749] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:07:02.101086 13241 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.108453 13236 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:02.109046 13236 ts_tablet_manager.cc:1403] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.020s	user 0.006s	sys 0.007s
I20260709 13:07:02.109830 13237 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:02.116351 13237 ts_tablet_manager.cc:1403] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.032s	user 0.009s	sys 0.006s
I20260709 13:07:02.112457 13236 raft_consensus.cc:359] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.118618 13236 raft_consensus.cc:385] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.117951 12943 catalog_manager.cc:5697] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.118983 13236 raft_consensus.cc:740] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.120097 13236 consensus_queue.cc:260] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.120323 13237 raft_consensus.cc:359] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.121337 13237 raft_consensus.cc:385] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.121799 13237 raft_consensus.cc:740] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.122925 13236 ts_tablet_manager.cc:1434] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.005s
I20260709 13:07:02.122709 13237 consensus_queue.cc:260] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.125375 13237 ts_tablet_manager.cc:1434] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 13:07:02.126853 13237 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:02.126828 13243 raft_consensus.cc:493] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.127669 13243 raft_consensus.cc:515] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.131076 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.132432 13235 tablet_bootstrap.cc:492] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:02.133095 13235 ts_tablet_manager.cc:1403] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.066s	user 0.004s	sys 0.013s
I20260709 13:07:02.133299 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
W20260709 13:07:02.134699 13160 leader_election.cc:343] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:02.135957 13159 leader_election.cc:343] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.136416 13159 leader_election.cc:304] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.136772 13235 raft_consensus.cc:359] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.137339 13244 raft_consensus.cc:2749] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:02.137631 13235 raft_consensus.cc:385] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.137080 12940 catalog_manager.cc:5697] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:02.137969 13235 raft_consensus.cc:740] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.138234 13243 leader_election.cc:290] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.139067 13235 consensus_queue.cc:260] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.141834 13235 ts_tablet_manager.cc:1434] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.004s
I20260709 13:07:02.142861 13237 tablet_bootstrap.cc:654] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.143055 13235 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:02.151315 13237 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:02.151911 13237 ts_tablet_manager.cc:1403] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.000s
I20260709 13:07:02.152107 13235 tablet_bootstrap.cc:654] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.155649 13237 raft_consensus.cc:359] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.156497 13237 raft_consensus.cc:385] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.156786 13237 raft_consensus.cc:740] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.157706 13237 consensus_queue.cc:260] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.163582 13237 ts_tablet_manager.cc:1434] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 13:07:02.164757 13237 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:02.165817 13241 raft_consensus.cc:515] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.169263 13266 raft_consensus.cc:493] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.170107 13241 leader_election.cc:290] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.171452 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.172200 13046 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.172520 13237 tablet_bootstrap.cc:654] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.171476 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.174306 13195 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.170167 13266 raft_consensus.cc:515] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.184767 13266 leader_election.cc:290] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.186251 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.186798 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.187273 13194 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:02.187850 13045 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:02.188340 13237 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:02.188771 13243 raft_consensus.cc:493] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.189065 13237 ts_tablet_manager.cc:1403] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.007s
I20260709 13:07:02.189828 13086 leader_election.cc:304] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:02.189432 13243 raft_consensus.cc:515] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.191437 13266 raft_consensus.cc:2804] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.191991 13266 raft_consensus.cc:493] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.192466 13237 raft_consensus.cc:359] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.192862 13266 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.193220 13237 raft_consensus.cc:385] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.193583 13237 raft_consensus.cc:740] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.193881 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.194500 13237 consensus_queue.cc:260] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
W20260709 13:07:02.195775 13159 leader_election.cc:343] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.197288 13237 ts_tablet_manager.cc:1434] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:07:02.198428 13237 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:02.198837 13244 raft_consensus.cc:493] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.199344 13244 raft_consensus.cc:515] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.203694 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.203939 13044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.204479 13045 raft_consensus.cc:2468] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
I20260709 13:07:02.205304 13244 leader_election.cc:290] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.206595 13242 raft_consensus.cc:493] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.207109 13242 raft_consensus.cc:515] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.209450 13242 leader_election.cc:290] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
W20260709 13:07:02.205910 13160 leader_election.cc:343] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.210538 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.210855 13160 leader_election.cc:304] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
W20260709 13:07:02.212240 13159 leader_election.cc:343] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.212728 13159 leader_election.cc:304] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.214090 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.214892 13237 tablet_bootstrap.cc:654] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.214834 13194 raft_consensus.cc:2468] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.216691 13266 raft_consensus.cc:515] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.219205 13243 leader_election.cc:290] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.220597 13243 raft_consensus.cc:2804] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.221237 13243 raft_consensus.cc:493] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.221832 13243 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.223338 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.216717 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.224185 13045 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.224434 13266 leader_election.cc:290] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.225733 13244 raft_consensus.cc:2749] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:02.226325 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
W20260709 13:07:02.226984 13010 leader_election.cc:343] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.226977 13194 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.227617 13009 leader_election.cc:304] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.228997 13242 raft_consensus.cc:2804] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.229410 13242 raft_consensus.cc:493] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.229782 13242 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.235711 13045 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:02.237388 13242 raft_consensus.cc:515] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.237466 13086 leader_election.cc:304] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:02.238883 13266 raft_consensus.cc:2804] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.243764 13194 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:02.244379 13195 raft_consensus.cc:2468] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:02.223781 13235 tablet_bootstrap.cc:492] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:02.240581 13266 raft_consensus.cc:697] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:02.247963 13241 raft_consensus.cc:493] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.248764 13084 leader_election.cc:304] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.250275 13237 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:02.250728 13237 ts_tablet_manager.cc:1403] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.053s	user 0.014s	sys 0.004s
I20260709 13:07:02.253337 13242 leader_election.cc:290] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.251500 13235 ts_tablet_manager.cc:1403] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.109s	user 0.036s	sys 0.040s
I20260709 13:07:02.249379 13241 raft_consensus.cc:515] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.249209 13266 consensus_queue.cc:237] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.258065 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.258778 13194 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.255475 13046 raft_consensus.cc:2468] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
W20260709 13:07:02.257035 13159 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.194:39583, user_credentials={real_user=slave}} blocked reactor thread for 44837.1us
I20260709 13:07:02.267007 13284 raft_consensus.cc:2804] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.267377 13241 leader_election.cc:290] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.268852 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:02.268922 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.270030 13046 raft_consensus.cc:2393] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 077c76b5bd474881905cb6cfe23c3124 in current term 1: Already voted for candidate 8a788c7ceff54a7683cb674b33080008 in this term.
I20260709 13:07:02.269718 13243 raft_consensus.cc:515] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.271072 13235 raft_consensus.cc:359] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.271492 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.269387 13237 raft_consensus.cc:359] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
W20260709 13:07:02.272424 13010 leader_election.cc:343] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:02.275589 13243 leader_election.cc:290] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.276161 13237 raft_consensus.cc:385] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.276657 13237 raft_consensus.cc:740] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.278498 13237 consensus_queue.cc:260] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.279572 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.280896 13237 ts_tablet_manager.cc:1434] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.030s	user 0.009s	sys 0.000s
I20260709 13:07:02.276255 13235 raft_consensus.cc:385] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.281551 13194 raft_consensus.cc:2468] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.281863 13235 raft_consensus.cc:740] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.275799 13084 leader_election.cc:304] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124; no voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:02.283130 13009 leader_election.cc:304] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.284759 13242 raft_consensus.cc:2804] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.285046 13235 consensus_queue.cc:260] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.285538 13242 raft_consensus.cc:697] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:02.286155 13271 raft_consensus.cc:493] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.286743 13271 raft_consensus.cc:515] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.288074 13235 ts_tablet_manager.cc:1434] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.002s
I20260709 13:07:02.289292 13120 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.289561 13235 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:02.291039 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.291855 13046 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.296051 13241 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.295874 13242 consensus_queue.cc:237] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.296710 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.297525 13121 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.298921 13235 tablet_bootstrap.cc:654] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.299259 13271 leader_election.cc:290] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.300957 13010 leader_election.cc:304] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:02.301614 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.302695 13194 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.302132 13271 raft_consensus.cc:2804] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.307811 13271 raft_consensus.cc:493] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.313752 13271 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.314239 13120 raft_consensus.cc:2468] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:02.303653 13284 raft_consensus.cc:697] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:02.319187 13284 consensus_queue.cc:237] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.323623 13159 leader_election.cc:304] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.325423 13243 raft_consensus.cc:2804] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.326186 13243 raft_consensus.cc:697] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:02.327425 13243 consensus_queue.cc:237] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.324146 12942 catalog_manager.cc:5697] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.339131 13241 raft_consensus.cc:2749] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:02.340967 13235 tablet_bootstrap.cc:492] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:02.344537 12940 catalog_manager.cc:5697] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.346104 13235 ts_tablet_manager.cc:1403] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.057s	user 0.019s	sys 0.015s
I20260709 13:07:02.346315 13271 raft_consensus.cc:515] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.349953 13271 leader_election.cc:290] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.350865 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.351536 13194 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.352460 13235 raft_consensus.cc:359] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.353222 13235 raft_consensus.cc:385] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.353478 13235 raft_consensus.cc:740] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.355144 13046 raft_consensus.cc:2468] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:02.360054 12942 catalog_manager.cc:5697] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.361003 13235 consensus_queue.cc:260] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.363401 13235 ts_tablet_manager.cc:1434] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.001s
I20260709 13:07:02.364447 13235 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:02.365451 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.366175 13120 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.371897 13235 tablet_bootstrap.cc:654] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:02.371075 12941 catalog_manager.cc:5697] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:02.372491 13194 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.374018 13009 leader_election.cc:304] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.374971 13271 raft_consensus.cc:2804] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.378482 13235 tablet_bootstrap.cc:492] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:02.379089 13235 ts_tablet_manager.cc:1403] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.015s	user 0.009s	sys 0.003s
I20260709 13:07:02.382233 13235 raft_consensus.cc:359] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.382985 13235 raft_consensus.cc:385] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:02.383266 13235 raft_consensus.cc:740] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:02.384001 13235 consensus_queue.cc:260] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.375623 13271 raft_consensus.cc:697] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:02.386014 13271 consensus_queue.cc:237] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.390800 13235 ts_tablet_manager.cc:1434] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:07:02.393926 13120 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.399909 12940 catalog_manager.cc:5697] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.474149 13271 raft_consensus.cc:493] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:02.474813 13271 raft_consensus.cc:515] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.476950 13271 leader_election.cc:290] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.478173 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:02.478041 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:02.479061 13194 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.479212 13120 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:02.480336 13009 leader_election.cc:304] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:02.481317 13271 raft_consensus.cc:2804] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:02.481813 13271 raft_consensus.cc:493] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:02.482198 13271 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.484261 13243 consensus_queue.cc:1048] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.490231 13271 raft_consensus.cc:515] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.493149 13271 leader_election.cc:290] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:02.494414 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:02.494566 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.495260 13194 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.495419 13120 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.505445 13194 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.506260 13120 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:02.508096 13010 leader_election.cc:304] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:02.509156 13271 raft_consensus.cc:2804] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:02.509697 13271 raft_consensus.cc:697] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:02.510296 13243 consensus_queue.cc:1048] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.510598 13271 consensus_queue.cc:237] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.526940 12940 catalog_manager.cc:5697] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:02.531610 13271 consensus_queue.cc:1048] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.549294 13271 consensus_queue.cc:1048] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.667173 13244 consensus_queue.cc:1048] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:02.668030 13243 consensus_queue.cc:1048] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:02.683720 13284 consensus_queue.cc:1048] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.708650 13266 consensus_queue.cc:1048] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.755966 13266 consensus_queue.cc:1048] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:02.763109 13305 consensus_queue.cc:1048] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:02.777586 13285 consensus_queue.cc:1048] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.803114 13305 consensus_queue.cc:1048] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.806236 13121 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:02.815730 13300 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.821075 13300 consensus_queue.cc:1048] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.839011 13300 consensus_queue.cc:1048] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.872514 13300 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.877662 13121 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:35054
I20260709 13:07:02.878422 13121 raft_consensus.cc:606] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Received request to transfer leadership to TS 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.908433 13194 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
mode: GRACEFUL
new_leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:50788
I20260709 13:07:02.909030 13194 raft_consensus.cc:606] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Received request to transfer leadership to TS 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.916002 13120 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e60047ab21874e2086ee857dce35d837"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:35054
I20260709 13:07:02.916538 13120 raft_consensus.cc:606] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Received request to transfer leadership to TS 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:02.922654 13194 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:50788
I20260709 13:07:02.923429 13194 raft_consensus.cc:606] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.942929 13046 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:59910
I20260709 13:07:02.943784 13046 raft_consensus.cc:606] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.943768 13300 consensus_queue.cc:1048] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.951452 13194 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:50788
I20260709 13:07:02.952183 13194 raft_consensus.cc:606] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.960412 13046 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:59910
I20260709 13:07:02.960515 13271 consensus_queue.cc:1048] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:02.961220 13046 raft_consensus.cc:606] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:02.969229 13046 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:59910
I20260709 13:07:02.970147 13046 raft_consensus.cc:606] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:02.974135 13301 raft_consensus.cc:993] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:02.974666 13300 raft_consensus.cc:1081] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:02.976415 13194 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50762
I20260709 13:07:02.976979 13194 raft_consensus.cc:493] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:02.977313 13194 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:02.983853 13194 raft_consensus.cc:515] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.986128 13194 leader_election.cc:290] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:02.987407 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:02.988247 13120 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:02.987516 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8809f7896b03429fa2253088c87b8a71" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:02.989166 13046 raft_consensus.cc:3055] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:02.989557 13046 raft_consensus.cc:740] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:02.990880 13046 consensus_queue.cc:260] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:02.992940 13046 raft_consensus.cc:3060] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:02.998752 13120 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:03.000567 13159 leader_election.cc:304] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:03.001670 13305 raft_consensus.cc:2804] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.002167 13305 raft_consensus.cc:697] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:03.002276 13046 raft_consensus.cc:2468] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:03.003422 13305 consensus_queue.cc:237] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.011775 13243 raft_consensus.cc:993] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.012440 13305 raft_consensus.cc:1081] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.014364 13120 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35042
I20260709 13:07:03.014285 12942 catalog_manager.cc:5697] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:03.015187 13120 raft_consensus.cc:493] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.015625 13120 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.023463 13120 raft_consensus.cc:515] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.025821 13120 leader_election.cc:290] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.026434 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:03.026971 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12a7ada4b5544ef8891f296e167a3a84" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.027180 13046 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.027611 13194 raft_consensus.cc:3055] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.028074 13194 raft_consensus.cc:740] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:03.029323 13194 consensus_queue.cc:260] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.030941 13194 raft_consensus.cc:3060] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.036096 13046 raft_consensus.cc:2468] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.037667 13086 leader_election.cc:304] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:03.038538 13241 raft_consensus.cc:2804] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.038982 13194 raft_consensus.cc:2468] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.039104 13241 raft_consensus.cc:697] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.040951 13241 consensus_queue.cc:237] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.050222 12942 catalog_manager.cc:5697] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 8a788c7ceff54a7683cb674b33080008 (127.0.107.195) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.191422 13310 raft_consensus.cc:993] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.192152 13241 raft_consensus.cc:1081] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Signalling peer 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.194278 13046 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:59908
I20260709 13:07:03.194826 13046 raft_consensus.cc:493] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.195098 13046 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.201376 13046 raft_consensus.cc:515] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.204030 13046 leader_election.cc:290] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.205338 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:03.206156 13120 raft_consensus.cc:3055] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.206517 13120 raft_consensus.cc:740] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.206251 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e31761a5de5e4572b87141de0d90cba8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.207100 13194 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.207415 13120 consensus_queue.cc:260] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.208987 13120 raft_consensus.cc:3060] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.212249 13243 raft_consensus.cc:993] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: : Instructing follower 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.212829 13305 raft_consensus.cc:1081] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Signalling peer 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.214876 13046 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:59892
I20260709 13:07:03.215534 13046 raft_consensus.cc:493] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.215948 13046 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.216426 13120 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.216454 13194 raft_consensus.cc:2468] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.218165 13009 leader_election.cc:304] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:03.219172 13300 raft_consensus.cc:2804] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.219663 13300 raft_consensus.cc:697] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:03.220544 13300 consensus_queue.cc:237] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.224536 13046 raft_consensus.cc:515] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.226994 13046 leader_election.cc:290] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.228009 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:03.228727 13120 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.229414 12942 catalog_manager.cc:5697] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 1 to 2, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.231272 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bce188e6a2944fb0abd11b7fba0b46a7" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.233443 13194 raft_consensus.cc:3055] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.233886 13194 raft_consensus.cc:740] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:03.234822 13194 consensus_queue.cc:260] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.236297 13194 raft_consensus.cc:3060] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.238730 13120 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.240145 13010 leader_election.cc:304] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:03.241057 13300 raft_consensus.cc:2804] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.241578 13300 raft_consensus.cc:697] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:03.242504 13300 consensus_queue.cc:237] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.244001 13194 raft_consensus.cc:2468] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.250093 12942 catalog_manager.cc:5697] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 1 to 2, leader changed from 8a788c7ceff54a7683cb674b33080008 (127.0.107.195) to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.322045 13241 raft_consensus.cc:993] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.322578 13241 raft_consensus.cc:1081] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Signalling peer 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:03.324388 13046 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e60047ab21874e2086ee857dce35d837"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:59908
I20260709 13:07:03.324949 13046 raft_consensus.cc:493] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.325268 13046 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.331691 13046 raft_consensus.cc:515] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.333580 13046 leader_election.cc:290] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.335016 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.335319 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e60047ab21874e2086ee857dce35d837" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:03.335793 13194 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.335940 13120 raft_consensus.cc:3055] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.336341 13120 raft_consensus.cc:740] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.337127 13120 consensus_queue.cc:260] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.338256 13120 raft_consensus.cc:3060] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.341929 13194 raft_consensus.cc:2468] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.343418 13009 leader_election.cc:304] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:03.344228 13300 raft_consensus.cc:2804] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.344640 13300 raft_consensus.cc:697] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:03.345009 13120 raft_consensus.cc:2468] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:03.345495 13300 consensus_queue.cc:237] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.352572 12942 catalog_manager.cc:5697] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 1 to 2, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.369746 13301 raft_consensus.cc:993] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.370347 13300 raft_consensus.cc:1081] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.372421 13120 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35052
I20260709 13:07:03.373183 13120 raft_consensus.cc:493] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.373653 13120 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.380015 13120 raft_consensus.cc:515] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.382949 13120 leader_election.cc:290] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.384223 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.384186 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:03.385234 13194 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.385246 13046 raft_consensus.cc:3055] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.385957 13046 raft_consensus.cc:740] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:03.386893 13046 consensus_queue.cc:260] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.388305 13046 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.394343 13194 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.396063 13084 leader_election.cc:304] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:03.396561 13046 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.397075 13241 raft_consensus.cc:2804] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.397984 13241 raft_consensus.cc:697] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.398980 13241 consensus_queue.cc:237] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.402310 13305 raft_consensus.cc:993] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.402846 13305 raft_consensus.cc:1081] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.406157 13120 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35042
I20260709 13:07:03.406894 13120 raft_consensus.cc:493] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.407322 13120 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.409363 12942 catalog_manager.cc:5697] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.414970 13120 raft_consensus.cc:515] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.417078 13120 leader_election.cc:290] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.418148 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:03.418213 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62db6cb99508458e9640d9841aa1998d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.418985 13046 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.419134 13194 raft_consensus.cc:3055] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.419656 13194 raft_consensus.cc:740] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:03.420609 13194 consensus_queue.cc:260] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.421890 13194 raft_consensus.cc:3060] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.427201 13046 raft_consensus.cc:2468] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.427747 13045 raft_consensus.cc:1275] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.428601 13086 leader_election.cc:304] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:03.428957 13194 raft_consensus.cc:2468] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.429984 13241 raft_consensus.cc:2804] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.429852 13282 consensus_queue.cc:1048] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:03.431041 13241 raft_consensus.cc:697] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.432072 13241 consensus_queue.cc:237] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.436291 13301 raft_consensus.cc:993] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.436794 13301 raft_consensus.cc:1081] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:03.438879 13120 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35052
I20260709 13:07:03.439508 13120 raft_consensus.cc:493] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:03.439901 13120 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.442919 12941 catalog_manager.cc:5697] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 8a788c7ceff54a7683cb674b33080008 (127.0.107.195) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.445044 13121 raft_consensus.cc:1275] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.447011 13305 consensus_queue.cc:1048] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.450222 13120 raft_consensus.cc:515] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.454902 13120 leader_election.cc:290] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:03.456161 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:03.456656 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cbad72123464a59a5a17a82b5cabe04" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:03.457005 13194 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.457321 13046 raft_consensus.cc:3055] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:03.457748 13046 raft_consensus.cc:740] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:03.462591 13045 raft_consensus.cc:1275] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.463182 13046 consensus_queue.cc:260] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.464716 13241 consensus_queue.cc:1048] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.465160 13046 raft_consensus.cc:3060] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:03.467738 13194 raft_consensus.cc:2468] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.469046 13084 leader_election.cc:304] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:03.469973 13241 raft_consensus.cc:2804] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:03.470484 13241 raft_consensus.cc:697] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:03.471385 13241 consensus_queue.cc:237] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:03.476590 13046 raft_consensus.cc:2468] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:03.478494 13194 raft_consensus.cc:1275] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.480624 13284 consensus_queue.cc:1048] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:03.480549 12941 catalog_manager.cc:5697] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:03.693639 13120 raft_consensus.cc:1275] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.695385 13300 consensus_queue.cc:1048] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.709901 13194 raft_consensus.cc:1275] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.711918 13242 consensus_queue.cc:1048] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.712730 13194 raft_consensus.cc:1275] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.715988 13327 consensus_queue.cc:1048] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.725586 13120 raft_consensus.cc:1275] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.727464 13327 consensus_queue.cc:1048] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.751744 13194 raft_consensus.cc:1275] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.753172 13327 consensus_queue.cc:1048] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.762591 13120 raft_consensus.cc:1275] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.764525 13300 consensus_queue.cc:1048] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.827271 13046 raft_consensus.cc:1275] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.828910 13284 consensus_queue.cc:1048] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.839542 13194 raft_consensus.cc:1275] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.841318 13325 consensus_queue.cc:1048] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:03.885387 13194 raft_consensus.cc:1275] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.886797 13325 consensus_queue.cc:1048] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:03.898639 13046 raft_consensus.cc:1275] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.901221 13325 consensus_queue.cc:1048] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:03.990932 13194 raft_consensus.cc:1275] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:03.992717 13325 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:04.005414 13046 raft_consensus.cc:1275] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:04.008610 13325 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:06.005389 12942 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54668:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:06.007905 12942 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:06.066663 13031 tablet_service.cc:1511] Processing CreateTablet for tablet e98e42e324104a14aaf360c3333581a5 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:06.067528 13110 tablet_service.cc:1511] Processing CreateTablet for tablet e98e42e324104a14aaf360c3333581a5 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:06.068226 13031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98e42e324104a14aaf360c3333581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.068917 13110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98e42e324104a14aaf360c3333581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.069751 13183 tablet_service.cc:1511] Processing CreateTablet for tablet e98e42e324104a14aaf360c3333581a5 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:06.071082 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e98e42e324104a14aaf360c3333581a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.071496 13033 tablet_service.cc:1511] Processing CreateTablet for tablet cbd3163b2f7b4547a520f8e81ec43eef (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:06.073269 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd3163b2f7b4547a520f8e81ec43eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.085698 13109 tablet_service.cc:1511] Processing CreateTablet for tablet cbd3163b2f7b4547a520f8e81ec43eef (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:06.087286 13109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd3163b2f7b4547a520f8e81ec43eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.089787 13030 tablet_service.cc:1511] Processing CreateTablet for tablet 810e036588a6404c8282f7cc58fcf780 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:06.091588 13030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810e036588a6404c8282f7cc58fcf780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.100446 13184 tablet_service.cc:1511] Processing CreateTablet for tablet cbd3163b2f7b4547a520f8e81ec43eef (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:06.102166 13184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbd3163b2f7b4547a520f8e81ec43eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.126058 13351 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.126058 13352 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.137809 13183 tablet_service.cc:1511] Processing CreateTablet for tablet 810e036588a6404c8282f7cc58fcf780 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:06.139024 13033 tablet_service.cc:1511] Processing CreateTablet for tablet b138505857d34311a9efe1d10c311b0d (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:06.139856 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810e036588a6404c8282f7cc58fcf780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.140416 13033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b138505857d34311a9efe1d10c311b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.140918 13110 tablet_service.cc:1511] Processing CreateTablet for tablet b138505857d34311a9efe1d10c311b0d (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:06.142701 13110 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b138505857d34311a9efe1d10c311b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.145782 13108 tablet_service.cc:1511] Processing CreateTablet for tablet 1226828512554b5b9c417a4830456199 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:06.144897 13030 tablet_service.cc:1511] Processing CreateTablet for tablet 1226828512554b5b9c417a4830456199 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:06.147217 13108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1226828512554b5b9c417a4830456199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.147295 13035 tablet_service.cc:1511] Processing CreateTablet for tablet 703c89ef8f234b46b4ede95403b2ff13 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:06.151089 13034 tablet_service.cc:1511] Processing CreateTablet for tablet 779ad34d591e489da454a3608a4c32c8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:06.151240 13107 tablet_service.cc:1511] Processing CreateTablet for tablet 703c89ef8f234b46b4ede95403b2ff13 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:06.152577 13107 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703c89ef8f234b46b4ede95403b2ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.154994 13181 tablet_service.cc:1511] Processing CreateTablet for tablet b138505857d34311a9efe1d10c311b0d (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:06.156574 13181 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b138505857d34311a9efe1d10c311b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.156837 13182 tablet_service.cc:1511] Processing CreateTablet for tablet 1226828512554b5b9c417a4830456199 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:06.158180 13182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1226828512554b5b9c417a4830456199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.158643 13032 tablet_service.cc:1511] Processing CreateTablet for tablet d33b7a21125f4669ae2ca2fc148095dd (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:06.161599 13111 tablet_service.cc:1511] Processing CreateTablet for tablet 779ad34d591e489da454a3608a4c32c8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:06.161871 13185 tablet_service.cc:1511] Processing CreateTablet for tablet 703c89ef8f234b46b4ede95403b2ff13 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:06.163081 13111 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779ad34d591e489da454a3608a4c32c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.164234 13180 tablet_service.cc:1511] Processing CreateTablet for tablet 779ad34d591e489da454a3608a4c32c8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:06.163081 13185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703c89ef8f234b46b4ede95403b2ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.167732 13106 tablet_service.cc:1511] Processing CreateTablet for tablet d33b7a21125f4669ae2ca2fc148095dd (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:06.169302 13106 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33b7a21125f4669ae2ca2fc148095dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.171551 13179 tablet_service.cc:1511] Processing CreateTablet for tablet d33b7a21125f4669ae2ca2fc148095dd (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:06.173135 13180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779ad34d591e489da454a3608a4c32c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.174431 13179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33b7a21125f4669ae2ca2fc148095dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.136992 13109 tablet_service.cc:1511] Processing CreateTablet for tablet 810e036588a6404c8282f7cc58fcf780 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:06.186898 13109 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810e036588a6404c8282f7cc58fcf780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.194728 13351 tablet_bootstrap.cc:654] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.136804 13353 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.147228 13030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1226828512554b5b9c417a4830456199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.210099 13352 tablet_bootstrap.cc:654] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.214737 13032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d33b7a21125f4669ae2ca2fc148095dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.219257 13351 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.220075 13351 ts_tablet_manager.cc:1403] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.094s	user 0.010s	sys 0.006s
I20260709 13:07:06.221748 13034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 779ad34d591e489da454a3608a4c32c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.224653 13351 raft_consensus.cc:359] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.225780 13351 raft_consensus.cc:385] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.226256 13351 raft_consensus.cc:740] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.229784 13031 tablet_service.cc:1511] Processing CreateTablet for tablet 656aa2e5a56f4331ad72b9328522d7a8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:06.231045 13105 tablet_service.cc:1511] Processing CreateTablet for tablet 656aa2e5a56f4331ad72b9328522d7a8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:06.225888 13035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 703c89ef8f234b46b4ede95403b2ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.232574 13105 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa2e5a56f4331ad72b9328522d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.233150 13031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa2e5a56f4331ad72b9328522d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.241907 13351 consensus_queue.cc:260] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.254807 13183 tablet_service.cc:1511] Processing CreateTablet for tablet 656aa2e5a56f4331ad72b9328522d7a8 (DEFAULT_TABLE table=soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:06.257303 13183 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 656aa2e5a56f4331ad72b9328522d7a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:06.265240 13351 ts_tablet_manager.cc:1434] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.001s
I20260709 13:07:06.307261 13351 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.258595 13353 tablet_bootstrap.cc:654] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:06.312450 13009 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45349, user_credentials={real_user=slave}} blocked reactor thread for 70679us
I20260709 13:07:06.375026 13352 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.377465 13352 ts_tablet_manager.cc:1403] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.252s	user 0.016s	sys 0.018s
I20260709 13:07:06.387527 13351 tablet_bootstrap.cc:654] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:06.390035 13084 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45349, user_credentials={real_user=slave}} blocked reactor thread for 61213.9us
I20260709 13:07:06.387630 13352 raft_consensus.cc:359] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.395097 13352 raft_consensus.cc:385] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.400342 13352 raft_consensus.cc:740] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.396960 13353 tablet_bootstrap.cc:492] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.402123 13353 ts_tablet_manager.cc:1403] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.266s	user 0.015s	sys 0.012s
I20260709 13:07:06.406031 13352 consensus_queue.cc:260] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.410552 13352 ts_tablet_manager.cc:1434] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.033s	user 0.009s	sys 0.003s
I20260709 13:07:06.412767 13352 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.406510 13353 raft_consensus.cc:359] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.414551 13353 raft_consensus.cc:385] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.414947 13353 raft_consensus.cc:740] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.415807 13353 consensus_queue.cc:260] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.418718 13353 ts_tablet_manager.cc:1434] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 13:07:06.419898 13353 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.424283 13351 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.425031 13351 ts_tablet_manager.cc:1403] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.118s	user 0.014s	sys 0.017s
I20260709 13:07:06.428272 13353 tablet_bootstrap.cc:654] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.428467 13351 raft_consensus.cc:359] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.429461 13351 raft_consensus.cc:385] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.429849 13351 raft_consensus.cc:740] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.430922 13351 consensus_queue.cc:260] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.436184 13351 ts_tablet_manager.cc:1434] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:07:06.437383 13351 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.438828 13359 raft_consensus.cc:493] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.439713 13359 raft_consensus.cc:515] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.442500 13352 tablet_bootstrap.cc:654] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.444950 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:06.446010 13046 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:06.447299 13351 tablet_bootstrap.cc:654] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.448901 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:06.449965 13195 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:06.451617 13086 leader_election.cc:304] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:06.453254 13359 leader_election.cc:290] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.454126 13359 raft_consensus.cc:2804] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.454568 13359 raft_consensus.cc:493] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.454902 13359 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.460636 13352 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.461611 13352 ts_tablet_manager.cc:1403] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.049s	user 0.018s	sys 0.005s
I20260709 13:07:06.466954 13352 raft_consensus.cc:359] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.468148 13352 raft_consensus.cc:385] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.468573 13352 raft_consensus.cc:740] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.470122 13352 consensus_queue.cc:260] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.474007 13352 ts_tablet_manager.cc:1434] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.001s
I20260709 13:07:06.477227 13352 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.478991 13351 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.479786 13351 ts_tablet_manager.cc:1403] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.043s	user 0.013s	sys 0.025s
I20260709 13:07:06.483915 13351 raft_consensus.cc:359] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.484920 13351 raft_consensus.cc:385] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.485321 13351 raft_consensus.cc:740] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.486586 13351 consensus_queue.cc:260] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.486570 13352 tablet_bootstrap.cc:654] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.493451 13351 ts_tablet_manager.cc:1434] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 13:07:06.493891 13359 raft_consensus.cc:515] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.494951 13351 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.503003 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:06.504027 13046 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.505897 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:06.506848 13195 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.504379 13352 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.498168 13359 leader_election.cc:290] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.514693 13351 tablet_bootstrap.cc:654] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.517027 13353 tablet_bootstrap.cc:492] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.517849 13353 ts_tablet_manager.cc:1403] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.098s	user 0.030s	sys 0.046s
I20260709 13:07:06.519176 13046 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:06.520328 13195 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:06.520968 13086 leader_election.cc:304] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:06.522488 13359 raft_consensus.cc:2804] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:06.521721 13352 ts_tablet_manager.cc:1403] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.045s	user 0.013s	sys 0.007s
I20260709 13:07:06.522552 13353 raft_consensus.cc:359] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.523128 13359 raft_consensus.cc:697] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:06.523470 13353 raft_consensus.cc:385] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.523785 13353 raft_consensus.cc:740] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.524152 13359 consensus_queue.cc:237] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.524583 13353 consensus_queue.cc:260] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.526811 13352 raft_consensus.cc:359] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.527457 13353 ts_tablet_manager.cc:1434] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 13:07:06.527554 13352 raft_consensus.cc:385] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.528007 13352 raft_consensus.cc:740] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.528818 13353 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.528935 13352 consensus_queue.cc:260] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.532078 13352 ts_tablet_manager.cc:1434] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:07:06.533222 13352 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.539177 13353 tablet_bootstrap.cc:654] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.540370 13352 tablet_bootstrap.cc:654] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.553848 12941 catalog_manager.cc:5697] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:06.555711 13351 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.556320 13351 ts_tablet_manager.cc:1403] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.062s	user 0.012s	sys 0.007s
I20260709 13:07:06.559983 13353 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.559787 13351 raft_consensus.cc:359] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.560491 13351 raft_consensus.cc:385] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.560789 13351 raft_consensus.cc:740] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.560652 13353 ts_tablet_manager.cc:1403] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.032s	user 0.020s	sys 0.001s
I20260709 13:07:06.561792 13351 consensus_queue.cc:260] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.563815 13352 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.564464 13352 ts_tablet_manager.cc:1403] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.004s
I20260709 13:07:06.565047 13353 raft_consensus.cc:359] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.566339 13353 raft_consensus.cc:385] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.566540 13351 ts_tablet_manager.cc:1434] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 13:07:06.566684 13353 raft_consensus.cc:740] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.568317 13353 consensus_queue.cc:260] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.568774 13351 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.571301 13353 ts_tablet_manager.cc:1434] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 13:07:06.572975 13353 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.569527 13352 raft_consensus.cc:359] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.574020 13352 raft_consensus.cc:385] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.574466 13352 raft_consensus.cc:740] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.575852 13352 consensus_queue.cc:260] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.579140 13352 ts_tablet_manager.cc:1434] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 13:07:06.580556 13352 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.582681 13353 tablet_bootstrap.cc:654] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.582917 13351 tablet_bootstrap.cc:654] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.590875 13352 tablet_bootstrap.cc:654] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.595808 13351 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.596443 13351 ts_tablet_manager.cc:1403] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.009s
I20260709 13:07:06.610726 13351 raft_consensus.cc:359] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.611869 13351 raft_consensus.cc:385] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.612246 13351 raft_consensus.cc:740] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.613482 13352 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.614192 13353 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.614216 13352 ts_tablet_manager.cc:1403] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.015s
I20260709 13:07:06.615260 13353 ts_tablet_manager.cc:1403] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.042s	user 0.029s	sys 0.008s
I20260709 13:07:06.617432 13351 consensus_queue.cc:260] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.618399 13352 raft_consensus.cc:359] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.619179 13352 raft_consensus.cc:385] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.619445 13352 raft_consensus.cc:740] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.620287 13352 consensus_queue.cc:260] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.622936 13352 ts_tablet_manager.cc:1434] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 13:07:06.622730 13353 raft_consensus.cc:359] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.623574 13353 raft_consensus.cc:385] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.623896 13353 raft_consensus.cc:740] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.624368 13352 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.624848 13351 ts_tablet_manager.cc:1434] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.028s	user 0.012s	sys 0.000s
I20260709 13:07:06.625386 13353 consensus_queue.cc:260] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.626123 13351 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.632774 13352 tablet_bootstrap.cc:654] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.636318 13353 ts_tablet_manager.cc:1434] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 13:07:06.637998 13353 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.646833 13353 tablet_bootstrap.cc:654] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.652884 13351 tablet_bootstrap.cc:654] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.660820 13353 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.661698 13353 ts_tablet_manager.cc:1403] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.024s	user 0.023s	sys 0.001s
I20260709 13:07:06.665212 13352 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.666109 13353 raft_consensus.cc:359] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.666957 13353 raft_consensus.cc:385] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.667167 13352 ts_tablet_manager.cc:1403] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.043s	user 0.020s	sys 0.012s
I20260709 13:07:06.667309 13353 raft_consensus.cc:740] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.669620 13351 tablet_bootstrap.cc:492] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.670010 13353 consensus_queue.cc:260] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.670374 13351 ts_tablet_manager.cc:1403] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.045s	user 0.023s	sys 0.003s
I20260709 13:07:06.670989 13352 raft_consensus.cc:359] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.672194 13352 raft_consensus.cc:385] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.672664 13352 raft_consensus.cc:740] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.673610 13352 consensus_queue.cc:260] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.674587 13353 ts_tablet_manager.cc:1434] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20260709 13:07:06.676380 13352 ts_tablet_manager.cc:1434] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20260709 13:07:06.683244 13353 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.683337 13352 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.692037 13352 tablet_bootstrap.cc:654] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.674455 13351 raft_consensus.cc:359] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.693241 13356 raft_consensus.cc:493] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.694391 13356 raft_consensus.cc:515] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.696506 13353 tablet_bootstrap.cc:654] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.698486 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "810e036588a6404c8282f7cc58fcf780" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:06.699215 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "810e036588a6404c8282f7cc58fcf780" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:06.700014 13195 raft_consensus.cc:2468] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:06.700733 13356 leader_election.cc:290] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.701411 13351 raft_consensus.cc:385] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.701723 13009 leader_election.cc:304] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.701872 13351 raft_consensus.cc:740] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.703452 13361 raft_consensus.cc:493] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.703167 13351 consensus_queue.cc:260] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.704002 13361 raft_consensus.cc:515] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.704939 13352 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.705029 13120 raft_consensus.cc:2468] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:06.705717 13352 ts_tablet_manager.cc:1403] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.004s
I20260709 13:07:06.706079 13351 ts_tablet_manager.cc:1434] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.035s	user 0.003s	sys 0.006s
I20260709 13:07:06.707347 13351 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:06.709465 13352 raft_consensus.cc:359] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.711086 13352 raft_consensus.cc:385] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.711544 13352 raft_consensus.cc:740] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.712648 13352 consensus_queue.cc:260] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.714594 13300 raft_consensus.cc:493] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.715209 13351 tablet_bootstrap.cc:654] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.715174 13300 raft_consensus.cc:515] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.715922 13352 ts_tablet_manager.cc:1434] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 13:07:06.718205 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:06.718930 13120 raft_consensus.cc:2468] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
I20260709 13:07:06.725097 13336 raft_consensus.cc:493] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.726092 13336 raft_consensus.cc:515] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.729283 13336 leader_election.cc:290] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:06.732300 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:06.732832 13352 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.738305 13300 leader_election.cc:290] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.738545 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:06.732362 13159 leader_election.cc:304] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.740316 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
W20260709 13:07:06.741825 13159 leader_election.cc:343] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.741986 13361 leader_election.cc:290] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:06.742133 13356 raft_consensus.cc:2804] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.745618 13356 raft_consensus.cc:493] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.748134 13356 raft_consensus.cc:3060] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.749567 13336 raft_consensus.cc:2804] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.750166 13336 raft_consensus.cc:493] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.752499 13352 tablet_bootstrap.cc:654] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.754334 13336 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
W20260709 13:07:06.760646 13160 leader_election.cc:343] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.762631 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd3163b2f7b4547a520f8e81ec43eef" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:06.763643 13353 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.763489 13195 raft_consensus.cc:2468] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
W20260709 13:07:06.767074 13160 leader_election.cc:343] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.767342 13353 ts_tablet_manager.cc:1403] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.084s	user 0.010s	sys 0.005s
I20260709 13:07:06.768321 13160 leader_election.cc:304] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:06.771629 13361 raft_consensus.cc:2749] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:06.772420 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd3163b2f7b4547a520f8e81ec43eef" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:06.773393 13120 raft_consensus.cc:2468] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:06.773113 13353 raft_consensus.cc:359] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.773823 13009 leader_election.cc:304] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.774124 13353 raft_consensus.cc:385] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.774531 13353 raft_consensus.cc:740] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.775344 13353 consensus_queue.cc:260] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.778241 13353 ts_tablet_manager.cc:1434] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 13:07:06.778550 13351 tablet_bootstrap.cc:492] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.779222 13351 ts_tablet_manager.cc:1403] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.072s	user 0.021s	sys 0.003s
I20260709 13:07:06.780552 13353 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.775132 13380 raft_consensus.cc:2804] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.782759 13380 raft_consensus.cc:493] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.783233 13380 raft_consensus.cc:3060] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.788709 13336 raft_consensus.cc:515] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.793592 13356 raft_consensus.cc:515] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.795256 13380 raft_consensus.cc:515] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.797576 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "810e036588a6404c8282f7cc58fcf780" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:06.790912 13353 tablet_bootstrap.cc:654] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.798404 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:06.799108 13121 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.800210 13356 leader_election.cc:290] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.800390 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "810e036588a6404c8282f7cc58fcf780" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:06.801048 13195 raft_consensus.cc:3060] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.798332 13352 tablet_bootstrap.cc:492] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.802202 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd3163b2f7b4547a520f8e81ec43eef" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:06.802582 13352 ts_tablet_manager.cc:1403] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.085s	user 0.018s	sys 0.000s
I20260709 13:07:06.802942 13119 raft_consensus.cc:3060] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.795650 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:06.806393 13352 raft_consensus.cc:359] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.807159 13352 raft_consensus.cc:385] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.807462 13352 raft_consensus.cc:740] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.791813 13351 raft_consensus.cc:359] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.798225 13120 raft_consensus.cc:3060] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.809087 13351 raft_consensus.cc:385] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.809613 13351 raft_consensus.cc:740] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.811081 13195 raft_consensus.cc:2468] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:06.811234 13351 consensus_queue.cc:260] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
W20260709 13:07:06.808256 13160 leader_election.cc:343] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.808231 13352 consensus_queue.cc:260] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.814482 13009 leader_election.cc:304] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.816246 13352 ts_tablet_manager.cc:1434] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260709 13:07:06.817561 13352 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: Bootstrap starting.
I20260709 13:07:06.819950 13120 raft_consensus.cc:2468] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:06.827864 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cbd3163b2f7b4547a520f8e81ec43eef" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:06.828545 13195 raft_consensus.cc:3060] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.825389 13119 raft_consensus.cc:2468] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:06.831033 13010 leader_election.cc:304] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:06.832638 13381 raft_consensus.cc:2804] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:06.833364 13381 raft_consensus.cc:697] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:06.834257 13381 consensus_queue.cc:237] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.843160 13195 raft_consensus.cc:2468] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:06.826464 13380 leader_election.cc:290] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.827148 13356 raft_consensus.cc:2804] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:06.825336 13352 tablet_bootstrap.cc:654] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.846616 13351 ts_tablet_manager.cc:1434] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.067s	user 0.009s	sys 0.003s
I20260709 13:07:06.796684 13336 leader_election.cc:290] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:06.857018 13356 raft_consensus.cc:697] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:06.858209 13356 consensus_queue.cc:237] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.862428 12941 catalog_manager.cc:5697] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:06.864087 13336 raft_consensus.cc:493] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.864713 13336 raft_consensus.cc:515] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.868080 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:06.869154 13336 leader_election.cc:290] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:06.870252 13351 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
W20260709 13:07:06.870931 13160 leader_election.cc:343] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.872347 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:06.873466 13119 raft_consensus.cc:2468] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 0.
I20260709 13:07:06.879565 13159 leader_election.cc:304] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:06.880870 13336 raft_consensus.cc:2804] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.881268 13336 raft_consensus.cc:493] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.881532 13359 raft_consensus.cc:493] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.881673 13336 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.882243 13359 raft_consensus.cc:515] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.884680 13359 leader_election.cc:290] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.886055 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:06.886550 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:06.887317 13195 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:06.887199 13045 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:06.888578 13381 raft_consensus.cc:493] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:06.889113 13381 raft_consensus.cc:515] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.889295 13084 leader_election.cc:304] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.890630 13359 raft_consensus.cc:2804] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.891039 13359 raft_consensus.cc:493] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:06.891448 13359 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.891134 13336 raft_consensus.cc:515] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.891537 13381 leader_election.cc:290] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.893407 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:06.894215 13195 raft_consensus.cc:2468] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:06.894588 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:06.895728 13009 leader_election.cc:304] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:06.896909 13381 raft_consensus.cc:2804] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:06.897310 13381 raft_consensus.cc:493] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 13:07:06.897460 13160 leader_election.cc:343] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.897697 13381 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.898270 13120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:06.898919 13120 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.899753 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
W20260709 13:07:06.901386 13010 leader_election.cc:343] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.906456 13120 raft_consensus.cc:2468] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:06.906296 13336 leader_election.cc:290] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:06.843851 13121 raft_consensus.cc:2468] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 1.
I20260709 13:07:06.909191 13159 leader_election.cc:304] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:06.915304 12941 catalog_manager.cc:5697] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:06.917850 13353 tablet_bootstrap.cc:492] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:06.918540 13353 ts_tablet_manager.cc:1403] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.138s	user 0.036s	sys 0.055s
I20260709 13:07:06.921676 13159 leader_election.cc:304] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:06.925036 13361 raft_consensus.cc:2804] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:06.925426 13336 raft_consensus.cc:2804] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:06.925810 13353 raft_consensus.cc:359] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.927099 13353 raft_consensus.cc:385] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.927525 13353 raft_consensus.cc:740] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.928344 13353 consensus_queue.cc:260] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
W20260709 13:07:06.945366 12992 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:06.947814 13359 raft_consensus.cc:515] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.946118 13353 ts_tablet_manager.cc:1434] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.005s
I20260709 13:07:06.947234 13381 raft_consensus.cc:515] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.949898 13351 tablet_bootstrap.cc:654] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.954433 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:06.955395 13353 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:06.955931 13352 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: No bootstrap required, opened a new log
I20260709 13:07:06.956570 13352 ts_tablet_manager.cc:1403] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: Time spent bootstrapping tablet: real 0.139s	user 0.028s	sys 0.010s
W20260709 13:07:06.956486 13010 leader_election.cc:343] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:06.963382 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:06.963982 13353 tablet_bootstrap.cc:654] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:06.964155 13195 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.974995 13359 leader_election.cc:290] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.971067 13361 raft_consensus.cc:697] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:06.976184 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:06.984779 13352 raft_consensus.cc:359] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.966724 13381 leader_election.cc:290] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:06.976872 13045 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.997996 13352 raft_consensus.cc:385] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:06.998456 13352 raft_consensus.cc:740] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Initialized, Role: FOLLOWER
I20260709 13:07:06.999527 13352 consensus_queue.cc:260] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.991117 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.002210 13194 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:06.976859 13361 consensus_queue.cc:237] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.002660 13352 ts_tablet_manager.cc:1434] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: Time spent starting tablet: real 0.046s	user 0.013s	sys 0.000s
I20260709 13:07:06.992244 13336 raft_consensus.cc:697] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:06.992986 13359 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.004501 13351 tablet_bootstrap.cc:492] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:06.996663 13353 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:07.005888 13353 ts_tablet_manager.cc:1403] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.051s	user 0.018s	sys 0.020s
I20260709 13:07:07.006803 13351 ts_tablet_manager.cc:1403] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.137s	user 0.014s	sys 0.010s
I20260709 13:07:07.009586 13353 raft_consensus.cc:359] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.011442 13353 raft_consensus.cc:385] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:07.011831 13353 raft_consensus.cc:740] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:07.011911 13359 raft_consensus.cc:493] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:07.012624 13359 raft_consensus.cc:515] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.014721 13351 raft_consensus.cc:359] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:06.996573 13195 raft_consensus.cc:2468] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:07.014145 13336 consensus_queue.cc:237] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.015479 13351 raft_consensus.cc:385] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:07.012777 13353 consensus_queue.cc:260] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.015805 13351 raft_consensus.cc:740] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:07.016681 13044 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:07.016654 13351 consensus_queue.cc:260] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.017746 13353 ts_tablet_manager.cc:1434] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.001s
I20260709 13:07:07.017642 13044 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:07.018975 13353 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Bootstrap starting.
I20260709 13:07:07.019745 13009 leader_election.cc:304] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:07.022053 13386 raft_consensus.cc:2804] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:07.022362 13351 ts_tablet_manager.cc:1434] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 13:07:07.022601 13386 raft_consensus.cc:697] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.023734 13351 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: Bootstrap starting.
I20260709 13:07:07.023851 13386 consensus_queue.cc:237] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.024680 13045 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:07.033795 13086 leader_election.cc:304] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.034085 13194 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:07.035079 13086 leader_election.cc:304] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.036415 13358 raft_consensus.cc:2804] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:07.036813 13358 raft_consensus.cc:493] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:07.037166 13358 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.028241 13353 tablet_bootstrap.cc:654] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:07.043177 13351 tablet_bootstrap.cc:654] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:07.044329 13359 leader_election.cc:290] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.049001 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:07.049770 13194 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 0.
I20260709 13:07:07.052877 13359 raft_consensus.cc:2804] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:07.053861 13359 raft_consensus.cc:697] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.057364 13356 raft_consensus.cc:493] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:07.056178 13359 consensus_queue.cc:237] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.058229 13356 raft_consensus.cc:515] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.062844 13351 tablet_bootstrap.cc:492] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: No bootstrap required, opened a new log
I20260709 13:07:07.063473 13351 ts_tablet_manager.cc:1403] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: Time spent bootstrapping tablet: real 0.040s	user 0.011s	sys 0.005s
I20260709 13:07:07.076843 13351 raft_consensus.cc:359] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.079741 13351 raft_consensus.cc:385] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:07.080901 13351 raft_consensus.cc:740] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Initialized, Role: FOLLOWER
I20260709 13:07:07.086292 13351 consensus_queue.cc:260] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.101992 13356 leader_election.cc:290] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.104476 13353 tablet_bootstrap.cc:492] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: No bootstrap required, opened a new log
I20260709 13:07:07.105274 13353 ts_tablet_manager.cc:1403] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent bootstrapping tablet: real 0.087s	user 0.028s	sys 0.026s
I20260709 13:07:07.105238 12942 catalog_manager.cc:5697] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:07.104437 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:07.107911 13121 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:07.108582 13358 raft_consensus.cc:515] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.123219 13194 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:07.124328 13194 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 0.
I20260709 13:07:07.134352 13356 raft_consensus.cc:493] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:07.134513 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:07.135281 13046 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.135221 13356 raft_consensus.cc:515] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.137652 13356 leader_election.cc:290] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.139204 13358 leader_election.cc:290] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.146860 13046 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:07.149628 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.150460 13195 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.151206 12941 catalog_manager.cc:5697] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.152222 13358 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.113821 13010 leader_election.cc:304] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.154196 13356 raft_consensus.cc:2804] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:07.154605 13356 raft_consensus.cc:493] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:07.154953 13356 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.156001 13086 leader_election.cc:304] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.156262 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:07.157153 13121 raft_consensus.cc:2393] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb587cb63854c42abf5c4f501ee32b3 in current term 1: Already voted for candidate 8a788c7ceff54a7683cb674b33080008 in this term.
I20260709 13:07:07.171595 13351 ts_tablet_manager.cc:1434] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: Time spent starting tablet: real 0.108s	user 0.051s	sys 0.010s
I20260709 13:07:07.166709 13358 raft_consensus.cc:2804] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:07.181579 13358 raft_consensus.cc:697] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.181707 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:07.182708 13358 consensus_queue.cc:237] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.188954 13356 raft_consensus.cc:515] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.191403 13009 leader_election.cc:304] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3; no voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:07.193329 13381 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.194350 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:07.195399 13121 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.201735 12942 catalog_manager.cc:5697] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:07.203851 12943 catalog_manager.cc:5697] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.208990 13121 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:07.211681 13356 leader_election.cc:290] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.213470 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.218003 13010 leader_election.cc:304] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.214936 13191 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.222394 13356 raft_consensus.cc:2804] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:07.234043 13356 raft_consensus.cc:697] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.235498 13356 consensus_queue.cc:237] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.244024 13191 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 1.
I20260709 13:07:07.246908 13386 consensus_queue.cc:1048] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.241715 13195 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 1.
I20260709 13:07:07.251542 13353 raft_consensus.cc:359] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.255858 13353 raft_consensus.cc:385] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:07.256194 13353 raft_consensus.cc:740] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Initialized, Role: FOLLOWER
I20260709 13:07:07.256981 13353 consensus_queue.cc:260] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.259459 13353 ts_tablet_manager.cc:1434] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Time spent starting tablet: real 0.154s	user 0.007s	sys 0.000s
I20260709 13:07:07.262202 13381 raft_consensus.cc:2749] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:07.262427 13300 consensus_queue.cc:1048] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:07.263990 12943 catalog_manager.cc:5697] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 0 to 1, leader changed from <none> to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 1 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.286053 12941 catalog_manager.cc:5697] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.296418 13405 raft_consensus.cc:493] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:07.297158 13405 raft_consensus.cc:515] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.299549 13405 leader_election.cc:290] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.301287 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3" is_pre_election: true
I20260709 13:07:07.301352 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:07.303619 13191 raft_consensus.cc:2393] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 077c76b5bd474881905cb6cfe23c3124 in current term 1: Already voted for candidate 3fb587cb63854c42abf5c4f501ee32b3 in this term.
I20260709 13:07:07.304801 13084 leader_election.cc:304] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124; no voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:07.305794 13405 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.313196 13386 raft_consensus.cc:493] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:07.313908 13386 raft_consensus.cc:515] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.316977 13386 leader_election.cc:290] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.316898 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "077c76b5bd474881905cb6cfe23c3124" is_pre_election: true
I20260709 13:07:07.317183 13405 raft_consensus.cc:2749] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:07.317219 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a788c7ceff54a7683cb674b33080008" is_pre_election: true
I20260709 13:07:07.317889 13121 raft_consensus.cc:2393] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3fb587cb63854c42abf5c4f501ee32b3 in current term 1: Already voted for candidate 8a788c7ceff54a7683cb674b33080008 in this term.
I20260709 13:07:07.319171 13010 leader_election.cc:304] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3; no voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:07.319944 13386 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:07.323711 13121 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:35054
I20260709 13:07:07.324488 13121 raft_consensus.cc:606] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Received request to transfer leadership to TS 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:07.327948 13386 raft_consensus.cc:2749] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:07.339193 13121 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:35054
I20260709 13:07:07.340333 13121 raft_consensus.cc:606] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Received request to transfer leadership to TS 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:07.347973 13191 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1226828512554b5b9c417a4830456199"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:50788
I20260709 13:07:07.348691 13191 raft_consensus.cc:606] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:07.355944 13195 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:50788
I20260709 13:07:07.356635 13195 raft_consensus.cc:606] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:07.357182 13386 consensus_queue.cc:1048] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.372121 13386 consensus_queue.cc:1048] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.374475 13045 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:59910
I20260709 13:07:07.375295 13045 raft_consensus.cc:606] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 077c76b5bd474881905cb6cfe23c3124
I20260709 13:07:07.385211 13045 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:59910
I20260709 13:07:07.386026 13045 raft_consensus.cc:606] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:07.409235 13361 consensus_queue.cc:1048] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.418013 13336 raft_consensus.cc:993] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.418973 13389 raft_consensus.cc:1081] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.420928 13120 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1226828512554b5b9c417a4830456199"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35042
I20260709 13:07:07.421532 13120 raft_consensus.cc:493] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.421907 13120 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.426995 13390 consensus_queue.cc:1048] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.428201 13120 raft_consensus.cc:515] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.430997 13046 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:07.432426 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1226828512554b5b9c417a4830456199" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.433127 13191 raft_consensus.cc:3055] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.433593 13191 raft_consensus.cc:740] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:07.435250 13191 consensus_queue.cc:260] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.436372 13191 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.433202 13120 leader_election.cc:290] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.447306 13191 raft_consensus.cc:2468] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:07.448864 13084 leader_election.cc:304] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 3fb587cb63854c42abf5c4f501ee32b3
I20260709 13:07:07.449831 13405 raft_consensus.cc:2804] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.450376 13405 raft_consensus.cc:697] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.451087 13405 consensus_queue.cc:237] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.458720 12942 catalog_manager.cc:5697] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 8a788c7ceff54a7683cb674b33080008 (127.0.107.195) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.475311 13405 consensus_queue.cc:1048] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:07.484668 13358 raft_consensus.cc:993] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:07.485419 13359 raft_consensus.cc:1081] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Signalling peer 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:07.487614 13046 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:59908
I20260709 13:07:07.488687 13046 raft_consensus.cc:493] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.489080 13046 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.492200 13405 consensus_queue.cc:1048] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.498811 13046 raft_consensus.cc:515] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.502199 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:07.502919 13121 raft_consensus.cc:3055] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.503356 13121 raft_consensus.cc:740] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.504312 13121 consensus_queue.cc:260] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.505743 13121 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.502377 13046 leader_election.cc:290] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.508796 13191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b138505857d34311a9efe1d10c311b0d" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.509537 13191 raft_consensus.cc:3060] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.515713 13121 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:07.517022 13191 raft_consensus.cc:2468] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:07.518709 13010 leader_election.cc:304] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.519904 13386 raft_consensus.cc:2804] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.520458 13386 raft_consensus.cc:697] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.521451 13386 consensus_queue.cc:237] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.522238 13413 consensus_queue.cc:1048] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.533077 12943 catalog_manager.cc:5697] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 1 to 2, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.544871 13413 consensus_queue.cc:1048] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.552585 13390 consensus_queue.cc:1048] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.560104 13356 raft_consensus.cc:993] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.560704 13413 raft_consensus.cc:1081] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.564049 13121 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35052
I20260709 13:07:07.564731 13121 raft_consensus.cc:493] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.565109 13121 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.574797 13121 raft_consensus.cc:515] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.580062 13121 leader_election.cc:290] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.581064 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.581910 13195 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.582846 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d33b7a21125f4669ae2ca2fc148095dd" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:07.583523 13045 raft_consensus.cc:3055] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.583842 13045 raft_consensus.cc:740] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.584719 13045 consensus_queue.cc:260] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.586202 13045 raft_consensus.cc:3060] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.593917 13195 raft_consensus.cc:2468] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:07.595569 13084 leader_election.cc:304] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:07.596580 13405 raft_consensus.cc:2804] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.596700 13390 consensus_queue.cc:1048] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.597157 13405 raft_consensus.cc:697] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.598690 13045 raft_consensus.cc:2468] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:07.598488 13405 consensus_queue.cc:237] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.606786 13389 raft_consensus.cc:993] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: : Instructing follower 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.607358 13390 raft_consensus.cc:1081] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Signalling peer 077c76b5bd474881905cb6cfe23c3124 to start an election
I20260709 13:07:07.608309 12943 catalog_manager.cc:5697] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.609208 13121 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
 from {username='slave'} at 127.0.0.1:35042
I20260709 13:07:07.609972 13121 raft_consensus.cc:493] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.610337 13121 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.618957 13121 raft_consensus.cc:515] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.621714 13121 leader_election.cc:290] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.622509 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:07.623304 13045 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.623695 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "703c89ef8f234b46b4ede95403b2ff13" candidate_uuid: "077c76b5bd474881905cb6cfe23c3124" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.624331 13195 raft_consensus.cc:3055] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.624667 13195 raft_consensus.cc:740] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:07.625751 13195 consensus_queue.cc:260] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.627300 13195 raft_consensus.cc:3060] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.632428 13045 raft_consensus.cc:2468] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:07.634035 13086 leader_election.cc:304] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 3fb587cb63854c42abf5c4f501ee32b3; no voters: 
I20260709 13:07:07.635039 13405 raft_consensus.cc:2804] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.635529 13405 raft_consensus.cc:697] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Leader. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.636150 13195 raft_consensus.cc:2468] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 077c76b5bd474881905cb6cfe23c3124 in term 2.
I20260709 13:07:07.636372 13405 consensus_queue.cc:237] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.644757 12943 catalog_manager.cc:5697] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 reported cstate change: term changed from 1 to 2, leader changed from 8a788c7ceff54a7683cb674b33080008 (127.0.107.195) to 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194). New cstate: current_term: 2 leader_uuid: "077c76b5bd474881905cb6cfe23c3124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.658669 13405 consensus_queue.cc:1048] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.673629 13359 consensus_queue.cc:1048] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.701105 13358 raft_consensus.cc:993] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:07.701685 13359 raft_consensus.cc:1081] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Signalling peer 3fb587cb63854c42abf5c4f501ee32b3 to start an election
I20260709 13:07:07.703548 13045 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
 from {username='slave'} at 127.0.0.1:59908
I20260709 13:07:07.704260 13045 raft_consensus.cc:493] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.704598 13045 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.712062 13045 raft_consensus.cc:515] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.714195 13045 leader_election.cc:290] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Requested vote from peers 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583), 8a788c7ceff54a7683cb674b33080008 (127.0.107.195:45349)
I20260709 13:07:07.715378 13195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a788c7ceff54a7683cb674b33080008"
I20260709 13:07:07.715378 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "3fb587cb63854c42abf5c4f501ee32b3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:07.716485 13195 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.716593 13121 raft_consensus.cc:3055] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.717074 13121 raft_consensus.cc:740] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:07.718555 13121 consensus_queue.cc:260] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.720240 13121 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.726938 13195 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:07.728560 13009 leader_election.cc:304] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:07.728730 13121 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fb587cb63854c42abf5c4f501ee32b3 in term 2.
I20260709 13:07:07.729604 13356 raft_consensus.cc:2804] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.730114 13356 raft_consensus.cc:697] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Leader. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.730940 13356 consensus_queue.cc:237] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.738783 12942 catalog_manager.cc:5697] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 reported cstate change: term changed from 1 to 2, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "3fb587cb63854c42abf5c4f501ee32b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:07.746035 13356 consensus_queue.cc:1048] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:07.756134 13380 raft_consensus.cc:993] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:07.756667 13380 raft_consensus.cc:1081] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:07.760244 13191 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50762
I20260709 13:07:07.763428 13191 raft_consensus.cc:493] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:07.763928 13191 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.765568 13356 consensus_queue.cc:1048] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:07.773100 13191 raft_consensus.cc:515] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.779613 13191 leader_election.cc:290] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:07.780900 13121 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:07.781821 13121 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.781934 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "656aa2e5a56f4331ad72b9328522d7a8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:07.784523 13045 raft_consensus.cc:3055] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:07.784930 13045 raft_consensus.cc:740] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:07.785930 13045 consensus_queue.cc:260] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.787421 13045 raft_consensus.cc:3060] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.790868 13121 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:07.792593 13159 leader_election.cc:304] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:07.793641 13390 raft_consensus.cc:2804] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:07.794206 13390 raft_consensus.cc:697] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:07.795150 13390 consensus_queue.cc:237] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:07.802177 13045 raft_consensus.cc:2468] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:07.805380 12943 catalog_manager.cc:5697] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:07.919759 13045 raft_consensus.cc:3060] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:07.921733 13191 raft_consensus.cc:1275] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:07.923699 13359 consensus_queue.cc:1048] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:07.929306 13045 raft_consensus.cc:1275] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:07.932437 13405 consensus_queue.cc:1048] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:07.945685 13191 raft_consensus.cc:1275] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:07.947475 13356 consensus_queue.cc:1048] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:07.957268 13121 raft_consensus.cc:1275] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:07.959148 13300 consensus_queue.cc:1048] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:08.042165 13191 raft_consensus.cc:1275] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.046551 13358 consensus_queue.cc:1048] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 13:07:08.072194 13045 raft_consensus.cc:1275] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.074553 13359 consensus_queue.cc:1048] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:08.088624 13046 raft_consensus.cc:1275] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.109035 13405 consensus_queue.cc:1048] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:08.126817 13195 raft_consensus.cc:1275] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 077c76b5bd474881905cb6cfe23c3124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.130337 13359 consensus_queue.cc:1048] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:08.193084 13191 raft_consensus.cc:1275] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.194658 13300 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:08.207058 13121 raft_consensus.cc:1275] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 3fb587cb63854c42abf5c4f501ee32b3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.208830 13300 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:08.316448 13046 raft_consensus.cc:1275] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.318135 13390 consensus_queue.cc:1048] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:08.331602 13121 raft_consensus.cc:1275] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:08.333468 13390 consensus_queue.cc:1048] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:10.390606 12941 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54668:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:07:10.418072 12941 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:54668:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:07:10.460209   431 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 13:07:10.574187 13454 log.cc:927] Time spent T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 13:07:10.755313 13478 tablet.cc:1724] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:10.794094 13084 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:45349, user_credentials={real_user=slave}} blocked reactor thread for 52104.4us
I20260709 13:07:10.854007 13493 tablet.cc:1724] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:10.853446 13490 tablet.cc:1724] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:10.846248 13486 tablet.cc:1724] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:10.845798 13487 tablet.cc:1724] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:10.898417 13496 tablet.cc:1724] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:10.905146 13481 log.cc:927] Time spent T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Append to log took a long time: real 0.163s	user 0.002s	sys 0.001s
W20260709 13:07:10.919550 13476 log.cc:927] Time spent T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Append to log took a long time: real 0.166s	user 0.003s	sys 0.000s
I20260709 13:07:10.945611 13115 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:35070
I20260709 13:07:10.947176 13115 raft_consensus.cc:606] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:10.876223 13489 tablet.cc:1724] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:10.967839 13147 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:1226828512554b5b9c417a4830456199)
I20260709 13:07:10.969352 13478 tablet.cc:1724] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:10.973688 13494 tablet.cc:1724] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:10.991281 13072 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:1226828512554b5b9c417a4830456199)
W20260709 13:07:10.992713 13072 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:1226828512554b5b9c417a4830456199)
I20260709 13:07:10.996872 13482 raft_consensus.cc:993] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:10.997562 13482 raft_consensus.cc:1081] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.001981 13500 tablet.cc:1724] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.001986 13192 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50772
I20260709 13:07:11.017622 13192 raft_consensus.cc:493] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:11.018175 13192 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.034055 13478 tablet.cc:1724] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.039716 13489 tablet.cc:1724] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.060571 13496 tablet.cc:1724] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.067309 13493 tablet.cc:1724] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.069792 13478 tablet.cc:1724] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.071425 13501 tablet.cc:1724] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.071985 13505 tablet.cc:1724] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.070910 13503 tablet.cc:1724] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.088222 13490 tablet.cc:1724] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.091933 13478 tablet.cc:1724] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.087663 13494 tablet.cc:1724] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.113370 13041 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:59914
I20260709 13:07:11.114557 13041 raft_consensus.cc:606] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:11.116492   431 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 13:07:11.117318   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:11.116989 13505 tablet.cc:1724] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.129420 13041 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5426131135564b57af11c0740aa0c357"
dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:59914
I20260709 13:07:11.130271 13041 raft_consensus.cc:606] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:11.138216 13503 tablet.cc:1724] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.144106 13496 tablet.cc:1724] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.149541 13119 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b"
dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
mode: GRACEFUL
new_leader_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:35070
I20260709 13:07:11.150404 13119 raft_consensus.cc:606] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Received request to transfer leadership to TS 8a788c7ceff54a7683cb674b33080008
I20260709 13:07:11.152792 13501 tablet.cc:1724] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.155287 13500 tablet.cc:1724] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.158304   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:11.159264   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:11.158694 13494 tablet.cc:1724] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:11.167853 13192 raft_consensus.cc:515] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.205492 13192 leader_election.cc:290] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:11.206468 13041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:11.207715 13041 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.207607 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "779ad34d591e489da454a3608a4c32c8" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:11.211074 13119 raft_consensus.cc:3055] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:11.211566 13119 raft_consensus.cc:740] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:11.212984 13119 consensus_queue.cc:260] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.218493 13119 raft_consensus.cc:3060] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.223244 13041 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:11.225265 13160 leader_election.cc:304] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:11.226717 12941 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: soft_deleted_table [id=ecc7d77e8148455f8a345b5e5cb32950] alter complete (version 1)
I20260709 13:07:11.226861 13432 raft_consensus.cc:2804] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:11.227546 13432 raft_consensus.cc:697] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:11.227737 13119 raft_consensus.cc:2468] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:11.229007 13432 consensus_queue.cc:237] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.241663 12941 catalog_manager.cc:5697] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 1 to 2, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:11.252746 13462 raft_consensus.cc:993] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.253602 13458 raft_consensus.cc:1081] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.255371 13192 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5426131135564b57af11c0740aa0c357"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50762
I20260709 13:07:11.255931 13192 raft_consensus.cc:493] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:11.256282 13192 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.262391 13192 raft_consensus.cc:515] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.264422 13192 leader_election.cc:290] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:11.265228 13041 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:11.266000 13041 raft_consensus.cc:3055] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:11.265900 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5426131135564b57af11c0740aa0c357" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:11.266464 13041 raft_consensus.cc:740] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:11.266707 13119 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.267472 13041 consensus_queue.cc:260] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.269192 13041 raft_consensus.cc:3060] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:11.275820 13119 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:11.277616 13159 leader_election.cc:304] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:11.277748 13041 raft_consensus.cc:2468] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 2.
I20260709 13:07:11.278697 13432 raft_consensus.cc:2804] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:11.279266 13432 raft_consensus.cc:697] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:11.280174 13432 consensus_queue.cc:237] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.290011 12941 catalog_manager.cc:5697] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 1 to 2, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:11.383949 13497 raft_consensus.cc:993] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.384763 13488 raft_consensus.cc:1081] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.386588 13192 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50772
I20260709 13:07:11.387185 13192 raft_consensus.cc:493] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:11.387528 13192 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.395371 13192 raft_consensus.cc:515] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.397439 13192 leader_election.cc:290] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:11.398360 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:11.398767 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d72ffe8729f143098d6418807c1a940b" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:11.399227 13045 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.399533 13119 raft_consensus.cc:3055] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:11.400079 13119 raft_consensus.cc:740] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 077c76b5bd474881905cb6cfe23c3124, State: Running, Role: LEADER
I20260709 13:07:11.401232 13119 consensus_queue.cc:260] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.402810 13119 raft_consensus.cc:3060] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.411551 13045 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 3.
I20260709 13:07:11.411779 13119 raft_consensus.cc:2468] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 3.
I20260709 13:07:11.412853 13160 leader_election.cc:304] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fb587cb63854c42abf5c4f501ee32b3, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:11.413862 13432 raft_consensus.cc:2804] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:11.414506 13432 raft_consensus.cc:697] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 3 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:11.415505 13432 consensus_queue.cc:237] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.428761 12941 catalog_manager.cc:5697] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 2 to 3, leader changed from 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:11.553426 13458 raft_consensus.cc:993] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: : Instructing follower 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.554095 13458 raft_consensus.cc:1081] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Signalling peer 8a788c7ceff54a7683cb674b33080008 to start an election
I20260709 13:07:11.556416 13194 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5"
dest_uuid: "8a788c7ceff54a7683cb674b33080008"
 from {username='slave'} at 127.0.0.1:50762
I20260709 13:07:11.557119 13194 raft_consensus.cc:493] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:11.557475 13194 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.565775 13194 raft_consensus.cc:515] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.567556 13194 leader_election.cc:290] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 3 election: Requested vote from peers 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505), 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583)
I20260709 13:07:11.569068 13045 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "3fb587cb63854c42abf5c4f501ee32b3"
I20260709 13:07:11.569068 13119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e98e42e324104a14aaf360c3333581a5" candidate_uuid: "8a788c7ceff54a7683cb674b33080008" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "077c76b5bd474881905cb6cfe23c3124"
I20260709 13:07:11.570144 13045 raft_consensus.cc:3055] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:11.570221 13119 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.570544 13045 raft_consensus.cc:740] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3fb587cb63854c42abf5c4f501ee32b3, State: Running, Role: LEADER
I20260709 13:07:11.571743 13045 consensus_queue.cc:260] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.573184 13045 raft_consensus.cc:3060] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:11.578346 13119 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 3.
I20260709 13:07:11.579854 13045 raft_consensus.cc:2468] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a788c7ceff54a7683cb674b33080008 in term 3.
I20260709 13:07:11.579952 13159 leader_election.cc:304] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 077c76b5bd474881905cb6cfe23c3124, 8a788c7ceff54a7683cb674b33080008; no voters: 
I20260709 13:07:11.581319 13432 raft_consensus.cc:2804] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:11.581948 13432 raft_consensus.cc:697] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 3 LEADER]: Becoming Leader. State: Replica: 8a788c7ceff54a7683cb674b33080008, State: Running, Role: LEADER
I20260709 13:07:11.583387 13432 consensus_queue.cc:237] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } }
I20260709 13:07:11.591928 12941 catalog_manager.cc:5697] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 reported cstate change: term changed from 2 to 3, leader changed from 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193) to 8a788c7ceff54a7683cb674b33080008 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "8a788c7ceff54a7683cb674b33080008" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a788c7ceff54a7683cb674b33080008" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 45349 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:11.646987 13041 raft_consensus.cc:1275] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:07:11.648604 13432 consensus_queue.cc:1048] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:11.657755 13119 raft_consensus.cc:1275] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 13:07:11.659699 13432 consensus_queue.cc:1048] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:11.731366 13119 raft_consensus.cc:1275] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:11.733264 13432 consensus_queue.cc:1048] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:11.746593 13041 raft_consensus.cc:1275] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:11.748315 13467 consensus_queue.cc:1048] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:11.919050 13119 raft_consensus.cc:1275] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:11.921214 13516 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:11.932893 13045 raft_consensus.cc:1275] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:11.934899 13432 consensus_queue.cc:1048] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:12.077564 13045 raft_consensus.cc:1275] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:07:12.079145 13432 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fb587cb63854c42abf5c4f501ee32b3" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 43505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:12.098215 13119 raft_consensus.cc:1275] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Refusing update from remote peer 8a788c7ceff54a7683cb674b33080008: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 13:07:12.100142 13467 consensus_queue.cc:1048] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [LEADER]: Connected to new peer: Peer: permanent_uuid: "077c76b5bd474881905cb6cfe23c3124" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 39583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 13:07:12.234521   431 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 13:07:12.239826   431 auto_leader_rebalancer.cc:452] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 13:07:12.240306   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 13:07:12.245427   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 13:07:12.245923   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:13.278623   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
W20260709 13:07:13.293131 13086 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 9 similar messages]
W20260709 13:07:13.296720 13086 consensus_peers.cc:597] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.306560 13160 consensus_peers.cc:597] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.315557 13086 consensus_peers.cc:597] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.327761   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:13.328701   431 tablet_replica.cc:333] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.329487   431 raft_consensus.cc:2243] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.330116   431 raft_consensus.cc:2272] T 1226828512554b5b9c417a4830456199 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.333570   431 tablet_replica.cc:333] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.334713   431 raft_consensus.cc:2243] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.335268   431 raft_consensus.cc:2272] T 656aa2e5a56f4331ad72b9328522d7a8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.337710   431 tablet_replica.cc:333] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.338294   431 raft_consensus.cc:2243] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.338742   431 raft_consensus.cc:2272] T 779ad34d591e489da454a3608a4c32c8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.341044   431 tablet_replica.cc:333] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.341758   431 raft_consensus.cc:2243] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.342233   431 raft_consensus.cc:2272] T d33b7a21125f4669ae2ca2fc148095dd P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.344825   431 tablet_replica.cc:333] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.345610   431 raft_consensus.cc:2243] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.346647   431 raft_consensus.cc:2272] T 810e036588a6404c8282f7cc58fcf780 P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:13.346913 13086 consensus_peers.cc:597] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.349241   431 tablet_replica.cc:333] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.349997   431 raft_consensus.cc:2243] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.350811   431 raft_consensus.cc:2272] T cbd3163b2f7b4547a520f8e81ec43eef P 3fb587cb63854c42abf5c4f501ee32b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.353667   431 tablet_replica.cc:333] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.354841   431 raft_consensus.cc:2243] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.355540   431 raft_consensus.cc:2272] T 9cbad72123464a59a5a17a82b5cabe04 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:13.356914 13160 consensus_peers.cc:597] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.359345   431 tablet_replica.cc:333] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.360349   431 raft_consensus.cc:2243] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.360988   431 raft_consensus.cc:2272] T 62db6cb99508458e9640d9841aa1998d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.364316   431 tablet_replica.cc:333] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.365275   431 raft_consensus.cc:2243] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.366245   431 raft_consensus.cc:2272] T b138505857d34311a9efe1d10c311b0d P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.369102   431 tablet_replica.cc:333] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.369864   431 raft_consensus.cc:2243] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.370893   431 raft_consensus.cc:2272] T bce188e6a2944fb0abd11b7fba0b46a7 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.373548   431 tablet_replica.cc:333] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.374188   431 raft_consensus.cc:2243] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.374696   431 raft_consensus.cc:2272] T d72ffe8729f143098d6418807c1a940b P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.377178   431 tablet_replica.cc:333] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.377796   431 raft_consensus.cc:2243] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.378247   431 raft_consensus.cc:2272] T 5426131135564b57af11c0740aa0c357 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.380661   431 tablet_replica.cc:333] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.381249   431 raft_consensus.cc:2243] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.381803   431 raft_consensus.cc:2272] T 703c89ef8f234b46b4ede95403b2ff13 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.384030   431 tablet_replica.cc:333] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.384639   431 raft_consensus.cc:2243] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.385592   431 raft_consensus.cc:2272] T e60047ab21874e2086ee857dce35d837 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.390254   431 tablet_replica.cc:333] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.390911   431 raft_consensus.cc:2243] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.392117   431 raft_consensus.cc:2272] T e31761a5de5e4572b87141de0d90cba8 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.395897   431 tablet_replica.cc:333] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.396624   431 raft_consensus.cc:2243] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.397189   431 raft_consensus.cc:2272] T e98e42e324104a14aaf360c3333581a5 P 3fb587cb63854c42abf5c4f501ee32b3 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:13.397955 13086 consensus_peers.cc:597] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.400070   431 tablet_replica.cc:333] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.400820   431 raft_consensus.cc:2243] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.401361   431 raft_consensus.cc:2272] T 8809f7896b03429fa2253088c87b8a71 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.403868   431 tablet_replica.cc:333] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3: stopping tablet replica
I20260709 13:07:13.404446   431 raft_consensus.cc:2243] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.404912   431 raft_consensus.cc:2272] T 12a7ada4b5544ef8891f296e167a3a84 P 3fb587cb63854c42abf5c4f501ee32b3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.450985   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
I20260709 13:07:13.501525   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:07:13.506841 13160 consensus_peers.cc:597] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.511991 13159 consensus_peers.cc:597] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 -> Peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Couldn't send request to peer 077c76b5bd474881905cb6cfe23c3124. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.516186 13086 consensus_peers.cc:597] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.541286 13160 consensus_peers.cc:597] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.548013   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:13.549045   431 tablet_replica.cc:333] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.550310   431 raft_consensus.cc:2243] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.551056   431 raft_consensus.cc:2272] T 656aa2e5a56f4331ad72b9328522d7a8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:13.551244 13159 consensus_peers.cc:597] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 -> Peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Couldn't send request to peer 077c76b5bd474881905cb6cfe23c3124. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.553479   431 tablet_replica.cc:333] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.554383   431 raft_consensus.cc:2243] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.555305   431 raft_consensus.cc:2272] T d33b7a21125f4669ae2ca2fc148095dd P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.557715   431 tablet_replica.cc:333] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.558404   431 raft_consensus.cc:2243] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.558909   431 raft_consensus.cc:2272] T 779ad34d591e489da454a3608a4c32c8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.561857   431 tablet_replica.cc:333] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.562686   431 raft_consensus.cc:2243] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.563230   431 raft_consensus.cc:2272] T 810e036588a6404c8282f7cc58fcf780 P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.565750   431 tablet_replica.cc:333] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.566346   431 raft_consensus.cc:2243] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.567319   431 raft_consensus.cc:2272] T 1226828512554b5b9c417a4830456199 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.570391   431 tablet_replica.cc:333] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.571115   431 raft_consensus.cc:2243] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.571621   431 raft_consensus.cc:2272] T cbd3163b2f7b4547a520f8e81ec43eef P 077c76b5bd474881905cb6cfe23c3124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.574447   431 tablet_replica.cc:333] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.575474   431 raft_consensus.cc:2243] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.576722   431 raft_consensus.cc:2272] T 9cbad72123464a59a5a17a82b5cabe04 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.579521   431 tablet_replica.cc:333] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.580312   431 raft_consensus.cc:2243] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.581528   431 raft_consensus.cc:2272] T 62db6cb99508458e9640d9841aa1998d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.584687   431 tablet_replica.cc:333] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.585371   431 raft_consensus.cc:2243] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.586005   431 raft_consensus.cc:2272] T b138505857d34311a9efe1d10c311b0d P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.588445   431 tablet_replica.cc:333] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
W20260709 13:07:13.588860 13160 consensus_peers.cc:597] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.589113   431 raft_consensus.cc:2243] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.589855   431 raft_consensus.cc:2272] T bce188e6a2944fb0abd11b7fba0b46a7 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.592433   431 tablet_replica.cc:333] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.593161   431 raft_consensus.cc:2243] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.593644   431 raft_consensus.cc:2272] T d72ffe8729f143098d6418807c1a940b P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.596230   431 tablet_replica.cc:333] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.596827   431 raft_consensus.cc:2243] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.597833   431 raft_consensus.cc:2272] T 703c89ef8f234b46b4ede95403b2ff13 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.600943   431 tablet_replica.cc:333] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.601625   431 raft_consensus.cc:2243] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.602099   431 raft_consensus.cc:2272] T e60047ab21874e2086ee857dce35d837 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.604365   431 tablet_replica.cc:333] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.604951   431 raft_consensus.cc:2243] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.605404   431 raft_consensus.cc:2272] T 5426131135564b57af11c0740aa0c357 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.607841   431 tablet_replica.cc:333] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.608402   431 raft_consensus.cc:2243] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.608842   431 raft_consensus.cc:2272] T e98e42e324104a14aaf360c3333581a5 P 077c76b5bd474881905cb6cfe23c3124 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.611228   431 tablet_replica.cc:333] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.611714   431 raft_consensus.cc:2243] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.612167   431 raft_consensus.cc:2272] T 8809f7896b03429fa2253088c87b8a71 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.614914   431 tablet_replica.cc:333] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.615536   431 raft_consensus.cc:2243] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.616529   431 raft_consensus.cc:2272] T 12a7ada4b5544ef8891f296e167a3a84 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.619200   431 tablet_replica.cc:333] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124: stopping tablet replica
I20260709 13:07:13.619748   431 raft_consensus.cc:2243] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 13:07:13.619798 13159 consensus_peers.cc:597] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 -> Peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Couldn't send request to peer 077c76b5bd474881905cb6cfe23c3124. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.620339   431 raft_consensus.cc:2272] T e31761a5de5e4572b87141de0d90cba8 P 077c76b5bd474881905cb6cfe23c3124 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.669678   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
I20260709 13:07:13.715372   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
W20260709 13:07:13.726230 13160 consensus_peers.cc:597] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 -> Peer 3fb587cb63854c42abf5c4f501ee32b3 (127.0.107.193:43505): Couldn't send request to peer 3fb587cb63854c42abf5c4f501ee32b3. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:43505: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:13.752713 13159 consensus_peers.cc:597] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 -> Peer 077c76b5bd474881905cb6cfe23c3124 (127.0.107.194:39583): Couldn't send request to peer 077c76b5bd474881905cb6cfe23c3124. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:39583: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:13.767115   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:13.768289   431 tablet_replica.cc:333] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.769328   431 raft_consensus.cc:2243] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.770692   431 raft_consensus.cc:2272] T 656aa2e5a56f4331ad72b9328522d7a8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.773270   431 tablet_replica.cc:333] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.774108   431 raft_consensus.cc:2243] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.775004   431 raft_consensus.cc:2272] T 779ad34d591e489da454a3608a4c32c8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.777694   431 tablet_replica.cc:333] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.778366   431 raft_consensus.cc:2243] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.779142   431 raft_consensus.cc:2272] T d33b7a21125f4669ae2ca2fc148095dd P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.782181   431 tablet_replica.cc:333] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.782851   431 raft_consensus.cc:2243] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.783358   431 raft_consensus.cc:2272] T 1226828512554b5b9c417a4830456199 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.785686   431 tablet_replica.cc:333] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.786283   431 raft_consensus.cc:2243] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.786748   431 raft_consensus.cc:2272] T 810e036588a6404c8282f7cc58fcf780 P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.789041   431 tablet_replica.cc:333] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.789753   431 raft_consensus.cc:2243] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.790213   431 raft_consensus.cc:2272] T 62db6cb99508458e9640d9841aa1998d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.792687   431 tablet_replica.cc:333] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.793320   431 raft_consensus.cc:2243] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.793758   431 raft_consensus.cc:2272] T cbd3163b2f7b4547a520f8e81ec43eef P 8a788c7ceff54a7683cb674b33080008 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.795773   431 tablet_replica.cc:333] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.796321   431 raft_consensus.cc:2243] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.796774   431 raft_consensus.cc:2272] T 9cbad72123464a59a5a17a82b5cabe04 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.799005   431 tablet_replica.cc:333] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.799535   431 raft_consensus.cc:2243] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.800446   431 raft_consensus.cc:2272] T d72ffe8729f143098d6418807c1a940b P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.802847   431 tablet_replica.cc:333] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.803366   431 raft_consensus.cc:2243] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.803794   431 raft_consensus.cc:2272] T 703c89ef8f234b46b4ede95403b2ff13 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.806648   431 tablet_replica.cc:333] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.807200   431 raft_consensus.cc:2243] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.807611   431 raft_consensus.cc:2272] T e60047ab21874e2086ee857dce35d837 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.809821   431 tablet_replica.cc:333] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.810330   431 raft_consensus.cc:2243] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.811091   431 raft_consensus.cc:2272] T 5426131135564b57af11c0740aa0c357 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.813130   431 tablet_replica.cc:333] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.813731   431 raft_consensus.cc:2243] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.814131   431 raft_consensus.cc:2272] T b138505857d34311a9efe1d10c311b0d P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.816319   431 tablet_replica.cc:333] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.817255   431 raft_consensus.cc:2243] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.817674   431 raft_consensus.cc:2272] T bce188e6a2944fb0abd11b7fba0b46a7 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.819892   431 tablet_replica.cc:333] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.820375   431 raft_consensus.cc:2243] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.821187   431 raft_consensus.cc:2272] T e98e42e324104a14aaf360c3333581a5 P 8a788c7ceff54a7683cb674b33080008 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.823629   431 tablet_replica.cc:333] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.824198   431 raft_consensus.cc:2243] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.825028   431 raft_consensus.cc:2272] T 8809f7896b03429fa2253088c87b8a71 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.827378   431 tablet_replica.cc:333] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.827926   431 raft_consensus.cc:2243] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.828405   431 raft_consensus.cc:2272] T 12a7ada4b5544ef8891f296e167a3a84 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.830643   431 tablet_replica.cc:333] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008: stopping tablet replica
I20260709 13:07:13.831161   431 raft_consensus.cc:2243] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:13.831651   431 raft_consensus.cc:2272] T e31761a5de5e4572b87141de0d90cba8 P 8a788c7ceff54a7683cb674b33080008 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.880364   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:07:13.928723   431 master.cc:562] Master@127.0.107.254:40339 shutting down...
I20260709 13:07:13.956051   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:13.956805   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:13.957224   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 90f12b937c774e499518ad9714a34cfe: stopping tablet replica
I20260709 13:07:13.981086   431 master.cc:584] Master@127.0.107.254:40339 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (13388 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 13:07:14.047204   431 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.107.254:37867
I20260709 13:07:14.048436   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:14.054836 13532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.054967 13533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.055639 13535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:14.056869   431 server_base.cc:1061] running on GCE node
I20260709 13:07:14.057937   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:14.058137   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:14.058320   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602434058303 us; error 0 us; skew 500 ppm
I20260709 13:07:14.058905   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:14.062088   431 webserver.cc:533] Webserver started at http://127.0.107.254:34469/ using document root <none> and password file <none>
I20260709 13:07:14.062606   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:14.062808   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:14.063097   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:14.064283   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/master-0-root/instance:
uuid: "44459bc0b4b04edcb412c73f6d2898bd"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.069142   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 13:07:14.073412 13540 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.074424   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 13:07:14.074751   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/master-0-root
uuid: "44459bc0b4b04edcb412c73f6d2898bd"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.075073   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:14.114075   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:14.115517   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:14.160085   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.254:37867
I20260709 13:07:14.160207 13591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.254:37867 every 8 connection(s)
I20260709 13:07:14.164350 13592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.179307 13592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: Bootstrap starting.
I20260709 13:07:14.186028 13592 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:14.191663 13592 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: No bootstrap required, opened a new log
I20260709 13:07:14.194252 13592 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER }
I20260709 13:07:14.194658 13592 raft_consensus.cc:385] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:14.194967 13592 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44459bc0b4b04edcb412c73f6d2898bd, State: Initialized, Role: FOLLOWER
I20260709 13:07:14.195732 13592 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER }
I20260709 13:07:14.196228 13592 raft_consensus.cc:399] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 13:07:14.196441 13592 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 13:07:14.196771 13592 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:14.202864 13592 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER }
I20260709 13:07:14.203480 13592 leader_election.cc:304] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44459bc0b4b04edcb412c73f6d2898bd; no voters: 
I20260709 13:07:14.204792 13592 leader_election.cc:290] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 13:07:14.205302 13595 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:14.207115 13595 raft_consensus.cc:697] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 1 LEADER]: Becoming Leader. State: Replica: 44459bc0b4b04edcb412c73f6d2898bd, State: Running, Role: LEADER
I20260709 13:07:14.207897 13595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER }
I20260709 13:07:14.208645 13592 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [sys.catalog]: configured and running, proceeding with master startup.
I20260709 13:07:14.213258 13596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44459bc0b4b04edcb412c73f6d2898bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER } }
I20260709 13:07:14.213295 13597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44459bc0b4b04edcb412c73f6d2898bd. Latest consensus state: current_term: 1 leader_uuid: "44459bc0b4b04edcb412c73f6d2898bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44459bc0b4b04edcb412c73f6d2898bd" member_type: VOTER } }
I20260709 13:07:14.214509 13597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:14.215898 13596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [sys.catalog]: This master's current role is: LEADER
I20260709 13:07:14.216706 13602 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 13:07:14.222203 13602 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 13:07:14.226889   431 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 13:07:14.232378 13602 catalog_manager.cc:1383] Generated new cluster ID: e2743bd25825400e9f4ad48102cf89f5
I20260709 13:07:14.232733 13602 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 13:07:14.265625 13602 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 13:07:14.267179 13602 catalog_manager.cc:1540] Loading token signing keys...
I20260709 13:07:14.286703 13602 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: Generated new TSK 0
I20260709 13:07:14.287490 13602 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 13:07:14.298430   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:14.306707 13613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.308110 13614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.310603 13616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:14.310662   431 server_base.cc:1061] running on GCE node
I20260709 13:07:14.311995   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 13:07:14.312237   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:14.312412   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602434312391 us; error 0 us; skew 500 ppm
I20260709 13:07:14.312995   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:14.315582   431 webserver.cc:533] Webserver started at http://127.0.107.193:45151/ using document root <none> and password file <none>
I20260709 13:07:14.316110   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:14.316337   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:14.316639   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:14.317925   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-0-root/instance:
uuid: "d1db7c835d25447f863c4568a0ebdfb0"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.322763   431 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 13:07:14.326426 13621 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.327471   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 13:07:14.327752   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-0-root
uuid: "d1db7c835d25447f863c4568a0ebdfb0"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.328056   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:14.353503   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:14.355172   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:14.357014   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:14.359948   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:14.360287   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.360589   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:14.360782   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.410243   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.193:35387
I20260709 13:07:14.410344 13683 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.193:35387 every 8 connection(s)
I20260709 13:07:14.416198   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:14.424823 13688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.428130 13689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.433462 13691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:14.433699   431 server_base.cc:1061] running on GCE node
I20260709 13:07:14.436095   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:07:14.436316 13684 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37867
W20260709 13:07:14.436414   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:14.436887   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602434436863 us; error 0 us; skew 500 ppm
I20260709 13:07:14.436983 13684 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:14.437820   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:14.438133 13684 heartbeater.cc:507] Master 127.0.107.254:37867 requested a full tablet report, sending...
I20260709 13:07:14.441082   431 webserver.cc:533] Webserver started at http://127.0.107.194:41991/ using document root <none> and password file <none>
I20260709 13:07:14.441298 13557 ts_manager.cc:194] Registered new tserver with Master: d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387)
I20260709 13:07:14.441792   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:14.442130   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:14.442427   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:14.443563 13557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43420
I20260709 13:07:14.443706   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-1-root/instance:
uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.449317   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 13:07:14.453091 13696 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.454109   431 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 13:07:14.454440   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-1-root
uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.454876   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:14.477969   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:14.479640   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:14.481400   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:14.483911   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:14.484163   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.484417   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:14.484606   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.536731   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.194:38169
I20260709 13:07:14.536875 13758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.194:38169 every 8 connection(s)
I20260709 13:07:14.542337   431 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 13:07:14.550827 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.553133 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 13:07:14.556520 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 13:07:14.557440   431 server_base.cc:1061] running on GCE node
I20260709 13:07:14.558394 13759 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37867
I20260709 13:07:14.559120   431 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 13:07:14.559165 13759 heartbeater.cc:461] Registering TS with master...
W20260709 13:07:14.559672   431 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 13:07:14.559966   431 hybrid_clock.cc:648] HybridClock initialized: now 1783602434559942 us; error 0 us; skew 500 ppm
I20260709 13:07:14.560799   431 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 13:07:14.560492 13759 heartbeater.cc:507] Master 127.0.107.254:37867 requested a full tablet report, sending...
I20260709 13:07:14.563629 13557 ts_manager.cc:194] Registered new tserver with Master: eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:14.563959   431 webserver.cc:533] Webserver started at http://127.0.107.195:39339/ using document root <none> and password file <none>
I20260709 13:07:14.564589   431 fs_manager.cc:362] Metadata directory not provided
I20260709 13:07:14.564812   431 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 13:07:14.565102   431 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 13:07:14.565575 13557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43424
I20260709 13:07:14.566538   431 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-2-root/instance:
uuid: "35925c1afcb24b089513c51bff743a60"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.572095   431 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.001s
I20260709 13:07:14.577621 13770 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.579217   431 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 13:07:14.579644   431 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-2-root
uuid: "35925c1afcb24b089513c51bff743a60"
format_stamp: "Formatted at 2026-07-09 13:07:14 on dist-test-slave-p8rg"
I20260709 13:07:14.580148   431 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGwNpjd/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783602010504718-431-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 13:07:14.594834   431 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 13:07:14.596330   431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 13:07:14.598368   431 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 13:07:14.601253   431 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 13:07:14.601522   431 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.601845   431 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 13:07:14.601999   431 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 13:07:14.654723   431 rpc_server.cc:307] RPC server started. Bound to: 127.0.107.195:33043
I20260709 13:07:14.654938 13832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.107.195:33043 every 8 connection(s)
I20260709 13:07:14.671836 13833 heartbeater.cc:344] Connected to a master server at 127.0.107.254:37867
I20260709 13:07:14.672422 13833 heartbeater.cc:461] Registering TS with master...
I20260709 13:07:14.673308 13833 heartbeater.cc:507] Master 127.0.107.254:37867 requested a full tablet report, sending...
I20260709 13:07:14.675544 13557 ts_manager.cc:194] Registered new tserver with Master: 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:14.676330   431 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017620837s
I20260709 13:07:14.677248 13557 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43434
I20260709 13:07:14.709888 13557 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43444:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:14.713335 13557 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:14.803825 13724 tablet_service.cc:1511] Processing CreateTablet for tablet c13cbf3a996844aeb37dfa1735fea267 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:14.805516 13724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13cbf3a996844aeb37dfa1735fea267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.811924 13723 tablet_service.cc:1511] Processing CreateTablet for tablet 02fe59bb30944d3bbdda33a2e5c61b7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:14.813045 13722 tablet_service.cc:1511] Processing CreateTablet for tablet 33c76e168db14336b308a33b6c8fe0a2 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:14.813835 13723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe59bb30944d3bbdda33a2e5c61b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.814646 13722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c76e168db14336b308a33b6c8fe0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.817754 13721 tablet_service.cc:1511] Processing CreateTablet for tablet 95f724ccd859431ab29f0d3a03c10d7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:14.819430 13721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f724ccd859431ab29f0d3a03c10d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.820710 13793 tablet_service.cc:1511] Processing CreateTablet for tablet 8f854ca56c664652bc916639fdb53422 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:14.821447 13720 tablet_service.cc:1511] Processing CreateTablet for tablet c9eb03140cf1455fad19606f10f303a0 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:14.822737 13649 tablet_service.cc:1511] Processing CreateTablet for tablet c13cbf3a996844aeb37dfa1735fea267 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:14.848335 13649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13cbf3a996844aeb37dfa1735fea267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.848403 13793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f854ca56c664652bc916639fdb53422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.823483 13798 tablet_service.cc:1511] Processing CreateTablet for tablet c13cbf3a996844aeb37dfa1735fea267 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:14.850267 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c13cbf3a996844aeb37dfa1735fea267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.848335 13720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9eb03140cf1455fad19606f10f303a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.824748 13648 tablet_service.cc:1511] Processing CreateTablet for tablet 02fe59bb30944d3bbdda33a2e5c61b7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:14.853935 13648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe59bb30944d3bbdda33a2e5c61b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.825449 13797 tablet_service.cc:1511] Processing CreateTablet for tablet 02fe59bb30944d3bbdda33a2e5c61b7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:14.856372 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02fe59bb30944d3bbdda33a2e5c61b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.826803 13647 tablet_service.cc:1511] Processing CreateTablet for tablet 33c76e168db14336b308a33b6c8fe0a2 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:14.858909 13647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c76e168db14336b308a33b6c8fe0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.827472 13644 tablet_service.cc:1511] Processing CreateTablet for tablet 8f854ca56c664652bc916639fdb53422 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:14.861837 13644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f854ca56c664652bc916639fdb53422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.829514 13643 tablet_service.cc:1511] Processing CreateTablet for tablet 9658f588616b4965afe18bde6242cd10 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:14.864363 13643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658f588616b4965afe18bde6242cd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.830731 13796 tablet_service.cc:1511] Processing CreateTablet for tablet 33c76e168db14336b308a33b6c8fe0a2 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:14.866775 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c76e168db14336b308a33b6c8fe0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.832511 13645 tablet_service.cc:1511] Processing CreateTablet for tablet c9eb03140cf1455fad19606f10f303a0 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:14.869200 13645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9eb03140cf1455fad19606f10f303a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.834794 13794 tablet_service.cc:1511] Processing CreateTablet for tablet c9eb03140cf1455fad19606f10f303a0 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:14.871655 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9eb03140cf1455fad19606f10f303a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.836280 13795 tablet_service.cc:1511] Processing CreateTablet for tablet 95f724ccd859431ab29f0d3a03c10d7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:14.874588 13795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f724ccd859431ab29f0d3a03c10d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.836800 13790 tablet_service.cc:1511] Processing CreateTablet for tablet e5e543699c1e4102aaab869bca2218da (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:14.877068 13790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e543699c1e4102aaab869bca2218da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.836670 13719 tablet_service.cc:1511] Processing CreateTablet for tablet 8f854ca56c664652bc916639fdb53422 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:14.879770 13719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f854ca56c664652bc916639fdb53422. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.838691 13791 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb0552c48d49e3b9b89e56a35f2139 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:14.839259 13718 tablet_service.cc:1511] Processing CreateTablet for tablet 9658f588616b4965afe18bde6242cd10 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:14.882742 13718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658f588616b4965afe18bde6242cd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.841637 13717 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb0552c48d49e3b9b89e56a35f2139 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:14.885330 13717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb0552c48d49e3b9b89e56a35f2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.841872 13792 tablet_service.cc:1511] Processing CreateTablet for tablet 9658f588616b4965afe18bde6242cd10 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:14.846357 13716 tablet_service.cc:1511] Processing CreateTablet for tablet e5e543699c1e4102aaab869bca2218da (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:14.888370 13716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e543699c1e4102aaab869bca2218da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.828820 13646 tablet_service.cc:1511] Processing CreateTablet for tablet 95f724ccd859431ab29f0d3a03c10d7d (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:14.890719 13646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f724ccd859431ab29f0d3a03c10d7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.895648 13791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb0552c48d49e3b9b89e56a35f2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.969719 13642 tablet_service.cc:1511] Processing CreateTablet for tablet aeeb0552c48d49e3b9b89e56a35f2139 (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:14.971761 13642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeeb0552c48d49e3b9b89e56a35f2139. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.976300 13792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9658f588616b4965afe18bde6242cd10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:14.987142 13641 tablet_service.cc:1511] Processing CreateTablet for tablet e5e543699c1e4102aaab869bca2218da (DEFAULT_TABLE table=test-workload [id=139f0f40d1f84bd3951592a8a5998bca]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:14.988953 13641 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5e543699c1e4102aaab869bca2218da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:15.082389 13848 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.119410 13848 tablet_bootstrap.cc:654] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.164996 13849 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.193531 13850 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.193707 13848 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.194675 13848 ts_tablet_manager.cc:1403] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.113s	user 0.027s	sys 0.013s
I20260709 13:07:15.196449 13849 tablet_bootstrap.cc:654] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.198402 13848 raft_consensus.cc:359] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.199263 13848 raft_consensus.cc:385] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.199584 13848 raft_consensus.cc:740] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.201646 13848 consensus_queue.cc:260] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.203115 13850 tablet_bootstrap.cc:654] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.204157 13759 heartbeater.cc:499] Master 127.0.107.254:37867 was elected leader, sending a full tablet report...
I20260709 13:07:15.204717 13848 ts_tablet_manager.cc:1434] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.004s
I20260709 13:07:15.204916 13849 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.205540 13849 ts_tablet_manager.cc:1403] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.041s	user 0.008s	sys 0.007s
I20260709 13:07:15.205935 13848 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.209203 13849 raft_consensus.cc:359] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.210189 13849 raft_consensus.cc:385] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.210516 13849 raft_consensus.cc:740] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.211340 13849 consensus_queue.cc:260] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.215632 13848 tablet_bootstrap.cc:654] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:15.221722 13605 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:15.231827 13684 heartbeater.cc:499] Master 127.0.107.254:37867 was elected leader, sending a full tablet report...
I20260709 13:07:15.233431 13850 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.234200 13850 ts_tablet_manager.cc:1403] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.041s	user 0.024s	sys 0.011s
I20260709 13:07:15.238188 13850 raft_consensus.cc:359] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.238940 13849 ts_tablet_manager.cc:1434] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.033s	user 0.018s	sys 0.008s
I20260709 13:07:15.239187 13850 raft_consensus.cc:385] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.239754 13850 raft_consensus.cc:740] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.240865 13850 consensus_queue.cc:260] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.242383 13849 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.251380 13850 ts_tablet_manager.cc:1434] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.004s
I20260709 13:07:15.252643 13833 heartbeater.cc:499] Master 127.0.107.254:37867 was elected leader, sending a full tablet report...
I20260709 13:07:15.253226 13849 tablet_bootstrap.cc:654] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.260234 13850 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.267596 13849 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.268461 13849 ts_tablet_manager.cc:1403] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.004s
I20260709 13:07:15.270882 13850 tablet_bootstrap.cc:654] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.272469 13849 raft_consensus.cc:359] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.273593 13849 raft_consensus.cc:385] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.273960 13849 raft_consensus.cc:740] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.275142 13848 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.275790 13848 ts_tablet_manager.cc:1403] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.070s	user 0.024s	sys 0.036s
I20260709 13:07:15.279582 13848 raft_consensus.cc:359] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.280431 13848 raft_consensus.cc:385] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.280772 13848 raft_consensus.cc:740] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.282145 13848 consensus_queue.cc:260] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.284734 13848 ts_tablet_manager.cc:1434] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.006s
I20260709 13:07:15.275280 13849 consensus_queue.cc:260] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.288329 13849 ts_tablet_manager.cc:1434] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.019s	user 0.011s	sys 0.000s
I20260709 13:07:15.285813 13848 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
W20260709 13:07:15.292202 13760 tablet.cc:2406] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 13:07:15.300202 13848 tablet_bootstrap.cc:654] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.301810 13849 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.308164 13850 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.308938 13850 ts_tablet_manager.cc:1403] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.049s	user 0.026s	sys 0.013s
I20260709 13:07:15.309998 13849 tablet_bootstrap.cc:654] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.311714 13848 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.312304 13848 ts_tablet_manager.cc:1403] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.002s
I20260709 13:07:15.313210 13850 raft_consensus.cc:359] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.314169 13850 raft_consensus.cc:385] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.314481 13850 raft_consensus.cc:740] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.315676 13849 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.315356 13850 consensus_queue.cc:260] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.315881 13848 raft_consensus.cc:359] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.316285 13849 ts_tablet_manager.cc:1403] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.027s	user 0.004s	sys 0.007s
I20260709 13:07:15.316709 13848 raft_consensus.cc:385] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.317184 13848 raft_consensus.cc:740] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.317864 13850 ts_tablet_manager.cc:1434] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.003s
I20260709 13:07:15.318102 13848 consensus_queue.cc:260] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.319092 13850 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.320359 13849 raft_consensus.cc:359] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.321398 13849 raft_consensus.cc:385] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.321815 13849 raft_consensus.cc:740] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.322849 13849 consensus_queue.cc:260] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.325407 13849 ts_tablet_manager.cc:1434] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.001s
I20260709 13:07:15.326681 13849 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.328591 13850 tablet_bootstrap.cc:654] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.320746 13848 ts_tablet_manager.cc:1434] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.006s
I20260709 13:07:15.332633 13848 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.336825 13849 tablet_bootstrap.cc:654] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.339895 13848 tablet_bootstrap.cc:654] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.343097 13850 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.343147 13849 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.343937 13850 ts_tablet_manager.cc:1403] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.025s	user 0.010s	sys 0.004s
I20260709 13:07:15.344416 13849 ts_tablet_manager.cc:1403] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.005s
I20260709 13:07:15.346601 13848 tablet_bootstrap.cc:492] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.347276 13848 ts_tablet_manager.cc:1403] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.015s	user 0.008s	sys 0.005s
I20260709 13:07:15.347805 13849 raft_consensus.cc:359] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.348686 13849 raft_consensus.cc:385] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.348351 13850 raft_consensus.cc:359] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.349067 13849 raft_consensus.cc:740] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.349368 13850 raft_consensus.cc:385] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.349977 13850 raft_consensus.cc:740] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.349963 13849 consensus_queue.cc:260] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.351279 13857 raft_consensus.cc:493] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.350970 13848 raft_consensus.cc:359] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.351042 13850 consensus_queue.cc:260] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.351949 13848 raft_consensus.cc:385] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.351853 13857 raft_consensus.cc:515] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.352434 13848 raft_consensus.cc:740] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.353667 13848 consensus_queue.cc:260] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.355628 13849 ts_tablet_manager.cc:1434] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 13:07:15.357074 13849 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.357329 13848 ts_tablet_manager.cc:1434] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 13:07:15.359184 13848 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.362558 13857 leader_election.cc:290] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.363154 13850 ts_tablet_manager.cc:1434] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.019s	user 0.012s	sys 0.003s
I20260709 13:07:15.364995 13850 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.366709 13849 tablet_bootstrap.cc:654] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.367100 13848 tablet_bootstrap.cc:654] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.382696 13850 tablet_bootstrap.cc:654] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.391394 13857 raft_consensus.cc:493] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.393240 13848 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.393894 13853 raft_consensus.cc:493] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.394201 13848 ts_tablet_manager.cc:1403] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.035s	user 0.016s	sys 0.017s
I20260709 13:07:15.394642 13853 raft_consensus.cc:515] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.399358 13848 raft_consensus.cc:359] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.400733 13848 raft_consensus.cc:385] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.404181 13849 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.405088 13849 ts_tablet_manager.cc:1403] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.048s	user 0.013s	sys 0.017s
I20260709 13:07:15.405959 13855 raft_consensus.cc:493] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.406667 13855 raft_consensus.cc:515] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.408202 13853 leader_election.cc:290] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.409895 13849 raft_consensus.cc:359] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.411288 13849 raft_consensus.cc:385] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.411777 13849 raft_consensus.cc:740] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.412079 13855 leader_election.cc:290] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.392974 13857 raft_consensus.cc:515] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.413144 13849 consensus_queue.cc:260] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.401173 13848 raft_consensus.cc:740] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.416841 13849 ts_tablet_manager.cc:1434] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.011s	user 0.013s	sys 0.000s
I20260709 13:07:15.417652 13848 consensus_queue.cc:260] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.422551 13849 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.416555 13857 leader_election.cc:290] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.425084 13848 ts_tablet_manager.cc:1434] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.030s	user 0.010s	sys 0.000s
I20260709 13:07:15.426268 13848 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.435024 13848 tablet_bootstrap.cc:654] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.448647 13850 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.454336 13855 raft_consensus.cc:493] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.455986 13855 raft_consensus.cc:515] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.456454 13850 ts_tablet_manager.cc:1403] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.092s	user 0.027s	sys 0.019s
I20260709 13:07:15.462954 13855 leader_election.cc:290] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.475360 13849 tablet_bootstrap.cc:654] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.479885 13855 raft_consensus.cc:493] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.480577 13855 raft_consensus.cc:515] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.483234 13855 leader_election.cc:290] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.486594 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.488106 13658 raft_consensus.cc:2468] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:15.486523 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e543699c1e4102aaab869bca2218da" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.490267 13659 raft_consensus.cc:2468] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:15.490640 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.491389 13658 raft_consensus.cc:2468] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:15.491885 13774 leader_election.cc:304] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.493901 13857 raft_consensus.cc:2804] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.494555 13857 raft_consensus.cc:493] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.492659 13700 leader_election.cc:304] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.495113 13857 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.495893 13853 raft_consensus.cc:2804] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.496312 13853 raft_consensus.cc:493] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.497061 13853 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.497255 13848 tablet_bootstrap.cc:492] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.497965 13848 ts_tablet_manager.cc:1403] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.072s	user 0.013s	sys 0.011s
I20260709 13:07:15.501962 13774 leader_election.cc:304] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.502017 13848 raft_consensus.cc:359] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.503348 13848 raft_consensus.cc:385] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.503435 13865 raft_consensus.cc:2804] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.504388 13865 raft_consensus.cc:493] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.504918 13865 raft_consensus.cc:3060] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.511847 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.513724 13848 raft_consensus.cc:740] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.514394 13734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.515177 13734 raft_consensus.cc:2468] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.513271 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.516263 13865 raft_consensus.cc:515] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.518218 13625 leader_election.cc:304] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.519232 13855 raft_consensus.cc:2804] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.486436 13850 raft_consensus.cc:359] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
W20260709 13:07:15.519335 13625 leader_election.cc:343] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.519690 13855 raft_consensus.cc:493] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.520144 13865 leader_election.cc:290] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.520275 13855 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.514741 13848 consensus_queue.cc:260] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.521901 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e543699c1e4102aaab869bca2218da" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:15.522631 13658 raft_consensus.cc:3060] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.524276 13853 raft_consensus.cc:515] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.526139 13733 raft_consensus.cc:2393] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1db7c835d25447f863c4568a0ebdfb0 in current term 1: Already voted for candidate eaaf91b262f947d0b34c629d7a1e1e9a in this term.
I20260709 13:07:15.526916 13853 leader_election.cc:290] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.527122 13857 raft_consensus.cc:515] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.531028 13857 leader_election.cc:290] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.531284 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:15.532554 13659 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.536568 13657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:15.537046 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.538003 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
W20260709 13:07:15.538733 13700 leader_election.cc:343] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:15.539529 13700 leader_election.cc:343] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.540735 13855 raft_consensus.cc:515] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.541878 13657 raft_consensus.cc:2393] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 35925c1afcb24b089513c51bff743a60 in current term 1: Already voted for candidate d1db7c835d25447f863c4568a0ebdfb0 in this term.
I20260709 13:07:15.545954 13658 raft_consensus.cc:2468] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:15.519802 13850 raft_consensus.cc:385] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.546903 13850 raft_consensus.cc:740] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.547466 13774 leader_election.cc:304] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.547621 13659 raft_consensus.cc:2468] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:15.547875 13850 consensus_queue.cc:260] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.548893 13857 raft_consensus.cc:2804] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.549958 13700 leader_election.cc:304] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60
I20260709 13:07:15.551498 13853 raft_consensus.cc:2804] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.551995 13853 raft_consensus.cc:697] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:15.552862 13853 consensus_queue.cc:237] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.554080 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.555224 13733 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.551501 13855 leader_election.cc:290] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.561049 13857 raft_consensus.cc:697] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:15.563829 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.564832 13857 consensus_queue.cc:237] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.566367 13808 raft_consensus.cc:2393] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1db7c835d25447f863c4568a0ebdfb0 in current term 1: Already voted for candidate 35925c1afcb24b089513c51bff743a60 in this term.
I20260709 13:07:15.568908 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.572105 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.575850 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.576196 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e543699c1e4102aaab869bca2218da" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.577695 13734 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.578794 13731 raft_consensus.cc:3060] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.578426 13849 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.578465 13807 raft_consensus.cc:2393] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1db7c835d25447f863c4568a0ebdfb0 in current term 1: Already voted for candidate 35925c1afcb24b089513c51bff743a60 in this term.
W20260709 13:07:15.576786 13625 leader_election.cc:343] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.576565 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5e543699c1e4102aaab869bca2218da" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.582456 13625 leader_election.cc:304] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0; no voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:15.583427 13850 ts_tablet_manager.cc:1434] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.124s	user 0.007s	sys 0.002s
I20260709 13:07:15.577116 13730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.583882 13855 raft_consensus.cc:2749] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 13:07:15.584426 13625 leader_election.cc:343] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.584978 13625 leader_election.cc:304] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0; no voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:15.585592 13774 leader_election.cc:304] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35925c1afcb24b089513c51bff743a60; no voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:15.586051 13855 raft_consensus.cc:2749] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:15.586968 13865 raft_consensus.cc:2749] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 13:07:15.582286 13849 ts_tablet_manager.cc:1403] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.160s	user 0.020s	sys 0.026s
I20260709 13:07:15.591382 13733 raft_consensus.cc:2468] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.591241 13849 raft_consensus.cc:359] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.592051 13849 raft_consensus.cc:385] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.596566 13850 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.592185 13731 raft_consensus.cc:2468] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:15.601992 13625 leader_election.cc:304] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60
I20260709 13:07:15.603157 13855 raft_consensus.cc:2804] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.603582 13554 catalog_manager.cc:5697] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.605248 13848 ts_tablet_manager.cc:1434] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.107s	user 0.019s	sys 0.038s
I20260709 13:07:15.606211 13865 raft_consensus.cc:493] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.606694 13848 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.606895 13865 raft_consensus.cc:515] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.592393 13849 raft_consensus.cc:740] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.612057 13849 consensus_queue.cc:260] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.612874 13865 leader_election.cc:290] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.618551 13850 tablet_bootstrap.cc:654] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.621073 13848 tablet_bootstrap.cc:654] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.623185 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.624548 13855 raft_consensus.cc:697] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:15.625118 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.630970 13849 ts_tablet_manager.cc:1434] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.043s	user 0.007s	sys 0.003s
W20260709 13:07:15.628958 13774 leader_election.cc:343] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.634526 13731 raft_consensus.cc:2468] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:15.636425 13849 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.639762 13855 consensus_queue.cc:237] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.640161 13557 catalog_manager.cc:5697] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:15.641019 13774 leader_election.cc:304] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:15.638990 13883 raft_consensus.cc:493] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.643400 13853 raft_consensus.cc:493] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.643288 13883 raft_consensus.cc:515] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.644510 13853 raft_consensus.cc:515] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.646202 13883 leader_election.cc:290] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.647303 13853 leader_election.cc:290] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.648425 13857 raft_consensus.cc:2804] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.649219 13857 raft_consensus.cc:493] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.650996 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.650378 13857 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.654080 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.655301 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
W20260709 13:07:15.657624 13700 leader_election.cc:343] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.658627 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
W20260709 13:07:15.661062 13700 leader_election.cc:343] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
W20260709 13:07:15.662601 13700 leader_election.cc:343] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.663051 13700 leader_election.cc:304] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:15.663975 13853 raft_consensus.cc:2749] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:15.674901 13849 tablet_bootstrap.cc:654] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.668093 13857 raft_consensus.cc:515] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.680963 13807 raft_consensus.cc:2393] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eaaf91b262f947d0b34c629d7a1e1e9a in current term 1: Already voted for candidate 35925c1afcb24b089513c51bff743a60 in this term.
I20260709 13:07:15.683375 13700 leader_election.cc:304] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:15.683885 13850 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.684680 13853 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.685117 13848 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.685681 13848 ts_tablet_manager.cc:1403] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.079s	user 0.021s	sys 0.024s
I20260709 13:07:15.689018 13849 tablet_bootstrap.cc:492] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.689459 13849 ts_tablet_manager.cc:1403] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.053s	user 0.015s	sys 0.005s
I20260709 13:07:15.684702 13850 ts_tablet_manager.cc:1403] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.100s	user 0.014s	sys 0.005s
I20260709 13:07:15.689400 13848 raft_consensus.cc:359] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.690416 13848 raft_consensus.cc:385] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.690757 13848 raft_consensus.cc:740] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.691742 13848 consensus_queue.cc:260] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.693156 13849 raft_consensus.cc:359] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.693451 13850 raft_consensus.cc:359] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.693989 13849 raft_consensus.cc:385] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.694231 13850 raft_consensus.cc:385] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.694451 13849 raft_consensus.cc:740] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.694625 13850 raft_consensus.cc:740] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.694718 13848 ts_tablet_manager.cc:1434] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 13:07:15.696027 13848 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.695504 13849 consensus_queue.cc:260] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.695618 13850 consensus_queue.cc:260] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.693949 13853 raft_consensus.cc:2749] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:15.697328 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:15.698416 13849 ts_tablet_manager.cc:1434] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 13:07:15.699558 13849 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
W20260709 13:07:15.699512 13774 leader_election.cc:343] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.700194 13850 ts_tablet_manager.cc:1434] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.004s
I20260709 13:07:15.701117 13857 leader_election.cc:290] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.701284 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.701354 13850 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.707671 13849 tablet_bootstrap.cc:654] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.712724 13850 tablet_bootstrap.cc:654] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.718219 13848 tablet_bootstrap.cc:654] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.720175 13731 raft_consensus.cc:2468] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:15.724323 13774 leader_election.cc:304] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:15.725380 13857 raft_consensus.cc:2804] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.726770 13857 raft_consensus.cc:697] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:15.727432 13849 tablet_bootstrap.cc:492] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.727692 13857 consensus_queue.cc:237] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.728082 13849 ts_tablet_manager.cc:1403] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.003s
I20260709 13:07:15.734395 13849 raft_consensus.cc:359] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.735165 13849 raft_consensus.cc:385] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.735535 13849 raft_consensus.cc:740] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.736356 13855 raft_consensus.cc:493] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.735008 13556 catalog_manager.cc:5697] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.736927 13855 raft_consensus.cc:515] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.739502 13855 leader_election.cc:290] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.736135 13849 consensus_queue.cc:260] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.740374 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.740676 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.741641 13848 tablet_bootstrap.cc:492] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.741914 13849 ts_tablet_manager.cc:1434] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
W20260709 13:07:15.741837 13625 leader_election.cc:343] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.742192 13848 ts_tablet_manager.cc:1403] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.005s
I20260709 13:07:15.741355 13807 raft_consensus.cc:2468] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.743171 13849 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:15.744102 13625 leader_election.cc:304] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:15.745384 13855 raft_consensus.cc:2804] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.745868 13855 raft_consensus.cc:493] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.746245 13855 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.749701 13888 raft_consensus.cc:493] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.750383 13849 tablet_bootstrap.cc:654] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.750281 13888 raft_consensus.cc:515] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.752911 13888 leader_election.cc:290] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.755796 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.759752 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.761046 13731 raft_consensus.cc:2393] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1db7c835d25447f863c4568a0ebdfb0 in current term 1: Already voted for candidate 35925c1afcb24b089513c51bff743a60 in this term.
I20260709 13:07:15.762951 13625 leader_election.cc:304] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0; no voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:15.764801 13888 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.745352 13848 raft_consensus.cc:359] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.783288 13848 raft_consensus.cc:385] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.783766 13848 raft_consensus.cc:740] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.785151 13848 consensus_queue.cc:260] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.793177 13888 raft_consensus.cc:2749] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:15.794180 13850 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.794854 13850 ts_tablet_manager.cc:1403] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.094s	user 0.024s	sys 0.028s
I20260709 13:07:15.795497 13848 ts_tablet_manager.cc:1434] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.053s	user 0.006s	sys 0.002s
I20260709 13:07:15.796664 13848 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:15.799805 13855 raft_consensus.cc:515] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.802731 13855 leader_election.cc:290] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.802542 13850 raft_consensus.cc:359] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.803598 13850 raft_consensus.cc:385] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.803984 13850 raft_consensus.cc:740] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.805020 13850 consensus_queue.cc:260] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.805248 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.806282 13731 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.806489 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.807385 13807 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.810065 13849 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:15.810714 13849 ts_tablet_manager.cc:1403] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.068s	user 0.023s	sys 0.043s
I20260709 13:07:15.810949 13848 tablet_bootstrap.cc:654] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.814239 13849 raft_consensus.cc:359] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.815125 13849 raft_consensus.cc:385] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.815426 13849 raft_consensus.cc:740] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.816196 13849 consensus_queue.cc:260] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.817562 13807 raft_consensus.cc:2468] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.819034 13625 leader_election.cc:304] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.819978 13855 raft_consensus.cc:2804] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.820463 13855 raft_consensus.cc:697] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:15.821060 13731 raft_consensus.cc:2468] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.821637 13849 ts_tablet_manager.cc:1434] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 13:07:15.821341 13855 consensus_queue.cc:237] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.821712 13850 ts_tablet_manager.cc:1434] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.002s
I20260709 13:07:15.824414 13850 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.801033 13554 catalog_manager.cc:5697] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:15.832026 13850 tablet_bootstrap.cc:654] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.834051 13848 tablet_bootstrap.cc:492] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:15.834722 13848 ts_tablet_manager.cc:1403] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.038s	user 0.014s	sys 0.006s
I20260709 13:07:15.835749 13855 raft_consensus.cc:493] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.836333 13855 raft_consensus.cc:515] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.838619 13855 leader_election.cc:290] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.838438 13848 raft_consensus.cc:359] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.838970 13850 tablet_bootstrap.cc:492] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.839318 13848 raft_consensus.cc:385] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.839766 13848 raft_consensus.cc:740] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.839838 13850 ts_tablet_manager.cc:1403] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.016s	user 0.010s	sys 0.004s
I20260709 13:07:15.840775 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.840649 13848 consensus_queue.cc:260] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.841271 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.841522 13731 raft_consensus.cc:2468] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.843182 13848 ts_tablet_manager.cc:1434] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
W20260709 13:07:15.843111 13625 leader_election.cc:343] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.845681 13625 leader_election.cc:304] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60
I20260709 13:07:15.846392 13855 raft_consensus.cc:2804] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.846818 13855 raft_consensus.cc:493] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.847159 13855 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.846542 13557 catalog_manager.cc:5697] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.848649 13850 raft_consensus.cc:359] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.849602 13850 raft_consensus.cc:385] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.853363 13888 raft_consensus.cc:493] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.853962 13888 raft_consensus.cc:515] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.854712 13855 raft_consensus.cc:515] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.856240 13888 leader_election.cc:290] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.856916 13855 leader_election.cc:290] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.858331 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.858384 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.858726 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.858858 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.859246 13807 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.859344 13731 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.859592 13733 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.860343 13850 raft_consensus.cc:740] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.861619 13850 consensus_queue.cc:260] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.866034 13850 ts_tablet_manager.cc:1434] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.026s	user 0.009s	sys 0.000s
I20260709 13:07:15.866463 13625 leader_election.cc:304] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.867290 13850 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.867460 13855 raft_consensus.cc:2804] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.868106 13855 raft_consensus.cc:493] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.868541 13855 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.872385 13733 raft_consensus.cc:2468] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.875869 13888 raft_consensus.cc:493] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.876574 13888 raft_consensus.cc:515] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
W20260709 13:07:15.867621 13625 leader_election.cc:343] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.878074 13855 raft_consensus.cc:515] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.879050 13625 leader_election.cc:304] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 35925c1afcb24b089513c51bff743a60
I20260709 13:07:15.879292 13888 leader_election.cc:290] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.880008 13886 raft_consensus.cc:2804] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.880492 13855 leader_election.cc:290] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.882398 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.883070 13731 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.883859 13850 tablet_bootstrap.cc:654] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.882290 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:15.889789 13731 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.891144 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.891827 13733 raft_consensus.cc:2468] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 0.
I20260709 13:07:15.893317 13625 leader_election.cc:304] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.893859 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.894338 13888 raft_consensus.cc:2804] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.894563 13807 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.894407 13625 leader_election.cc:304] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.895431 13888 raft_consensus.cc:697] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
W20260709 13:07:15.895853 13625 leader_election.cc:343] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:15.896263 13904 raft_consensus.cc:2804] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.896785 13904 raft_consensus.cc:493] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.897171 13904 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.896564 13888 consensus_queue.cc:237] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.900894 13850 tablet_bootstrap.cc:492] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.901561 13850 ts_tablet_manager.cc:1403] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.035s	user 0.017s	sys 0.004s
I20260709 13:07:15.901630 13886 raft_consensus.cc:697] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:15.902794 13886 consensus_queue.cc:237] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.904603 13857 raft_consensus.cc:493] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:15.905230 13857 raft_consensus.cc:515] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.906631 13850 raft_consensus.cc:359] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.907598 13850 raft_consensus.cc:385] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.907997 13850 raft_consensus.cc:740] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.908563 13857 leader_election.cc:290] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.909063 13850 consensus_queue.cc:260] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.911175 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:15.905274 13807 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.911644 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:15.912364 13733 raft_consensus.cc:2468] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:15.914897 13850 ts_tablet_manager.cc:1434] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 13:07:15.916216 13850 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:15.911947 13658 raft_consensus.cc:2468] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:15.921663 13774 leader_election.cc:304] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.922566 13857 raft_consensus.cc:2804] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:15.922961 13857 raft_consensus.cc:493] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:15.923370 13857 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.925922 13850 tablet_bootstrap.cc:654] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:15.933245 13857 raft_consensus.cc:515] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.933260 13904 raft_consensus.cc:515] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.936386 13850 tablet_bootstrap.cc:492] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:15.937084 13850 ts_tablet_manager.cc:1403] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.021s	user 0.008s	sys 0.008s
I20260709 13:07:15.937973 13857 leader_election.cc:290] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:15.937847 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:15.938776 13658 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.938930 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.939210 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:15.939669 13733 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.939877 13731 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.941129 13850 raft_consensus.cc:359] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.941483 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:15.941949 13850 raft_consensus.cc:385] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:15.942399 13850 raft_consensus.cc:740] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:15.943372 13850 consensus_queue.cc:260] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.944307 13807 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.947337 13731 raft_consensus.cc:2468] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.948093 13658 raft_consensus.cc:2468] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:15.948503 13625 leader_election.cc:304] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:15.949452 13888 raft_consensus.cc:2804] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.949537 13774 leader_election.cc:304] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:15.950105 13888 raft_consensus.cc:697] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:15.950976 13865 raft_consensus.cc:2804] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:15.951072 13888 consensus_queue.cc:237] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.952970 13807 raft_consensus.cc:2468] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 1.
I20260709 13:07:15.953832 13733 raft_consensus.cc:2468] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:15.937028 13904 leader_election.cc:290] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:15.962072 13850 ts_tablet_manager.cc:1434] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.024s	user 0.011s	sys 0.000s
I20260709 13:07:15.964072 13865 raft_consensus.cc:697] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:15.965734 13865 consensus_queue.cc:237] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:15.966001 13554 catalog_manager.cc:5697] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.976498 13557 catalog_manager.cc:5697] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:15.983989 13554 catalog_manager.cc:5697] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.986066 13554 catalog_manager.cc:5697] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 1 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:15.988945 13807 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:15.999661 13853 consensus_queue.cc:1048] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.017686 13853 consensus_queue.cc:1048] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.051554 13888 consensus_queue.cc:1048] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.064270 13904 consensus_queue.cc:1048] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.122975 13865 consensus_queue.cc:1048] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.135358 13909 consensus_queue.cc:1048] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.152848 13909 consensus_queue.cc:1048] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.165917 13909 consensus_queue.cc:1048] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.267616 13733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:43302
I20260709 13:07:16.268340 13733 raft_consensus.cc:606] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:16.287278 13807 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:16.288005 13807 raft_consensus.cc:606] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:16.293913 13807 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:16.294525 13807 raft_consensus.cc:606] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:16.310853 13658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:56572
I20260709 13:07:16.311551 13658 raft_consensus.cc:606] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.318044 13658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f854ca56c664652bc916639fdb53422"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:56572
I20260709 13:07:16.318720 13658 raft_consensus.cc:606] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.319414 13904 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.319931 13909 raft_consensus.cc:493] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:16.320649 13909 raft_consensus.cc:515] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.323222 13909 leader_election.cc:290] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:16.325963 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:16.326157 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:16.327207 13733 raft_consensus.cc:2393] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 35925c1afcb24b089513c51bff743a60 in current term 1: Already voted for candidate d1db7c835d25447f863c4568a0ebdfb0 in this term.
I20260709 13:07:16.328613 13774 leader_election.cc:304] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35925c1afcb24b089513c51bff743a60; no voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.329564 13865 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:16.337649 13886 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.341768 13865 raft_consensus.cc:2749] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:16.342307 13658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9658f588616b4965afe18bde6242cd10"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:56572
I20260709 13:07:16.343029 13658 raft_consensus.cc:606] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.349550 13888 raft_consensus.cc:993] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.350375 13855 raft_consensus.cc:1081] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.350977 13888 consensus_queue.cc:1048] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.352721 13733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f854ca56c664652bc916639fdb53422"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43286
I20260709 13:07:16.353457 13733 raft_consensus.cc:493] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.353936 13733 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.354480 13658 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:56572
I20260709 13:07:16.355152 13658 raft_consensus.cc:606] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.366773 13855 consensus_queue.cc:1048] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.367611 13904 consensus_queue.cc:1048] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.370508 13733 raft_consensus.cc:515] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.375191 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:16.375885 13658 raft_consensus.cc:3055] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.376225 13658 raft_consensus.cc:740] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.377142 13658 consensus_queue.cc:260] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.378381 13658 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.386375 13904 consensus_queue.cc:1048] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.387959 13658 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.389441 13733 leader_election.cc:290] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.389410 13700 leader_election.cc:304] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:16.391113 13914 raft_consensus.cc:2804] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.391618 13914 raft_consensus.cc:697] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:16.392571 13914 consensus_queue.cc:237] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.393219 13855 raft_consensus.cc:993] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.394121 13855 raft_consensus.cc:1081] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.395372 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.396085 13807 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.397714 13730 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9658f588616b4965afe18bde6242cd10"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43286
I20260709 13:07:16.398701 13730 raft_consensus.cc:493] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.399252 13730 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.399888 13886 consensus_queue.cc:1048] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:16.411149 13865 consensus_queue.cc:1048] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.418007 13807 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.431967 13730 raft_consensus.cc:515] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.440145 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:16.440975 13659 raft_consensus.cc:3055] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.441354 13659 raft_consensus.cc:740] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.442198 13886 raft_consensus.cc:993] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.442970 13886 raft_consensus.cc:1081] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.442312 13659 consensus_queue.cc:260] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.443477 13865 raft_consensus.cc:993] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.443930 13659 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.450129 13806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9658f588616b4965afe18bde6242cd10" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.450954 13806 raft_consensus.cc:3060] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.451490 13732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43286
I20260709 13:07:16.452873 13659 raft_consensus.cc:2468] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.453557 13732 raft_consensus.cc:493] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.453946 13730 leader_election.cc:290] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.454041 13732 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.443964 13865 raft_consensus.cc:1081] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.460505 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56542
I20260709 13:07:16.460805 13865 consensus_queue.cc:1048] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.462198 13658 raft_consensus.cc:493] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.462649 13658 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.466681 13732 raft_consensus.cc:515] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.471712 13658 raft_consensus.cc:515] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.464731 13557 catalog_manager.cc:5697] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.472294 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.476643 13808 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.472995 13732 leader_election.cc:290] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.477360 13700 leader_election.cc:304] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:16.478636 13914 raft_consensus.cc:2804] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.479161 13914 raft_consensus.cc:697] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:16.480180 13914 consensus_queue.cc:237] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.484858 13658 leader_election.cc:290] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.489293 13806 raft_consensus.cc:2468] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.491496 13904 consensus_queue.cc:1048] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:16.493686 13808 raft_consensus.cc:2468] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.495059 13855 raft_consensus.cc:993] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.495651 13886 raft_consensus.cc:1081] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:16.497870 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.496945 13730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33c76e168db14336b308a33b6c8fe0a2" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:16.499315 13807 raft_consensus.cc:3055] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.499658 13807 raft_consensus.cc:740] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:16.499788 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9eb03140cf1455fad19606f10f303a0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:16.500623 13658 raft_consensus.cc:3055] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.498487 13732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43286
I20260709 13:07:16.498183 13700 leader_election.cc:304] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:16.501010 13658 raft_consensus.cc:740] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.501971 13732 raft_consensus.cc:493] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.502372 13914 raft_consensus.cc:2804] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.502429 13732 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.503120 13914 raft_consensus.cc:697] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:16.502931 13658 consensus_queue.cc:260] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.504364 13658 raft_consensus.cc:3060] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.504110 13914 consensus_queue.cc:237] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.498268 13805 raft_consensus.cc:1240] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Rejecting Update request from peer d1db7c835d25447f863c4568a0ebdfb0 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 13:07:16.501022 13807 consensus_queue.cc:260] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.511173 13807 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.512104 13732 raft_consensus.cc:515] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.515482 13658 raft_consensus.cc:2468] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.519258 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:16.519923 13658 raft_consensus.cc:3055] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.520301 13658 raft_consensus.cc:740] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.521201 13658 consensus_queue.cc:260] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.521416 13807 raft_consensus.cc:2468] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:16.522830 13732 leader_election.cc:290] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.523332 13625 leader_election.cc:304] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:16.524317 13658 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.524693 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeeb0552c48d49e3b9b89e56a35f2139" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.525179 13904 raft_consensus.cc:2804] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.525466 13807 raft_consensus.cc:3060] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.525763 13904 raft_consensus.cc:697] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.528447 13557 catalog_manager.cc:5697] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.530105 13904 consensus_queue.cc:237] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.530606 13557 catalog_manager.cc:5697] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.533987 13658 raft_consensus.cc:2468] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.535612 13700 leader_election.cc:304] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:16.536401 13914 raft_consensus.cc:2804] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.536929 13914 raft_consensus.cc:697] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:16.537910 13914 consensus_queue.cc:237] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.539908 13807 raft_consensus.cc:2468] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:16.543722 13554 catalog_manager.cc:5697] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.553071 13557 catalog_manager.cc:5697] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.627394 13883 raft_consensus.cc:993] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.627979 13883 raft_consensus.cc:1081] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.629669 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56556
I20260709 13:07:16.630262 13658 raft_consensus.cc:493] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.630599 13658 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.637176 13658 raft_consensus.cc:515] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.639456 13658 leader_election.cc:290] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.640429 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:16.640739 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c13cbf3a996844aeb37dfa1735fea267" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.641232 13732 raft_consensus.cc:3055] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.641467 13807 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.641660 13732 raft_consensus.cc:740] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:16.642637 13732 consensus_queue.cc:260] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.643783 13732 raft_consensus.cc:3060] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.648088 13807 raft_consensus.cc:2468] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:16.649926 13625 leader_election.cc:304] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:16.650851 13732 raft_consensus.cc:2468] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:16.651098 13904 raft_consensus.cc:2804] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.651872 13904 raft_consensus.cc:697] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.652781 13904 consensus_queue.cc:237] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.661067 13557 catalog_manager.cc:5697] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.715934 13857 raft_consensus.cc:993] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.716488 13865 raft_consensus.cc:1081] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:16.718403 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56542
I20260709 13:07:16.719043 13658 raft_consensus.cc:493] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:16.719410 13658 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.726413 13658 raft_consensus.cc:515] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.728243 13658 leader_election.cc:290] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:16.729475 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:16.729727 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "95f724ccd859431ab29f0d3a03c10d7d" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:16.730540 13807 raft_consensus.cc:3055] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:16.730655 13732 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.730840 13807 raft_consensus.cc:740] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:16.731814 13807 consensus_queue.cc:260] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.732879 13807 raft_consensus.cc:3060] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.737138 13732 raft_consensus.cc:2468] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:16.738653 13625 leader_election.cc:304] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:16.739193 13807 raft_consensus.cc:2468] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:16.739423 13904 raft_consensus.cc:2804] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:16.739998 13904 raft_consensus.cc:697] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:16.740809 13904 consensus_queue.cc:237] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:16.747494 13557 catalog_manager.cc:5697] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:16.867276 13807 raft_consensus.cc:1275] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.868883 13883 consensus_queue.cc:1048] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.877626 13658 raft_consensus.cc:1275] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.879736 13853 consensus_queue.cc:1048] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.909196 13807 raft_consensus.cc:1275] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.910831 13853 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.919584 13658 raft_consensus.cc:1275] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.921936 13883 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.958022 13807 raft_consensus.cc:1275] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.959582 13904 consensus_queue.cc:1048] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:16.965121 13732 raft_consensus.cc:3060] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:16.975952 13732 raft_consensus.cc:1275] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:16.977438 13904 consensus_queue.cc:1048] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:17.024551 13807 raft_consensus.cc:1275] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.026147 13883 consensus_queue.cc:1048] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:17.038201 13658 raft_consensus.cc:1275] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.040427 13938 consensus_queue.cc:1048] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:17.116370 13807 raft_consensus.cc:1275] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.118022 13904 consensus_queue.cc:1048] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:17.119709 13732 raft_consensus.cc:1275] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.121423 13904 consensus_queue.cc:1048] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:17.137480 13807 raft_consensus.cc:1275] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.138998 13938 consensus_queue.cc:1048] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:17.148298 13658 raft_consensus.cc:1275] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.149875 13938 consensus_queue.cc:1048] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:17.193783 13807 raft_consensus.cc:1275] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.195333 13904 consensus_queue.cc:1048] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:17.205646 13731 raft_consensus.cc:1275] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:17.207336 13888 consensus_queue.cc:1048] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:19.400421 13554 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43454:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 13:07:19.403625 13554 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 13:07:19.467859 13643 tablet_service.cc:1511] Processing CreateTablet for tablet 250d27f18efd4d40b06571188ac8f886 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:19.469177 13722 tablet_service.cc:1511] Processing CreateTablet for tablet 250d27f18efd4d40b06571188ac8f886 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:19.470353 13643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250d27f18efd4d40b06571188ac8f886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.471208 13722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250d27f18efd4d40b06571188ac8f886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.471323 13790 tablet_service.cc:1511] Processing CreateTablet for tablet 250d27f18efd4d40b06571188ac8f886 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 13:07:19.473158 13790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 250d27f18efd4d40b06571188ac8f886. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.477303 13646 tablet_service.cc:1511] Processing CreateTablet for tablet 0fef9ad967ec49f58cad358e2f6db6f0 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:19.479301 13646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9ad967ec49f58cad358e2f6db6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.486706 13723 tablet_service.cc:1511] Processing CreateTablet for tablet 0fef9ad967ec49f58cad358e2f6db6f0 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:19.488571 13723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9ad967ec49f58cad358e2f6db6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.490864 13716 tablet_service.cc:1511] Processing CreateTablet for tablet ea8693d094624e5bb9e8707e2e36a94f (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:19.492624 13716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8693d094624e5bb9e8707e2e36a94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.496165 13794 tablet_service.cc:1511] Processing CreateTablet for tablet ea8693d094624e5bb9e8707e2e36a94f (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:19.497872 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8693d094624e5bb9e8707e2e36a94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.498222 13644 tablet_service.cc:1511] Processing CreateTablet for tablet ea8693d094624e5bb9e8707e2e36a94f (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 13:07:19.500267 13644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea8693d094624e5bb9e8707e2e36a94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.496212 13792 tablet_service.cc:1511] Processing CreateTablet for tablet 0fef9ad967ec49f58cad358e2f6db6f0 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 13:07:19.503929 13792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fef9ad967ec49f58cad358e2f6db6f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.503575 13720 tablet_service.cc:1511] Processing CreateTablet for tablet 4b6fb7d0f2d643d8b074d13ff01714cc (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:19.505143 13720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6fb7d0f2d643d8b074d13ff01714cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.511005 13719 tablet_service.cc:1511] Processing CreateTablet for tablet 159c42149fc34fbfbcfe3e7f307470d4 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:19.512714 13719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159c42149fc34fbfbcfe3e7f307470d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.516804 13798 tablet_service.cc:1511] Processing CreateTablet for tablet 159c42149fc34fbfbcfe3e7f307470d4 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:19.516876 13796 tablet_service.cc:1511] Processing CreateTablet for tablet 4b6fb7d0f2d643d8b074d13ff01714cc (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:19.518910 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159c42149fc34fbfbcfe3e7f307470d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.527673 13647 tablet_service.cc:1511] Processing CreateTablet for tablet 4b6fb7d0f2d643d8b074d13ff01714cc (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 13:07:19.530196 13647 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6fb7d0f2d643d8b074d13ff01714cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.530789 13796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b6fb7d0f2d643d8b074d13ff01714cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.527675 13648 tablet_service.cc:1511] Processing CreateTablet for tablet 159c42149fc34fbfbcfe3e7f307470d4 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 13:07:19.535157 13648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 159c42149fc34fbfbcfe3e7f307470d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.553648 13649 tablet_service.cc:1511] Processing CreateTablet for tablet aeb67901eab84bbf868d1df5a856f74b (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:19.555413 13649 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb67901eab84bbf868d1df5a856f74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.559826 13956 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.570024 13724 tablet_service.cc:1511] Processing CreateTablet for tablet aeb67901eab84bbf868d1df5a856f74b (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:19.571755 13724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb67901eab84bbf868d1df5a856f74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.582402 13956 tablet_bootstrap.cc:654] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.583403 13957 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:19.590081 13797 tablet_service.cc:1511] Processing CreateTablet for tablet aeb67901eab84bbf868d1df5a856f74b (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 13:07:19.594485 13797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aeb67901eab84bbf868d1df5a856f74b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.603837 13643 tablet_service.cc:1511] Processing CreateTablet for tablet 463deb9a0ac04dc289f2d82912573bf3 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:19.610226 13722 tablet_service.cc:1511] Processing CreateTablet for tablet 463deb9a0ac04dc289f2d82912573bf3 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:19.612083 13722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463deb9a0ac04dc289f2d82912573bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.636224 13643 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463deb9a0ac04dc289f2d82912573bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.650750 13646 tablet_service.cc:1511] Processing CreateTablet for tablet 338f7381a9124fcd974ca807bbd6d79e (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:19.653785 13646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338f7381a9124fcd974ca807bbd6d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.659569 13716 tablet_service.cc:1511] Processing CreateTablet for tablet 338f7381a9124fcd974ca807bbd6d79e (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:19.662788 13958 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:19.629102 13794 tablet_service.cc:1511] Processing CreateTablet for tablet 463deb9a0ac04dc289f2d82912573bf3 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 13:07:19.676805 13794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 463deb9a0ac04dc289f2d82912573bf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.678678 13716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338f7381a9124fcd974ca807bbd6d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.680866 13645 tablet_service.cc:1511] Processing CreateTablet for tablet da7c8845c9c047e8b1137cc9ffa18653 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:19.692377 13645 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7c8845c9c047e8b1137cc9ffa18653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.701968 13798 tablet_service.cc:1511] Processing CreateTablet for tablet 338f7381a9124fcd974ca807bbd6d79e (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 13:07:19.704094 13798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 338f7381a9124fcd974ca807bbd6d79e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.715354 13720 tablet_service.cc:1511] Processing CreateTablet for tablet da7c8845c9c047e8b1137cc9ffa18653 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:19.716898 13720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7c8845c9c047e8b1137cc9ffa18653. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 13:07:19.727993 13790 tablet_service.cc:1511] Processing CreateTablet for tablet da7c8845c9c047e8b1137cc9ffa18653 (DEFAULT_TABLE table=soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 13:07:19.709690 13956 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:19.730475 13956 ts_tablet_manager.cc:1403] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.171s	user 0.017s	sys 0.000s
I20260709 13:07:19.736186 13956 raft_consensus.cc:359] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.737227 13956 raft_consensus.cc:385] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.737596 13956 raft_consensus.cc:740] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.738617 13790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da7c8845c9c047e8b1137cc9ffa18653. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 13:07:19.755889 13774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.194:38169, user_credentials={real_user=slave}} blocked reactor thread for 67182us
I20260709 13:07:19.758425 13958 tablet_bootstrap.cc:654] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.761281 13956 consensus_queue.cc:260] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.765692 13957 tablet_bootstrap.cc:654] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.777529 13956 ts_tablet_manager.cc:1434] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.047s	user 0.000s	sys 0.010s
I20260709 13:07:19.778892 13956 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.795135 13957 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:19.795751 13957 ts_tablet_manager.cc:1403] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.213s	user 0.010s	sys 0.005s
I20260709 13:07:19.799629 13957 raft_consensus.cc:359] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.800534 13957 raft_consensus.cc:385] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.800889 13957 raft_consensus.cc:740] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.801348 13958 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:19.802042 13958 ts_tablet_manager.cc:1403] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.140s	user 0.013s	sys 0.003s
I20260709 13:07:19.801911 13957 consensus_queue.cc:260] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.805325 13957 ts_tablet_manager.cc:1434] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 13:07:19.806504 13957 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:19.815142 13957 tablet_bootstrap.cc:654] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.818285 13958 raft_consensus.cc:359] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.819200 13958 raft_consensus.cc:385] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.819540 13958 raft_consensus.cc:740] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.820228 13956 tablet_bootstrap.cc:654] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.820473 13958 consensus_queue.cc:260] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.827105 13956 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:19.827790 13956 ts_tablet_manager.cc:1403] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.049s	user 0.012s	sys 0.004s
I20260709 13:07:19.827880 13957 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:19.828289 13958 ts_tablet_manager.cc:1434] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 13:07:19.828562 13957 ts_tablet_manager.cc:1403] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.000s
I20260709 13:07:19.829571 13958 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:19.831849 13956 raft_consensus.cc:359] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.832165 13957 raft_consensus.cc:359] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.832966 13956 raft_consensus.cc:385] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.833334 13957 raft_consensus.cc:385] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.833456 13956 raft_consensus.cc:740] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.833784 13957 raft_consensus.cc:740] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.834668 13957 consensus_queue.cc:260] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.836997 13957 ts_tablet_manager.cc:1434] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.004s
I20260709 13:07:19.834546 13956 consensus_queue.cc:260] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.839433 13958 tablet_bootstrap.cc:654] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.839990 13956 ts_tablet_manager.cc:1434] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260709 13:07:19.841357 13956 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.850961 13956 tablet_bootstrap.cc:654] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.852504 13958 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:19.853173 13958 ts_tablet_manager.cc:1403] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.011s
I20260709 13:07:19.855676 13957 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:19.857822 13958 raft_consensus.cc:359] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.859000 13958 raft_consensus.cc:385] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.859438 13958 raft_consensus.cc:740] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.860569 13958 consensus_queue.cc:260] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.863152 13958 ts_tablet_manager.cc:1434] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.010s	user 0.011s	sys 0.000s
I20260709 13:07:19.865404 13958 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:19.867388 13956 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:19.868134 13956 ts_tablet_manager.cc:1403] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.006s
I20260709 13:07:19.871723 13956 raft_consensus.cc:359] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.872501 13956 raft_consensus.cc:385] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.872812 13956 raft_consensus.cc:740] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.873011 13957 tablet_bootstrap.cc:654] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.873049 13958 tablet_bootstrap.cc:654] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.873696 13956 consensus_queue.cc:260] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.876681 13956 ts_tablet_manager.cc:1434] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 13:07:19.880344 13956 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.889566 13956 tablet_bootstrap.cc:654] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.895211 13960 raft_consensus.cc:493] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:19.896260 13960 raft_consensus.cc:515] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.900935 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:19.902047 13658 raft_consensus.cc:2468] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:19.904299 13700 leader_election.cc:304] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:19.905408 13853 raft_consensus.cc:2804] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:19.905826 13853 raft_consensus.cc:493] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:19.906785 13853 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.907017 13960 leader_election.cc:290] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:19.908126 13958 tablet_bootstrap.cc:492] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:19.908424 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:19.908788 13958 ts_tablet_manager.cc:1403] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.044s	user 0.026s	sys 0.012s
W20260709 13:07:19.910518 13700 leader_election.cc:343] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:19.910571 13956 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:19.912211 13956 ts_tablet_manager.cc:1403] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.007s
I20260709 13:07:19.915861 13957 tablet_bootstrap.cc:492] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:19.915632 13956 raft_consensus.cc:359] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.916584 13956 raft_consensus.cc:385] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.916759 13957 ts_tablet_manager.cc:1403] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.079s	user 0.013s	sys 0.014s
I20260709 13:07:19.918289 13958 raft_consensus.cc:359] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.919031 13958 raft_consensus.cc:385] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.919309 13958 raft_consensus.cc:740] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.920132 13958 consensus_queue.cc:260] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.917155 13956 raft_consensus.cc:740] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.921052 13957 raft_consensus.cc:359] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.922559 13957 raft_consensus.cc:385] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.922099 13956 consensus_queue.cc:260] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.922943 13957 raft_consensus.cc:740] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.925168 13956 ts_tablet_manager.cc:1434] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.013s	user 0.000s	sys 0.007s
I20260709 13:07:19.925489 13958 ts_tablet_manager.cc:1434] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.002s
I20260709 13:07:19.926334 13956 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.926672 13958 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:19.929282 13853 raft_consensus.cc:515] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.933674 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:19.934324 13658 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.935185 13958 tablet_bootstrap.cc:654] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.935278 13956 tablet_bootstrap.cc:654] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.945228 13956 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:19.945916 13956 ts_tablet_manager.cc:1403] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.011s
I20260709 13:07:19.947691 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:19.948431 13808 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:19.950703 13853 leader_election.cc:290] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:19.951337 13658 raft_consensus.cc:2468] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:19.951210 13956 raft_consensus.cc:359] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.951992 13956 raft_consensus.cc:385] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.952327 13956 raft_consensus.cc:740] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.953104 13700 leader_election.cc:304] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:19.953142 13956 consensus_queue.cc:260] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.954517 13960 raft_consensus.cc:2804] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:19.955032 13960 raft_consensus.cc:697] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:19.954533 13958 tablet_bootstrap.cc:492] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:19.956169 13958 ts_tablet_manager.cc:1403] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.030s	user 0.020s	sys 0.004s
I20260709 13:07:19.955986 13960 consensus_queue.cc:237] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.959796 13958 raft_consensus.cc:359] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.960551 13958 raft_consensus.cc:385] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:19.961637 13808 raft_consensus.cc:2468] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:19.960893 13958 raft_consensus.cc:740] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:19.948896 13957 consensus_queue.cc:260] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.963167 13958 consensus_queue.cc:260] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:19.968449 13956 ts_tablet_manager.cc:1434] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.004s
I20260709 13:07:19.970796 13956 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:19.972222 13958 ts_tablet_manager.cc:1434] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:07:19.973549 13958 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:19.978642 13957 ts_tablet_manager.cc:1434] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.061s	user 0.006s	sys 0.001s
I20260709 13:07:19.980149 13957 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:19.981527 13958 tablet_bootstrap.cc:654] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.981597 13956 tablet_bootstrap.cc:654] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.989965 13957 tablet_bootstrap.cc:654] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:19.991077 13557 catalog_manager.cc:5697] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.001549 13958 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:20.002298 13958 ts_tablet_manager.cc:1403] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.029s	user 0.008s	sys 0.012s
I20260709 13:07:20.006072 13958 raft_consensus.cc:359] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.006958 13958 raft_consensus.cc:385] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.007326 13958 raft_consensus.cc:740] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.008435 13958 consensus_queue.cc:260] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.019970 13956 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:20.020848 13956 ts_tablet_manager.cc:1403] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.050s	user 0.021s	sys 0.017s
I20260709 13:07:20.021665 13958 ts_tablet_manager.cc:1434] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 13:07:20.023383 13958 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:20.024441 13956 raft_consensus.cc:359] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.025278 13956 raft_consensus.cc:385] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.025653 13956 raft_consensus.cc:740] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.026494 13956 consensus_queue.cc:260] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.027243 13957 tablet_bootstrap.cc:492] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.028258 13957 ts_tablet_manager.cc:1403] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.048s	user 0.005s	sys 0.011s
I20260709 13:07:20.029196 13956 ts_tablet_manager.cc:1434] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 13:07:20.030517 13956 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:20.035632 13960 raft_consensus.cc:493] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.036422 13960 raft_consensus.cc:515] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.037690 13956 tablet_bootstrap.cc:654] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.036906 13957 raft_consensus.cc:359] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.039116 13957 raft_consensus.cc:385] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.039470 13957 raft_consensus.cc:740] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.040081 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.040797 13658 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.040513 13957 consensus_queue.cc:260] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.042469 13958 tablet_bootstrap.cc:654] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.043583 13957 ts_tablet_manager.cc:1434] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.015s	user 0.011s	sys 0.000s
I20260709 13:07:20.043254 13700 leader_election.cc:304] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.044499 13853 raft_consensus.cc:2804] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.044847 13853 raft_consensus.cc:493] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.045300 13853 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.046787 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:20.047737 13808 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.050565 13960 leader_election.cc:290] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.044832 13957 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:20.056825 13853 raft_consensus.cc:515] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.072403 13853 leader_election.cc:290] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.077621 13957 tablet_bootstrap.cc:654] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.081077 13958 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:20.081766 13958 ts_tablet_manager.cc:1403] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.059s	user 0.011s	sys 0.019s
I20260709 13:07:20.087078 13958 raft_consensus.cc:359] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.088294 13958 raft_consensus.cc:385] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.088742 13958 raft_consensus.cc:740] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.089962 13958 consensus_queue.cc:260] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.094924 13958 ts_tablet_manager.cc:1434] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 13:07:20.101732 13956 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:20.102475 13956 ts_tablet_manager.cc:1403] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.072s	user 0.016s	sys 0.040s
I20260709 13:07:20.106248 13956 raft_consensus.cc:359] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.107033 13956 raft_consensus.cc:385] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.107339 13956 raft_consensus.cc:740] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.107506 13958 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:20.108556 13956 consensus_queue.cc:260] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.110491 13853 raft_consensus.cc:493] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.111711 13853 raft_consensus.cc:515] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.116499 13853 leader_election.cc:290] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.117947 13962 raft_consensus.cc:493] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 13:07:20.118750 13700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.193:35387, user_credentials={real_user=slave}} blocked reactor thread for 60330.3us
I20260709 13:07:20.118855 13962 raft_consensus.cc:515] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.121649 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:20.122054 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.122938 13658 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.124308 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.125244 13659 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.122320 13808 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.128158 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:20.128881 13807 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.118319 13956 ts_tablet_manager.cc:1434] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 13:07:20.131899 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.132939 13659 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.135408 13956 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:20.136691 13774 leader_election.cc:304] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:20.138429 13961 raft_consensus.cc:2804] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.138936 13961 raft_consensus.cc:493] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.139361 13961 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.144786 13956 tablet_bootstrap.cc:654] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.146327 13962 leader_election.cc:290] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.148191 13958 tablet_bootstrap.cc:654] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.149135 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:20.151680 13731 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.155483 13658 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.157872 13700 leader_election.cc:304] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.158317 13957 tablet_bootstrap.cc:492] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.159042 13957 ts_tablet_manager.cc:1403] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.114s	user 0.040s	sys 0.044s
I20260709 13:07:20.162897 13984 raft_consensus.cc:2804] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.163548 13984 raft_consensus.cc:493] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.164470 13984 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.162904 13957 raft_consensus.cc:359] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.165079 13957 raft_consensus.cc:385] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.166020 13957 raft_consensus.cc:740] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.163113 13961 raft_consensus.cc:515] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.168329 13700 leader_election.cc:304] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.171339 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.172721 13658 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.174147 13960 raft_consensus.cc:2804] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.174861 13960 raft_consensus.cc:697] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:20.175804 13960 consensus_queue.cc:237] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.177443 13984 raft_consensus.cc:515] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.179945 13957 consensus_queue.cc:260] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.181681 13984 leader_election.cc:290] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.183636 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:20.185672 13658 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.187001 13731 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.188936 13961 leader_election.cc:290] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.190063 13774 leader_election.cc:304] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:20.197243 13961 raft_consensus.cc:2804] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.198817 13961 raft_consensus.cc:697] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:20.199982 13961 consensus_queue.cc:237] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.200330 13731 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.191407 13957 ts_tablet_manager.cc:1434] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.004s
I20260709 13:07:20.204031 13957 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:20.202687 13557 catalog_manager.cc:5697] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.206311 13962 raft_consensus.cc:493] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.207036 13962 raft_consensus.cc:515] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
W20260709 13:07:20.214978 13700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.107.193:35387, user_credentials={real_user=slave}} blocked reactor thread for 46758.9us
I20260709 13:07:20.216259 13957 tablet_bootstrap.cc:654] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.217271 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.218143 13659 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.217530 13958 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:20.219848 13958 ts_tablet_manager.cc:1403] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.113s	user 0.017s	sys 0.003s
I20260709 13:07:20.225772 13961 raft_consensus.cc:493] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.225811 13958 raft_consensus.cc:359] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.226879 13958 raft_consensus.cc:385] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.226495 13961 raft_consensus.cc:515] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.227185 13958 raft_consensus.cc:740] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.228108 13958 consensus_queue.cc:260] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.229292 13961 leader_election.cc:290] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.231226 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
W20260709 13:07:20.234195 13774 leader_election.cc:343] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.236369 13808 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.236683 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.237478 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:20.237773 13658 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.238140 13807 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.241890 13659 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.232767 13555 catalog_manager.cc:5697] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:20.246345 13774 leader_election.cc:304] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:20.247634 13961 raft_consensus.cc:2804] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.248116 13961 raft_consensus.cc:493] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.248533 13961 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.248720 13958 ts_tablet_manager.cc:1434] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.028s	user 0.015s	sys 0.001s
I20260709 13:07:20.250612 13958 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:20.255671 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.255726 13956 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:20.257046 13956 ts_tablet_manager.cc:1403] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.122s	user 0.037s	sys 0.033s
W20260709 13:07:20.258103 13774 leader_election.cc:343] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.258741 13774 leader_election.cc:304] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 35925c1afcb24b089513c51bff743a60; no voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:20.260620 13991 raft_consensus.cc:2749] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.264842 13958 tablet_bootstrap.cc:654] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.265728 13962 leader_election.cc:290] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.266386 13956 raft_consensus.cc:359] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.266680 13961 raft_consensus.cc:515] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.267244 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:20.268226 13731 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.269850 13961 leader_election.cc:290] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.267189 13956 raft_consensus.cc:385] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.270530 13956 raft_consensus.cc:740] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.271446 13956 consensus_queue.cc:260] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.271582 13731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:20.274144 13956 ts_tablet_manager.cc:1434] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 13:07:20.274572 13700 leader_election.cc:304] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.274977 13807 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.275547 13960 raft_consensus.cc:2804] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
W20260709 13:07:20.281893 13605 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 13:07:20.276358 13731 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.281934 13960 raft_consensus.cc:697] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:20.283453 13960 consensus_queue.cc:237] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.279111 13957 tablet_bootstrap.cc:492] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.280288 13956 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: Bootstrap starting.
I20260709 13:07:20.271582 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.288425 13658 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.288576 13958 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:20.289242 13958 ts_tablet_manager.cc:1403] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.004s
I20260709 13:07:20.293718 13958 raft_consensus.cc:359] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.294524 13958 raft_consensus.cc:385] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.294808 13958 raft_consensus.cc:740] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.295714 13958 consensus_queue.cc:260] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.297207 13957 ts_tablet_manager.cc:1403] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.084s	user 0.016s	sys 0.013s
I20260709 13:07:20.302850 13658 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.303494 13957 raft_consensus.cc:359] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.304311 13957 raft_consensus.cc:385] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.304409 13774 leader_election.cc:304] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:20.304656 13957 raft_consensus.cc:740] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.298749 13961 raft_consensus.cc:493] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.305583 13962 raft_consensus.cc:2804] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.306135 13962 raft_consensus.cc:697] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:20.306015 13961 raft_consensus.cc:515] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.307489 13962 consensus_queue.cc:237] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.309016 13961 leader_election.cc:290] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.298362 13958 ts_tablet_manager.cc:1434] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 13:07:20.299858 13956 tablet_bootstrap.cc:654] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.312490 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:20.312716 13731 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.313433 13732 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.314736 13958 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: Bootstrap starting.
I20260709 13:07:20.314707 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.305608 13957 consensus_queue.cc:260] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.316473 13774 leader_election.cc:304] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.317767 13659 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.319891 13961 raft_consensus.cc:2804] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.320569 13961 raft_consensus.cc:493] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.321040 13961 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.324435 13958 tablet_bootstrap.cc:654] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.302613 13555 catalog_manager.cc:5697] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.333859 13956 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: No bootstrap required, opened a new log
I20260709 13:07:20.334574 13956 ts_tablet_manager.cc:1403] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent bootstrapping tablet: real 0.055s	user 0.013s	sys 0.002s
I20260709 13:07:20.339849 13956 raft_consensus.cc:359] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.340737 13956 raft_consensus.cc:385] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.340330 13961 raft_consensus.cc:515] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.341156 13956 raft_consensus.cc:740] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.341821 13957 ts_tablet_manager.cc:1434] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.042s	user 0.010s	sys 0.000s
I20260709 13:07:20.342347 13956 consensus_queue.cc:260] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.344348 13961 leader_election.cc:290] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.344412 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.354018 13659 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.346802 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:20.353231 13956 ts_tablet_manager.cc:1434] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 13:07:20.355468 13732 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.357964 13958 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: No bootstrap required, opened a new log
I20260709 13:07:20.358702 13958 ts_tablet_manager.cc:1403] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: Time spent bootstrapping tablet: real 0.044s	user 0.020s	sys 0.003s
I20260709 13:07:20.363365 13958 raft_consensus.cc:359] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.364341 13958 raft_consensus.cc:385] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.364674 13958 raft_consensus.cc:740] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.365888 13958 consensus_queue.cc:260] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.367465 13957 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:20.370025 13958 ts_tablet_manager.cc:1434] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20260709 13:07:20.366222 13557 catalog_manager.cc:5697] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:20.377017 13732 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.379204 13774 leader_election.cc:304] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.380445 13962 raft_consensus.cc:2804] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.381639 13962 raft_consensus.cc:697] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:20.383008 13962 consensus_queue.cc:237] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.385116 13957 tablet_bootstrap.cc:654] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.391949 13659 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.400411 13957 tablet_bootstrap.cc:492] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.399994 13853 consensus_queue.cc:1048] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.401182 13957 ts_tablet_manager.cc:1403] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.034s	user 0.022s	sys 0.000s
I20260709 13:07:20.403681 13853 raft_consensus.cc:493] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.404433 13853 raft_consensus.cc:515] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.404901 13957 raft_consensus.cc:359] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.406037 13957 raft_consensus.cc:385] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.406351 13957 raft_consensus.cc:740] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.407806 13853 leader_election.cc:290] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.407719 13957 consensus_queue.cc:260] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.412138 13957 ts_tablet_manager.cc:1434] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 13:07:20.413352 13957 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:20.414264 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:20.415606 13808 raft_consensus.cc:2468] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.417840 13700 leader_election.cc:304] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.414974 13557 catalog_manager.cc:5697] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:20.417922 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.419160 13853 raft_consensus.cc:2804] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.419667 13853 raft_consensus.cc:493] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.420113 13853 raft_consensus.cc:3060] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.423254 13957 tablet_bootstrap.cc:654] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
W20260709 13:07:20.419975 13700 leader_election.cc:343] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.433318 13853 raft_consensus.cc:515] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.437757 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.437919 13853 leader_election.cc:290] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.438946 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
W20260709 13:07:20.439306 13700 leader_election.cc:343] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Illegal state: must be running to vote when last-logged opid is not known
I20260709 13:07:20.439800 13808 raft_consensus.cc:3060] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.444348 13957 tablet_bootstrap.cc:492] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.445132 13957 ts_tablet_manager.cc:1403] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.032s	user 0.021s	sys 0.000s
I20260709 13:07:20.448822 13957 raft_consensus.cc:359] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.449904 13957 raft_consensus.cc:385] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.450320 13957 raft_consensus.cc:740] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.451367 13957 consensus_queue.cc:260] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.452111 13808 raft_consensus.cc:2468] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.454216 13957 ts_tablet_manager.cc:1434] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 13:07:20.454520 13700 leader_election.cc:304] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:20.455804 13957 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Bootstrap starting.
I20260709 13:07:20.464591 13957 tablet_bootstrap.cc:654] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Neither blocks nor log segments found. Creating new log.
I20260709 13:07:20.467185 13853 raft_consensus.cc:2804] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.470218 13853 raft_consensus.cc:697] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:20.471627 13853 consensus_queue.cc:237] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.475214 13957 tablet_bootstrap.cc:492] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: No bootstrap required, opened a new log
I20260709 13:07:20.475916 13957 ts_tablet_manager.cc:1403] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.006s
I20260709 13:07:20.479322 13957 raft_consensus.cc:359] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.480141 13957 raft_consensus.cc:385] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 13:07:20.480427 13957 raft_consensus.cc:740] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Initialized, Role: FOLLOWER
I20260709 13:07:20.483573 13984 consensus_queue.cc:1048] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.481303 13957 consensus_queue.cc:260] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.488606 13957 ts_tablet_manager.cc:1434] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.004s
I20260709 13:07:20.496407 13557 catalog_manager.cc:5697] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.498373 13853 raft_consensus.cc:493] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.499059 13853 raft_consensus.cc:515] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.501657 13853 leader_election.cc:290] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.503284 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.504551 13659 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.503705 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:20.505360 13808 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 0.
I20260709 13:07:20.506345 13700 leader_election.cc:304] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.508111 13853 raft_consensus.cc:2804] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.508692 13853 raft_consensus.cc:493] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.509052 13853 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.517362 13853 raft_consensus.cc:515] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.519884 13853 leader_election.cc:290] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.522069 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.522958 13659 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.528796 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:20.532290 13808 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.533308 13659 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.535320 13700 leader_election.cc:304] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.536180 13853 raft_consensus.cc:2804] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.537189 13853 raft_consensus.cc:697] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:20.538861 13853 consensus_queue.cc:237] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.542624 13808 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 1.
I20260709 13:07:20.552089 13555 catalog_manager.cc:5697] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 1 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:20.598102 13962 raft_consensus.cc:493] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.599015 13962 raft_consensus.cc:515] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.602128 13962 leader_election.cc:290] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.603031 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0" is_pre_election: true
I20260709 13:07:20.603487 13853 consensus_queue.cc:1048] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.603590 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:20.603996 13658 raft_consensus.cc:2468] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.604830 13732 raft_consensus.cc:2468] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 0.
I20260709 13:07:20.605748 13774 leader_election.cc:304] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:20.607167 13962 raft_consensus.cc:2804] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 13:07:20.607803 13962 raft_consensus.cc:493] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 13:07:20.608580 13962 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.622184 13984 consensus_queue.cc:1048] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.622184 13962 raft_consensus.cc:515] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.627225 13962 leader_election.cc:290] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:20.628201 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:20.628407 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:20.628986 13658 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.629158 13732 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.640492 13732 raft_consensus.cc:2468] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.642061 13774 leader_election.cc:304] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:20.642911 13962 raft_consensus.cc:2804] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 13:07:20.643056 13658 raft_consensus.cc:2468] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 1.
I20260709 13:07:20.643575 13962 raft_consensus.cc:697] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:20.644600 13962 consensus_queue.cc:237] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.647154 13855 raft_consensus.cc:493] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 13:07:20.647843 13855 raft_consensus.cc:515] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:20.651153 13855 leader_election.cc:290] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:20.651613 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" is_pre_election: true
I20260709 13:07:20.653228 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "338f7381a9124fcd974ca807bbd6d79e" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35925c1afcb24b089513c51bff743a60" is_pre_election: true
I20260709 13:07:20.654081 13808 raft_consensus.cc:2393] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1db7c835d25447f863c4568a0ebdfb0 in current term 1: Already voted for candidate eaaf91b262f947d0b34c629d7a1e1e9a in this term.
I20260709 13:07:20.655234 13625 leader_election.cc:304] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0; no voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:20.656009 13855 raft_consensus.cc:3060] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 13:07:20.658545 13554 catalog_manager.cc:5697] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 1 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:20.665863 13855 raft_consensus.cc:2749] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 13:07:20.679565 13962 consensus_queue.cc:1048] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.694495 13961 consensus_queue.cc:1048] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.731248 13961 consensus_queue.cc:1048] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.745851 13991 consensus_queue.cc:1048] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.801268 13991 consensus_queue.cc:1048] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.816074 14012 consensus_queue.cc:1048] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.859068 13984 consensus_queue.cc:1048] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:20.876821 13853 consensus_queue.cc:1048] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.878108 13960 consensus_queue.cc:1048] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.892537 13960 consensus_queue.cc:1048] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:20.903770 13732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:43302
I20260709 13:07:20.905146 13732 raft_consensus.cc:606] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:20.914120 13732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:43302
I20260709 13:07:20.914894 13732 raft_consensus.cc:606] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:20.925943 13808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:20.926743 13808 raft_consensus.cc:606] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:20.934000 13732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:43302
I20260709 13:07:20.934656 13732 raft_consensus.cc:606] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Received request to transfer leadership to TS d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:20.940481 13808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:20.941126 13808 raft_consensus.cc:606] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:20.947670 13808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:20.948571 13808 raft_consensus.cc:606] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:20.955399 13808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
mode: GRACEFUL
new_leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:54666
I20260709 13:07:20.956095 13808 raft_consensus.cc:606] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Received request to transfer leadership to TS eaaf91b262f947d0b34c629d7a1e1e9a
I20260709 13:07:20.965921 13732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:43302
I20260709 13:07:20.966711 13732 raft_consensus.cc:606] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Received request to transfer leadership to TS 35925c1afcb24b089513c51bff743a60
I20260709 13:07:21.000236 13960 consensus_queue.cc:1048] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.006757 13960 consensus_queue.cc:1048] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.013173 13984 raft_consensus.cc:993] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:21.013851 13984 raft_consensus.cc:1081] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Signalling peer 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:21.017124 13808 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:54644
I20260709 13:07:21.017822 13808 raft_consensus.cc:493] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.018214 13808 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.027802 13808 raft_consensus.cc:515] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.030244 13808 leader_election.cc:290] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:21.032665 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:21.032969 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "da7c8845c9c047e8b1137cc9ffa18653" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:21.033237 13658 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.033751 13732 raft_consensus.cc:3055] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.034240 13732 raft_consensus.cc:740] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.035593 13732 consensus_queue.cc:260] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.037000 13732 raft_consensus.cc:3060] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.041705 13658 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 2.
I20260709 13:07:21.043067 13774 leader_election.cc:304] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:21.043901 14012 raft_consensus.cc:2804] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.044410 14012 raft_consensus.cc:697] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 2 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:21.044690 13732 raft_consensus.cc:2468] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 2.
I20260709 13:07:21.045300 14012 consensus_queue.cc:237] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.056504 13556 catalog_manager.cc:5697] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 1 to 2, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:21.104339 13853 raft_consensus.cc:993] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.104851 13960 raft_consensus.cc:1081] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.106779 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56556
I20260709 13:07:21.107352 13658 raft_consensus.cc:493] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.107681 13658 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.114872 13658 raft_consensus.cc:515] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.116989 13658 leader_election.cc:290] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.118299 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:21.118315 13808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fef9ad967ec49f58cad358e2f6db6f0" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.119122 13732 raft_consensus.cc:3055] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.119199 13808 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.119469 13732 raft_consensus.cc:740] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.120489 13732 consensus_queue.cc:260] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.121763 13732 raft_consensus.cc:3060] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.125886 13808 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.127677 13625 leader_election.cc:304] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:21.128968 13855 raft_consensus.cc:2804] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.129446 13855 raft_consensus.cc:697] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:21.130511 13732 raft_consensus.cc:2468] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.130419 13855 consensus_queue.cc:237] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.133095 14012 consensus_queue.cc:1048] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 13:07:21.139431 13556 catalog_manager.cc:5697] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.141942 13962 raft_consensus.cc:993] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.142603 13961 raft_consensus.cc:1081] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.144843 13732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43292
I20260709 13:07:21.145534 13732 raft_consensus.cc:493] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.145989 13732 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.156095 13732 raft_consensus.cc:515] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.156436 13962 consensus_queue.cc:1048] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 13:07:21.159605 13659 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:21.165000 13732 leader_election.cc:290] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.166091 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "463deb9a0ac04dc289f2d82912573bf3" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.166870 13807 raft_consensus.cc:3055] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.167177 13807 raft_consensus.cc:740] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:21.173177 13807 consensus_queue.cc:260] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.174816 13807 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.181349 13807 raft_consensus.cc:2468] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:21.182739 13700 leader_election.cc:304] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: d1db7c835d25447f863c4568a0ebdfb0
I20260709 13:07:21.183576 13960 raft_consensus.cc:2804] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.184158 13960 raft_consensus.cc:697] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.185226 13960 consensus_queue.cc:237] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.194782 13555 catalog_manager.cc:5697] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.196066 14012 raft_consensus.cc:993] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.196666 13962 raft_consensus.cc:1081] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.198446 13732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43292
I20260709 13:07:21.198968 13732 raft_consensus.cc:493] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.199337 13732 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.210111 13732 raft_consensus.cc:515] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.212293 13732 leader_election.cc:290] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.213762 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.213977 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "159c42149fc34fbfbcfe3e7f307470d4" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:21.214632 13807 raft_consensus.cc:3055] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.214643 13658 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.215236 13807 raft_consensus.cc:740] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:21.216138 13807 consensus_queue.cc:260] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.217561 13807 raft_consensus.cc:3060] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.224367 13658 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:21.226583 13700 leader_election.cc:304] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:21.226671 13807 raft_consensus.cc:2468] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:21.227531 13960 raft_consensus.cc:2804] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.228089 13960 raft_consensus.cc:697] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.228880 13960 consensus_queue.cc:237] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.243300 13555 catalog_manager.cc:5697] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.261973 14012 raft_consensus.cc:993] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.262621 13962 raft_consensus.cc:1081] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.264266 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56542
I20260709 13:07:21.264765 13658 raft_consensus.cc:493] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.265566 13658 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.272013 13658 raft_consensus.cc:515] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.273802 13658 leader_election.cc:290] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.275305 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:21.275604 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea8693d094624e5bb9e8707e2e36a94f" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.276139 13732 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.276237 13807 raft_consensus.cc:3055] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.276654 13807 raft_consensus.cc:740] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:21.277575 13807 consensus_queue.cc:260] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.278620 13807 raft_consensus.cc:3060] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.283114 13732 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.284893 13625 leader_election.cc:304] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:21.285142 13807 raft_consensus.cc:2468] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.286183 13855 raft_consensus.cc:2804] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.286682 13855 raft_consensus.cc:697] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:21.287487 13855 consensus_queue.cc:237] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.297714 13555 catalog_manager.cc:5697] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.300666 14012 raft_consensus.cc:993] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: : Instructing follower eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.301353 13962 raft_consensus.cc:1081] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Signalling peer eaaf91b262f947d0b34c629d7a1e1e9a to start an election
I20260709 13:07:21.303347 13732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
 from {username='slave'} at 127.0.0.1:43292
I20260709 13:07:21.303952 13732 raft_consensus.cc:493] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.304303 13732 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.317030 13732 raft_consensus.cc:515] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.320286 13732 leader_election.cc:290] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.321820 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.322410 13658 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aeb67901eab84bbf868d1df5a856f74b" candidate_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:21.322540 13807 raft_consensus.cc:3055] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.322992 13658 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.322994 13807 raft_consensus.cc:740] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:21.324004 13807 consensus_queue.cc:260] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.325393 13807 raft_consensus.cc:3060] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.331135 13658 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:21.332108 13807 raft_consensus.cc:2468] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eaaf91b262f947d0b34c629d7a1e1e9a in term 2.
I20260709 13:07:21.332582 13700 leader_election.cc:304] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d1db7c835d25447f863c4568a0ebdfb0, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:21.333902 13960 raft_consensus.cc:2804] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.334408 13960 raft_consensus.cc:697] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Leader. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.335886 13960 consensus_queue.cc:237] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.344105 13555 catalog_manager.cc:5697] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a reported cstate change: term changed from 1 to 2, leader changed from 35925c1afcb24b089513c51bff743a60 (127.0.107.195) to eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194). New cstate: current_term: 2 leader_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.348400 13984 raft_consensus.cc:993] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.349119 13984 raft_consensus.cc:1081] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.352348 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56556
I20260709 13:07:21.353291 13658 raft_consensus.cc:493] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.353868 13658 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.362005 13658 raft_consensus.cc:515] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.365291 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:21.365561 13658 leader_election.cc:290] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.365689 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4b6fb7d0f2d643d8b074d13ff01714cc" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.366155 13732 raft_consensus.cc:3055] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.366698 13807 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.366938 13732 raft_consensus.cc:740] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.367769 13732 consensus_queue.cc:260] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.369025 13732 raft_consensus.cc:3060] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.374812 13807 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.376271 13625 leader_election.cc:304] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:21.376710 13732 raft_consensus.cc:2468] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.377269 13855 raft_consensus.cc:2804] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.377920 13855 raft_consensus.cc:697] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:21.378700 13855 consensus_queue.cc:237] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.387430 13555 catalog_manager.cc:5697] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.430325 13984 raft_consensus.cc:993] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.431092 13853 raft_consensus.cc:1081] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Signalling peer d1db7c835d25447f863c4568a0ebdfb0 to start an election
I20260709 13:07:21.432650 13658 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
 from {username='slave'} at 127.0.0.1:56556
I20260709 13:07:21.433494 13658 raft_consensus.cc:493] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:21.434056 13658 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.439713 13658 raft_consensus.cc:515] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.441996 13658 leader_election.cc:290] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Requested vote from peers eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169), 35925c1afcb24b089513c51bff743a60 (127.0.107.195:33043)
I20260709 13:07:21.442925 13732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:21.443123 13807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "250d27f18efd4d40b06571188ac8f886" candidate_uuid: "d1db7c835d25447f863c4568a0ebdfb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "35925c1afcb24b089513c51bff743a60"
I20260709 13:07:21.443763 13732 raft_consensus.cc:3055] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:21.443951 13807 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.444051 13732 raft_consensus.cc:740] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:21.445258 13732 consensus_queue.cc:260] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.446626 13732 raft_consensus.cc:3060] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.448390 13658 raft_consensus.cc:1275] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.450039 13962 consensus_queue.cc:1048] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.451874 13807 raft_consensus.cc:2468] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.453094 13625 leader_election.cc:304] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:21.453910 13855 raft_consensus.cc:2804] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:21.454332 13855 raft_consensus.cc:697] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Becoming Leader. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:21.454625 13732 raft_consensus.cc:2468] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1db7c835d25447f863c4568a0ebdfb0 in term 2.
I20260709 13:07:21.455140 13855 consensus_queue.cc:237] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:21.469316 13732 raft_consensus.cc:1275] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.470964 14026 consensus_queue.cc:1048] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.472122 13556 catalog_manager.cc:5697] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 reported cstate change: term changed from 1 to 2, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193). New cstate: current_term: 2 leader_uuid: "d1db7c835d25447f863c4568a0ebdfb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: UNKNOWN } } }
I20260709 13:07:21.577907 13807 raft_consensus.cc:1275] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.580226 13855 consensus_queue.cc:1048] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.590515 13732 raft_consensus.cc:1275] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.592409 13979 consensus_queue.cc:1048] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.598951 13807 raft_consensus.cc:1275] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.601118 13853 consensus_queue.cc:1048] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.614930 13658 raft_consensus.cc:3060] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:21.624060 13658 raft_consensus.cc:1275] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.626327 13960 consensus_queue.cc:1048] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.634913 13807 raft_consensus.cc:1275] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.636538 13960 consensus_queue.cc:1048] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.650653 13658 raft_consensus.cc:1275] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.653573 13853 consensus_queue.cc:1048] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.734083 13808 raft_consensus.cc:1275] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.735512 14027 consensus_queue.cc:1048] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.744966 13732 raft_consensus.cc:1275] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.747398 13979 consensus_queue.cc:1048] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.779752 13732 raft_consensus.cc:1275] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.781623 13979 consensus_queue.cc:1048] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.792891 13808 raft_consensus.cc:1275] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.795018 13855 consensus_queue.cc:1048] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.922165 13658 raft_consensus.cc:1275] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.923728 13853 consensus_queue.cc:1048] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.935887 13807 raft_consensus.cc:1275] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer eaaf91b262f947d0b34c629d7a1e1e9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.938362 14029 consensus_queue.cc:1048] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:21.944840 13808 raft_consensus.cc:1275] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.946594 13855 consensus_queue.cc:1048] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:21.959277 13731 raft_consensus.cc:1275] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer d1db7c835d25447f863c4568a0ebdfb0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:21.961973 13855 consensus_queue.cc:1048] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:23.970351 13556 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43454:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 13:07:23.992841 13556 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:43454:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 13:07:24.056106   431 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 13:07:24.113667 13700 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33043, user_credentials={real_user=slave}} blocked reactor thread for 62864.2us
I20260709 13:07:24.364523 13655 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d"
dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
mode: GRACEFUL
new_leader_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:35484
I20260709 13:07:24.365285 13655 raft_consensus.cc:606] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Received request to transfer leadership to TS 35925c1afcb24b089513c51bff743a60
I20260709 13:07:24.378049 14086 tablet.cc:1724] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.407248 14094 tablet.cc:1724] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.423738 14085 tablet.cc:1724] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:24.462774 14074 log.cc:927] Time spent T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
W20260709 13:07:24.527249 13625 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.107.195:33043, user_credentials={real_user=slave}} blocked reactor thread for 115347us
I20260709 13:07:24.546672 14097 tablet.cc:1724] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 13:07:24.551885 14093 log.cc:927] Time spent T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Append to log took a long time: real 0.083s	user 0.002s	sys 0.000s
I20260709 13:07:24.493319 14088 tablet.cc:1724] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.559309 14086 tablet.cc:1724] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.579404 14102 tablet.cc:1724] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.579291 14087 raft_consensus.cc:993] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: : Instructing follower 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:24.594245 14100 tablet.cc:1724] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.597211 13733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f854ca56c664652bc916639fdb53422"
dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
mode: GRACEFUL
new_leader_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:54834
I20260709 13:07:24.598230 13733 raft_consensus.cc:606] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Received request to transfer leadership to TS 35925c1afcb24b089513c51bff743a60
I20260709 13:07:24.600404   431 auto_leader_rebalancer.cc:452] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 13:07:24.601246   431 auto_leader_rebalancer.cc:527] All tables' leader rebalancing finished this round
I20260709 13:07:24.611618 14104 tablet.cc:1724] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.613042 14054 raft_consensus.cc:1081] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Signalling peer 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:24.579077 14106 tablet.cc:1724] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.606747 14105 tablet.cc:1724] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.617269 14107 tablet.cc:1724] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.621989 14085 tablet.cc:1724] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.626253 14103 tablet.cc:1724] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.627216 14094 tablet.cc:1724] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.639933 13804 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:54650
I20260709 13:07:24.640697 13804 raft_consensus.cc:493] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:24.641117 13804 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:24.656209 14097 tablet.cc:1724] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.667361 14106 tablet.cc:1724] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.672196 13804 raft_consensus.cc:515] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.680552 13657 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:24.681327 13657 raft_consensus.cc:3055] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 13:07:24.681713 13657 raft_consensus.cc:740] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1db7c835d25447f863c4568a0ebdfb0, State: Running, Role: LEADER
I20260709 13:07:24.682837 13657 consensus_queue.cc:260] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.673270 14105 tablet.cc:1724] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.682879 14050 raft_consensus.cc:993] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: : Instructing follower 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:24.685461 14050 raft_consensus.cc:1081] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Signalling peer 35925c1afcb24b089513c51bff743a60 to start an election
I20260709 13:07:24.687778 13657 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:24.688019 13803 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f854ca56c664652bc916639fdb53422"
dest_uuid: "35925c1afcb24b089513c51bff743a60"
 from {username='slave'} at 127.0.0.1:54644
I20260709 13:07:24.689226 13803 raft_consensus.cc:493] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 13:07:24.690189 13803 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:24.699654 14103 tablet.cc:1724] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.708899 14086 tablet.cc:1724] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.716655 14088 tablet.cc:1724] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.717399 14100 tablet.cc:1724] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.726862 13733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02fe59bb30944d3bbdda33a2e5c61b7d" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:24.746622 13733 raft_consensus.cc:3060] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 13:07:24.747002 14107 tablet.cc:1724] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.727459 13804 leader_election.cc:290] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 2 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:24.730505 14102 tablet.cc:1724] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.762115 13803 raft_consensus.cc:515] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.770437 13656 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d1db7c835d25447f863c4568a0ebdfb0"
I20260709 13:07:24.771210 13656 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:24.772275 13657 raft_consensus.cc:2468] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 2.
I20260709 13:07:24.720036 14094 tablet.cc:1724] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.779500 13774 leader_election.cc:304] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, d1db7c835d25447f863c4568a0ebdfb0; no voters: 
I20260709 13:07:24.775732 14085 tablet.cc:1724] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.788213 13730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f854ca56c664652bc916639fdb53422" candidate_uuid: "35925c1afcb24b089513c51bff743a60" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a"
I20260709 13:07:24.789531 13730 raft_consensus.cc:3055] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 13:07:24.790087 13730 raft_consensus.cc:740] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: eaaf91b262f947d0b34c629d7a1e1e9a, State: Running, Role: LEADER
I20260709 13:07:24.790915 14026 raft_consensus.cc:2804] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 13:07:24.791895 14026 raft_consensus.cc:697] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 2 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:24.791895 13730 consensus_queue.cc:260] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.794440 13730 raft_consensus.cc:3060] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 13:07:24.793272 14026 consensus_queue.cc:237] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.781786 14106 tablet.cc:1724] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 13:07:24.775966 13803 leader_election.cc:290] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 3 election: Requested vote from peers d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387), eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169)
I20260709 13:07:24.815047 13733 raft_consensus.cc:2468] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 2.
I20260709 13:07:24.823624 13730 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 3.
I20260709 13:07:24.825321 13774 leader_election.cc:304] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35925c1afcb24b089513c51bff743a60, eaaf91b262f947d0b34c629d7a1e1e9a; no voters: 
I20260709 13:07:24.826694 14099 raft_consensus.cc:2804] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 13:07:24.827217 14099 raft_consensus.cc:697] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 3 LEADER]: Becoming Leader. State: Replica: 35925c1afcb24b089513c51bff743a60, State: Running, Role: LEADER
I20260709 13:07:24.828159 14099 consensus_queue.cc:237] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } }
I20260709 13:07:24.805308 13656 raft_consensus.cc:2468] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35925c1afcb24b089513c51bff743a60 in term 3.
I20260709 13:07:24.862984 13556 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: soft_deleted_table [id=df8bfaf046e440a4823ea9eac16be33c] alter complete (version 1)
I20260709 13:07:24.882818 13556 catalog_manager.cc:5697] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 1 to 2, leader changed from d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193) to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 2 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:24.884572 13556 catalog_manager.cc:5697] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 reported cstate change: term changed from 2 to 3, leader changed from eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194) to 35925c1afcb24b089513c51bff743a60 (127.0.107.195). New cstate: current_term: 3 leader_uuid: "35925c1afcb24b089513c51bff743a60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35925c1afcb24b089513c51bff743a60" member_type: VOTER last_known_addr { host: "127.0.107.195" port: 33043 } health_report { overall_health: HEALTHY } } }
I20260709 13:07:25.247444 13657 raft_consensus.cc:1275] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 3 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:25.249774 14026 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 13:07:25.263761 13730 raft_consensus.cc:1275] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 3 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 13:07:25.266072 14091 consensus_queue.cc:1048] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 13:07:25.322965 13657 raft_consensus.cc:1275] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:25.324882 14091 consensus_queue.cc:1048] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1db7c835d25447f863c4568a0ebdfb0" member_type: VOTER last_known_addr { host: "127.0.107.193" port: 35387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 13:07:25.337222 13730 raft_consensus.cc:1275] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Refusing update from remote peer 35925c1afcb24b089513c51bff743a60: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 13:07:25.339491 14091 consensus_queue.cc:1048] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eaaf91b262f947d0b34c629d7a1e1e9a" member_type: VOTER last_known_addr { host: "127.0.107.194" port: 38169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 13:07:25.686340   431 tablet_server.cc:179] TabletServer@127.0.107.193:0 shutting down...
W20260709 13:07:25.730792 13774 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 15 similar messages]
W20260709 13:07:25.737967 13774 consensus_peers.cc:597] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.753782   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:25.754951   431 tablet_replica.cc:333] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.755694   431 raft_consensus.cc:2243] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.756402   431 raft_consensus.cc:2272] T 338f7381a9124fcd974ca807bbd6d79e P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.760090   431 tablet_replica.cc:333] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.760841   431 raft_consensus.cc:2243] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.761420   431 raft_consensus.cc:2272] T da7c8845c9c047e8b1137cc9ffa18653 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.764086   431 tablet_replica.cc:333] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.765091   431 raft_consensus.cc:2243] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.765739   431 raft_consensus.cc:2272] T aeb67901eab84bbf868d1df5a856f74b P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.768843   431 tablet_replica.cc:333] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.770066   431 raft_consensus.cc:2243] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.771472   431 raft_consensus.cc:2272] T ea8693d094624e5bb9e8707e2e36a94f P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.773880   431 tablet_replica.cc:333] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.774585   431 raft_consensus.cc:2243] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.775333   431 raft_consensus.cc:2272] T 159c42149fc34fbfbcfe3e7f307470d4 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.777736   431 tablet_replica.cc:333] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.778409   431 raft_consensus.cc:2243] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.778980   431 raft_consensus.cc:2272] T 8f854ca56c664652bc916639fdb53422 P d1db7c835d25447f863c4568a0ebdfb0 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.783707   431 tablet_replica.cc:333] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.784484   431 raft_consensus.cc:2243] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.786090   431 raft_consensus.cc:2272] T 250d27f18efd4d40b06571188ac8f886 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.794533   431 tablet_replica.cc:333] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.796159   431 raft_consensus.cc:2243] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.796974   431 raft_consensus.cc:2272] T 9658f588616b4965afe18bde6242cd10 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.802094   431 tablet_replica.cc:333] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
W20260709 13:07:25.804692 13700 consensus_peers.cc:597] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.805043   431 raft_consensus.cc:2243] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.807499   431 raft_consensus.cc:2272] T c13cbf3a996844aeb37dfa1735fea267 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:25.814235 13774 consensus_peers.cc:597] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.816198   431 tablet_replica.cc:333] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.817215   431 raft_consensus.cc:2243] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.818881   431 raft_consensus.cc:2272] T 95f724ccd859431ab29f0d3a03c10d7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.821748   431 tablet_replica.cc:333] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.822400   431 raft_consensus.cc:2243] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.823292   431 raft_consensus.cc:2272] T 0fef9ad967ec49f58cad358e2f6db6f0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.825630   431 tablet_replica.cc:333] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.826253   431 raft_consensus.cc:2243] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.826751   431 raft_consensus.cc:2272] T 02fe59bb30944d3bbdda33a2e5c61b7d P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.832516   431 tablet_replica.cc:333] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.833456   431 raft_consensus.cc:2243] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.834954   431 raft_consensus.cc:2272] T 4b6fb7d0f2d643d8b074d13ff01714cc P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.839288   431 tablet_replica.cc:333] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.840341   431 raft_consensus.cc:2243] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.841135   431 raft_consensus.cc:2272] T c9eb03140cf1455fad19606f10f303a0 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.845950   431 tablet_replica.cc:333] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.846661   431 raft_consensus.cc:2243] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.847213   431 raft_consensus.cc:2272] T 463deb9a0ac04dc289f2d82912573bf3 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.850449   431 tablet_replica.cc:333] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.852831   431 raft_consensus.cc:2243] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.853434   431 raft_consensus.cc:2272] T e5e543699c1e4102aaab869bca2218da P d1db7c835d25447f863c4568a0ebdfb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.857173   431 tablet_replica.cc:333] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.857935   431 raft_consensus.cc:2243] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:25.858830   431 raft_consensus.cc:2272] T aeeb0552c48d49e3b9b89e56a35f2139 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:25.863255   431 tablet_replica.cc:333] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0: stopping tablet replica
I20260709 13:07:25.863868   431 raft_consensus.cc:2243] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:25.864899   431 raft_consensus.cc:2272] T 33c76e168db14336b308a33b6c8fe0a2 P d1db7c835d25447f863c4568a0ebdfb0 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 13:07:25.901103 13700 consensus_peers.cc:597] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.912326   431 tablet_server.cc:196] TabletServer@127.0.107.193:0 shutdown complete.
W20260709 13:07:25.914599 13774 consensus_peers.cc:597] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:35387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:25.947582 13774 consensus_peers.cc:597] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:35387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:25.961771   431 tablet_server.cc:179] TabletServer@127.0.107.194:0 shutting down...
W20260709 13:07:25.971771 13700 consensus_peers.cc:597] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a -> Peer d1db7c835d25447f863c4568a0ebdfb0 (127.0.107.193:35387): Couldn't send request to peer d1db7c835d25447f863c4568a0ebdfb0. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.193:35387: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:26.022156   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:26.023447   431 tablet_replica.cc:333] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.024617   431 raft_consensus.cc:2243] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.025954   431 raft_consensus.cc:2272] T 338f7381a9124fcd974ca807bbd6d79e P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.028270   431 tablet_replica.cc:333] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.028893   431 raft_consensus.cc:2243] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.029359   431 raft_consensus.cc:2272] T da7c8845c9c047e8b1137cc9ffa18653 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.031921   431 tablet_replica.cc:333] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.032533   431 raft_consensus.cc:2243] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.033442   431 raft_consensus.cc:2272] T aeb67901eab84bbf868d1df5a856f74b P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.036100   431 tablet_replica.cc:333] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.036760   431 raft_consensus.cc:2243] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.037211   431 raft_consensus.cc:2272] T ea8693d094624e5bb9e8707e2e36a94f P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.040032   431 tablet_replica.cc:333] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.040700   431 raft_consensus.cc:2243] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.041221   431 raft_consensus.cc:2272] T 0fef9ad967ec49f58cad358e2f6db6f0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.043409   431 tablet_replica.cc:333] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.044193   431 raft_consensus.cc:2243] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.045289   431 raft_consensus.cc:2272] T 159c42149fc34fbfbcfe3e7f307470d4 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.047606   431 tablet_replica.cc:333] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.048264   431 raft_consensus.cc:2243] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.048744   431 raft_consensus.cc:2272] T 8f854ca56c664652bc916639fdb53422 P eaaf91b262f947d0b34c629d7a1e1e9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.052517   431 tablet_replica.cc:333] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.053196   431 raft_consensus.cc:2243] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.053826   431 raft_consensus.cc:2272] T 250d27f18efd4d40b06571188ac8f886 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.056082   431 tablet_replica.cc:333] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.056758   431 raft_consensus.cc:2243] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.057839   431 raft_consensus.cc:2272] T 463deb9a0ac04dc289f2d82912573bf3 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.060139   431 tablet_replica.cc:333] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.060777   431 raft_consensus.cc:2243] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.061854   431 raft_consensus.cc:2272] T aeeb0552c48d49e3b9b89e56a35f2139 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.064414   431 tablet_replica.cc:333] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.065065   431 raft_consensus.cc:2243] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.065629   431 raft_consensus.cc:2272] T e5e543699c1e4102aaab869bca2218da P eaaf91b262f947d0b34c629d7a1e1e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.068511   431 tablet_replica.cc:333] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.069147   431 raft_consensus.cc:2243] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.069782   431 raft_consensus.cc:2272] T 33c76e168db14336b308a33b6c8fe0a2 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.072374   431 tablet_replica.cc:333] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.072948   431 raft_consensus.cc:2243] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.073424   431 raft_consensus.cc:2272] T 4b6fb7d0f2d643d8b074d13ff01714cc P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.075780   431 tablet_replica.cc:333] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.076345   431 raft_consensus.cc:2243] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.077488   431 raft_consensus.cc:2272] T c9eb03140cf1455fad19606f10f303a0 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.080045   431 tablet_replica.cc:333] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.080616   431 raft_consensus.cc:2243] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.081108   431 raft_consensus.cc:2272] T 02fe59bb30944d3bbdda33a2e5c61b7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.084276   431 tablet_replica.cc:333] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.085480   431 raft_consensus.cc:2243] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.086592   431 raft_consensus.cc:2272] T 9658f588616b4965afe18bde6242cd10 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.089948   431 tablet_replica.cc:333] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.090701   431 raft_consensus.cc:2243] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.091290   431 raft_consensus.cc:2272] T 95f724ccd859431ab29f0d3a03c10d7d P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.093854   431 tablet_replica.cc:333] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a: stopping tablet replica
I20260709 13:07:26.094357   431 raft_consensus.cc:2243] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.094830   431 raft_consensus.cc:2272] T c13cbf3a996844aeb37dfa1735fea267 P eaaf91b262f947d0b34c629d7a1e1e9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.144402   431 tablet_server.cc:196] TabletServer@127.0.107.194:0 shutdown complete.
W20260709 13:07:26.145476 13774 consensus_peers.cc:597] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 -> Peer eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169): Couldn't send request to peer eaaf91b262f947d0b34c629d7a1e1e9a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:38169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 13:07:26.184983 13774 consensus_peers.cc:597] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 -> Peer eaaf91b262f947d0b34c629d7a1e1e9a (127.0.107.194:38169): Couldn't send request to peer eaaf91b262f947d0b34c629d7a1e1e9a. Status: Network error: Client connection negotiation failed: client connection to 127.0.107.194:38169: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 13:07:26.189414   431 tablet_server.cc:179] TabletServer@127.0.107.195:0 shutting down...
I20260709 13:07:26.226619   431 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 13:07:26.227622   431 tablet_replica.cc:333] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.228482   431 raft_consensus.cc:2243] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.229584   431 raft_consensus.cc:2272] T da7c8845c9c047e8b1137cc9ffa18653 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.231998   431 tablet_replica.cc:333] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.232949   431 raft_consensus.cc:2243] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.233570   431 raft_consensus.cc:2272] T aeb67901eab84bbf868d1df5a856f74b P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.236387   431 tablet_replica.cc:333] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.237121   431 raft_consensus.cc:2243] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.237738   431 raft_consensus.cc:2272] T 338f7381a9124fcd974ca807bbd6d79e P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.240048   431 tablet_replica.cc:333] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.240764   431 raft_consensus.cc:2243] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.241314   431 raft_consensus.cc:2272] T 0fef9ad967ec49f58cad358e2f6db6f0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.243628   431 tablet_replica.cc:333] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.244326   431 raft_consensus.cc:2243] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.244874   431 raft_consensus.cc:2272] T 159c42149fc34fbfbcfe3e7f307470d4 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.247233   431 tablet_replica.cc:333] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.248000   431 raft_consensus.cc:2243] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.248546   431 raft_consensus.cc:2272] T ea8693d094624e5bb9e8707e2e36a94f P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.250754   431 tablet_replica.cc:333] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.251475   431 raft_consensus.cc:2243] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.251991   431 raft_consensus.cc:2272] T 9658f588616b4965afe18bde6242cd10 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.254232   431 tablet_replica.cc:333] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.254851   431 raft_consensus.cc:2243] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.255353   431 raft_consensus.cc:2272] T 95f724ccd859431ab29f0d3a03c10d7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.257755   431 tablet_replica.cc:333] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.258535   431 raft_consensus.cc:2243] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.259114   431 raft_consensus.cc:2272] T c13cbf3a996844aeb37dfa1735fea267 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.261575   431 tablet_replica.cc:333] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.262526   431 raft_consensus.cc:2243] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.263222   431 raft_consensus.cc:2272] T 463deb9a0ac04dc289f2d82912573bf3 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.266333   431 tablet_replica.cc:333] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.267032   431 raft_consensus.cc:2243] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.267966   431 raft_consensus.cc:2272] T e5e543699c1e4102aaab869bca2218da P 35925c1afcb24b089513c51bff743a60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.270452   431 tablet_replica.cc:333] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.271055   431 raft_consensus.cc:2243] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.271499   431 raft_consensus.cc:2272] T aeeb0552c48d49e3b9b89e56a35f2139 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.274057   431 tablet_replica.cc:333] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.274616   431 raft_consensus.cc:2243] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.275043   431 raft_consensus.cc:2272] T 33c76e168db14336b308a33b6c8fe0a2 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.277273   431 tablet_replica.cc:333] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.277940   431 raft_consensus.cc:2243] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 2 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.278775   431 raft_consensus.cc:2272] T 02fe59bb30944d3bbdda33a2e5c61b7d P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.283007   431 tablet_replica.cc:333] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.283923   431 raft_consensus.cc:2243] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.284672   431 raft_consensus.cc:2272] T 250d27f18efd4d40b06571188ac8f886 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.287348   431 tablet_replica.cc:333] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.287916   431 raft_consensus.cc:2243] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 3 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.288897   431 raft_consensus.cc:2272] T 8f854ca56c664652bc916639fdb53422 P 35925c1afcb24b089513c51bff743a60 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.291580   431 tablet_replica.cc:333] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.292196   431 raft_consensus.cc:2243] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.292722   431 raft_consensus.cc:2272] T 4b6fb7d0f2d643d8b074d13ff01714cc P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.295060   431 tablet_replica.cc:333] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60: stopping tablet replica
I20260709 13:07:26.295660   431 raft_consensus.cc:2243] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 13:07:26.296161   431 raft_consensus.cc:2272] T c9eb03140cf1455fad19606f10f303a0 P 35925c1afcb24b089513c51bff743a60 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.335747   431 tablet_server.cc:196] TabletServer@127.0.107.195:0 shutdown complete.
I20260709 13:07:26.378690   431 master.cc:562] Master@127.0.107.254:37867 shutting down...
I20260709 13:07:26.407770   431 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 1 LEADER]: Raft consensus shutting down.
I20260709 13:07:26.408752   431 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 13:07:26.409181   431 tablet_replica.cc:333] T 00000000000000000000000000000000 P 44459bc0b4b04edcb412c73f6d2898bd: stopping tablet replica
I20260709 13:07:26.433238   431 master.cc:584] Master@127.0.107.254:37867 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (12451 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (25840 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (435800 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] LeaderRebalancerTest.AddTserver

 1 FAILED TEST
I20260709 13:07:26.503011   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 396 messages since previous log ~15 seconds ago
I20260709 13:07:26.503269   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 395 messages since previous log ~15 seconds ago
I20260709 13:07:26.503433   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 394 messages since previous log ~15 seconds ago
I20260709 13:07:26.503592   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~35 seconds ago
I20260709 13:07:26.503750   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~35 seconds ago
I20260709 13:07:26.503922   431 logging.cc:436] LogThrottler Service Queue Overflow: suppressed but not reported on 2 messages since previous log ~176 seconds ago
I20260709 13:07:26.504073   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/service_pool.cc:136: suppressed but not reported on 2 messages since previous log ~176 seconds ago
I20260709 13:07:26.504248   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 7 messages since previous log ~187 seconds ago
I20260709 13:07:26.504505   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 9 messages since previous log ~0 seconds ago
I20260709 13:07:26.504679   431 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~11 seconds ago
